Message-ID: <CAHXsExwuSyn7eVMqiOcatU5C3WHsdHEnLJcVh-jf2LjmzW2Edg@mail.gmail.com>
Date: Fri, 12 Jul 2024 17:02:13 -0400
From: Jason Zhou <jasonzhou@...om>
To: Ido Schimmel <idosch@...sch.org>
Cc: netdev@...r.kernel.org, Benjamin Mahler <bmahler@...om>, Jun Wang <junwang@...om>
Subject: Re: PROBLEM: Issue with setting veth MAC address being unreliable.

Hi all,

We have performed our own testing with the MacAddressPolicy set to
none rather than persistent on CentOS 9, and it fixed the problem we
were seeing with the MAC address mismatches before and after us trying
to manually set it.
So we're pretty confident that the cause is what Ido stated, and that
we were racing against udev as we did not set a MAC address when
creating our veth device pair, making udev think it should give out a
new MAC address.
We will release a patch on Apache Mesos to mitigate this potential
race condition on systems with systemd version > 242.
Thank you so much for the help!

For documenting this issue, I believe that this race condition would
also be present for the peer veth interface?
We create the peer along with veth and move the peer to another
namespace, but udev would be notified of its creation, so it will try
to also overwrite the peer's MAC address.
However this is not an issue for the loopback interface that comes
with every namespace creation, as they will not be affected by
NetworkManager and hence udev will not try to modify them.
Please correct me if I'm wrong!

Once again, thanks for all the help!
Best regards,
Jason Zhou
Software Engineering Intern
jasonzhou@...om


On Thu, Jul 11, 2024 at 10:07 AM Jason Zhou <jasonzhou@...om> wrote:
>
> Apologies for the late reply,
>
> > When you say 5.15.161, do you mean true 5.15.161, or is this CentOS
> > version of 5.15.161 with 1000s of patches on top? Did you download the
> > sources from www.kernel.org and build it yourself?
>
> For Andrew's question, yes, we do use the mainline 5.15 LTS kernel
> which we built ourselves from the www.kernel.org sources.
>
> On Thu, Jul 11, 2024 at 9:18 AM Ido Schimmel <idosch@...sch.org> wrote:
> >
> > On Wed, Jul 10, 2024 at 04:45:55PM -0400, Jason Zhou wrote:
> > > [1.] One line summary of the problem:
> > >
> > > Issue with setting veth address being unreliable.
> > >
> > > [2.] Full description of the problem/report:
> > >
> > > Hello!
> > >
> > > We have been investigating a strange behavior within Apache Mesos
> > > where after setting the MAC address on a veth device to the same
> > > address as our eth0 MAC address, the change is sometimes not reflected
> > > appropriately despite the ioctl call succeeding (~4% of the time in
> > > our testing). Note that we also tried using libnl to set the MAC
> > > address but the issue still persists.
> > >
> > > Included below is the github link to the section where we set the veth
> > > address, to clarify what we were trying to do. We first create the
> > > veth pair [1] using a libnl function [2], then we set the veth device
> > > MAC addresses to that of our host public interface (eth0) [3] using a
> > > function called setMAC. Inside the setMAC [4] is where we are
> > > observing the aforementioned issue with unreliable setting of veth
> > > addresses..
> > >
> > > This behavior was observed when re-fetching the MAC address on said
> > > veth device after we made the function call to set its MAC address. We
> > > have observed this issue on CentOS 9 only, but not on CentOS 7. We
> > > have tried Linux kernels 5.15.147, 5.15.160 & 5.15.161 for CentOS 9,
> > > CentOS 7 was using 5.10, but we also tried upgrading the Centos 7 host
> > > to 5.15.160 but could not reproduce the bug.
> >
> > This suggests that the change in behavior is due to a change in user
> > space and an obvious suspect is systemd / udev. AFAICT, CentOS 7 is
> > using systemd 219 whereas CentOS 9 is using systemd 252. In version 242
> > systemd started setting a persistent MAC address on virtual interfaces.
> > See:
> >
> > https://github.com/Mellanox/mlxsw/wiki/Persistent-Configuration#required-changes-to-macaddresspolicy
> >
> > If that is the issue, then you can either change MACAddressPolicy or
> > modify your code to create the veth pair with the correct MAC address
> > from the start.
> >
> > As I understand it, a possible explanation for the race is that your
> > program is racing with udev. Udev receives an event about a new device
> > and reads "addr_assign_type" from sysfs. If your program changed the MAC
> > address before udev read the file, then udev will read 3 (NET_ADDR_SET)
> > and will not try to change the MAC address. Otherwise, both your program
> > and udev will try to change the MAC address.
> >
> > If it is not udev that is changing the MAC address, then you can run the
> > following bpftrace script in a different terminal and check which
> > processes try to change the address:
> >
> > bpftrace -e 'k:dev_set_mac_address_user { @[comm] = count(); }'
> >
> > It is only a theory. Actual issue might be entirely different.
> >
>
> Thank you Ido, we will test whether the systemd change is the true
> culprit behind this, and will follow up with what we find :)
>
> Appreciate all the help so far!
>
> Best regards,
> Jason Zhou
> Software Engineering Intern
> jasonzhou@...om
>
>
> > >
> > > We were re-fetching the addresses via the ioctl SIOCGIFHWADDR syscall
> > > as well as via getifaddr (which appears to use netlink under the
> > > covers), and, in problematic cases, both functions reported
> > > discrepancies from the target MAC address we were initially setting
> > > to. We also performed a fetch before we set the MAC addresses and
> > > found that there are instances where getifaddr and ioctl results do
> > > not match for our veth device *even before we perform any setting of
> > > the MAC address*. It's also worth noting that after setting the MAC
> > > address: there are no cases where ioctl or getifaddr come back with
> > > the same MAC address as before we set the address. So, the set
> > > operation always seems to have an effect.
> > >
> > > Observed scenarios with incorrectly assigned MAC addresses:
> > >
> > > (1) After setting the mac address: ioctl returns the correct MAC
> > > address, but the results from getifaddr, returns an incorrect MAC
> > > address (different from the original value before setting as well!)
> > >
> > > (2) After setting the MAC address: both ioctl and getifaddr return the
> > > same MAC address, but are both wrong (and different from the original
> > > one!)
> > >
> > > (3) There is a possibility that the MAC address we set ends up
> > > overwritten by a garbage value *after* we have already updated the MAC
> > > address, and checked that the MAC address was set correctly. Since
> > > this error happens after this function has finished, we cannot log nor
> > > detect it in the function where we set the MAC address because we have
> > > not yet studied at what point this late overwriting of MAC address
> > > occurs. It’s worth noting that this is the rarest scenario that we
> > > have encountered, and we were only able to reproduce it in our testing
> > > cluster machine, not in any of the production cluster machines.
> > >
> > > [3.] Keywords:
> > >
> > > networking, veth, kernel, MAC, netlink
> > >
> > > [X.] Other notes, patches, fixes, workarounds:
> > >
> > > Notes:
> > >
> > > More specific kernel and environment information will be available on
> > > request for security reasons, please let us know if you are interested
> > > and we will be happy to provide you with the necessary information.
> > >
> > > We have observed this behavior only on CentOS 9 systems at the moment,
> > > CentOS 7 systems under various kernels do not seem to have the issue
> > > (which is quite strange if this was purely a kernel bug).
> > >
> > > We have tried kernels 5.15.147, 5.15.160, 5.15.161, all of these have
> > > this issue on CentOS 9.
> > >
> > > We have also tried rewriting our function for setting MAC address to
> > > use libnl rather than ioctl to perform the MAC address setting, but it
> > > did not eliminate the issue.
> > >
> > > To work around this bug, we checked that the MAC address is set
> > > correctly after the ioctl set call, and retry the address setting if
> > > necessary. In our testing, this workaround appears to remedy scenarios
> > > (1) and (2) above, but it does not address scenario (3).  You can see
> > > it here:
> > >
> > > https://github.com/apache/mesos/commit/8b202bbebdc89429ad82c6983aa1c514eb1b8d95
> > >
> > > We would greatly appreciate any insights or guidance on this matter.
> > > Please let me know if you need further information or if there are any
> > > specific tests we should run to assist in diagnosing the issue. Again,
> > > specific details for the production machines on which we encountered
> > > this error can be provided upon request, so please let us know if
> > > there is anything we can provide to help.
> > >
> > > Thank you for your time and assistance.
> > >
> > > Best regards,
> > > Jason Zhou
> > > Software Engineering Intern
> > > jasonzhou@...om
> > >
> > > embedded links:
> > > [1] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/slave/containerizer/mesos/isolators/network/port_mapping.cpp#L3599
> > > [2] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/linux/routing/link/veth.cpp#L45
> > > [3] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/slave/containerizer/mesos/isolators/network/port_mapping.cpp#L3628
> > > [4] https://github.com/apache/mesos/blob/8cf287778371c13ee7e88fa428424b3c0fbc7ff0/src/linux/routing/link/link.cpp#L283
> > >

Powered by blists - more mailing lists