netdev - Re: [PATCH net] ipv4: tcp: Fix SO

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20200701012248.7t6tif6mspaggy7j@kafai-mbp.dhcp.thefacebook.com>
Date:   Tue, 30 Jun 2020 18:22:48 -0700
From:   Martin KaFai Lau <kafai@...com>
To:     Willem de Bruijn <willemdebruijn.kernel@...il.com>
CC:     Network Development <netdev@...r.kernel.org>,
        David Miller <davem@...emloft.net>,
        Kernel Team <kernel-team@...com>
Subject: Re: [PATCH net] ipv4: tcp: Fix SO_MARK in RST and ACK packet

On Tue, Jun 30, 2020 at 08:45:13PM -0400, Willem de Bruijn wrote:
> On Tue, Jun 30, 2020 at 7:46 PM Martin KaFai Lau <kafai@...com> wrote:
> >
> > On Tue, Jun 30, 2020 at 07:20:46PM -0400, Willem de Bruijn wrote:
> > > On Tue, Jun 30, 2020 at 6:18 PM Martin KaFai Lau <kafai@...com> wrote:
> > > >
> > > > When testing a recent kernel (5.6 in our case), the skb->mark of the
> > > > IPv4 TCP RST pkt does not carry the mark from sk->sk_mark.  It is
> > > > discovered by the bpf@tc that depends on skb->mark to work properly.
> > > > The same bpf prog has been working in the earlier kernel version.
> > > > After reverting commit c6af0c227a22 ("ip: support SO_MARK cmsg"),
> > > > the skb->mark is set and seen by bpf@tc properly.
> > > >
> > > > We have noticed that in IPv4 TCP RST but it should also
> > > > happen to the ACK based on tcp_v4_send_ack() is also depending
> > > > on ip_send_unicast_reply().
> > > >
> > > > This patch tries to fix it by initializing the ipc.sockc.mark to
> > > > fl4.flowi4_mark.
> > > >
> > > > Fixes: c6af0c227a22 ("ip: support SO_MARK cmsg")
> > > > Cc: Willem de Bruijn <willemb@...gle.com>
> > > > Signed-off-by: Martin KaFai Lau <kafai@...com>
> > > > ---
> > > >  net/ipv4/ip_output.c | 1 +
> > > >  1 file changed, 1 insertion(+)
> > > >
> > > > diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
> > > > index 090d3097ee15..033512f719ec 100644
> > > > --- a/net/ipv4/ip_output.c
> > > > +++ b/net/ipv4/ip_output.c
> > > > @@ -1703,6 +1703,7 @@ void ip_send_unicast_reply(struct sock *sk, struct sk_buff *skb,
> > > >         sk->sk_bound_dev_if = arg->bound_dev_if;
> > > >         sk->sk_sndbuf = sysctl_wmem_default;
> > > >         sk->sk_mark = fl4.flowi4_mark;
> > > > +       ipc.sockc.mark = fl4.flowi4_mark;
> > > >         err = ip_append_data(sk, &fl4, ip_reply_glue_bits, arg->iov->iov_base,
> > > >                              len, 0, &ipc, &rt, MSG_DONTWAIT);
> > > >         if (unlikely(err)) {
> > >
> > > Yes, this total sense. I missed these cases.
> > >
> > > Slight modification, the line above then no longer needs to be set.
> > > That line was added in commit bf99b4ded5f8 ("tcp: fix mark propagation
> > > with fwmark_reflect enabled"). Basically, it pretends that the socket
> > > has a mark associated, but sk here is always the (netns) global
> > > control sock. So your BPF program was depending on fwmark_reflect?
> > Make sense.  I was also tempting to remove the line above.
> > Thanks for the commit pointer.
> >
> > No, the BPF program does not depend on fwmark_reflect.  It depends
> > on the sk->sk_mark set by a user space process.
> 
> Then I don't fully understand, as ip_send_unicast_reply is only called
> with the per-netns percpu ctl_sk.
Before this proposed patch, my understanding is,
the ctl_sk->sk_mark is correctly set here in ip_send_unicast_reply().
The ctl_sk->sk_mark was actually set earlier in the tcp_v4_send_reset().

However, ctl_sk->sk_mark is not used to set the skb->mark.
Instead, cork->mark is now used to initialize the skb->mark
in __ipmake_skb().

The cork->mark is not properly set in ip_setup_cork()
because ipc->sockc.mark is 0 here which this proposed patch
is trying to address.

The call stack is something like this:

tcp_v4_send_reset()
    ctl_sk->sk_mark = (sk->sk_state == TCP_TIME_WAIT) ?... /* <- sk_mark is set */
=> ip_send_unicast_reply()
   => ip_append_data(..., &ipc, ...);
       => ip_setup_cork(..., &inet->cork.base, ipc, ...);
          cork->mark = ipc->sockc.mark; /* <-- ipc->sockc.mark is 0 */
   => ip_push_pending_frames(sk, &fl4);
      => ip_finish_skb(sk, fl4)
         => __ip_make_skb(sk, fl4, ..., &inet_sk(sk)->cork.base);
            skb->mark = cork->mark; /* <-- cork->mark is 0  */