| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <87tu0xckab.fsf@cloudflare.com>
Date: Wed, 11 Jan 2023 13:44:03 +0100
From: Jakub Sitnicki <jakub@...udflare.com>
To: Kuniyuki Iwashima <kuniyu@...zon.com>
Cc: davem@...emloft.net, edumazet@...gle.com,
kernel-team@...udflare.com, kuba@...nel.org, marek@...udflare.com,
netdev@...r.kernel.org, pabeni@...hat.com
Subject: Re: [PATCH net-next v2 1/2] inet: Add IP_LOCAL_PORT_RANGE socket
option
On Wed, Jan 11, 2023 at 09:59 AM +09, Kuniyuki Iwashima wrote:
> From: Jakub Sitnicki <jakub@...udflare.com>
> Date: Tue, 10 Jan 2023 14:37:29 +0100
>> Users who want to share a single public IP address for outgoing connections
>> between several hosts traditionally reach for SNAT. However, SNAT requires
>> state keeping on the node(s) performing the NAT.
>>
>> A stateless alternative exists, where a single IP address used for egress
>> can be shared between several hosts by partitioning the available ephemeral
>> port range. In such a setup:
>>
>> 1. Each host gets assigned a disjoint range of ephemeral ports.
>> 2. Applications open connections from the host-assigned port range.
>> 3. Return traffic gets routed to the host based on both, the destination IP
>> and the destination port.
>>
>> An application which wants to open an outgoing connection (connect) from a
>> given port range today can choose between two solutions:
>>
>> 1. Manually pick the source port by bind()'ing to it before connect()'ing
>> the socket.
>>
>> This approach has a couple of downsides:
>>
>> a) Search for a free port has to be implemented in the user-space. If
>> the chosen 4-tuple happens to be busy, the application needs to retry
>> from a different local port number.
>>
>> Detecting if 4-tuple is busy can be either easy (TCP) or hard
>> (UDP). In TCP case, the application simply has to check if connect()
>> returned an error (EADDRNOTAVAIL). That is assuming that the local
>> port sharing was enabled (REUSEADDR) by all the sockets.
>>
>> # Assume desired local port range is 60_000-60_511
>> s = socket(AF_INET, SOCK_STREAM)
>> s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
>> s.bind(("192.0.2.1", 60_000))
>> s.connect(("1.1.1.1", 53))
>> # Fails only if 192.0.2.1:60000 -> 1.1.1.1:53 is busy
>> # Application must retry with another local port
>>
>> In case of UDP, the network stack allows binding more than one socket
>> to the same 4-tuple, when local port sharing is enabled
>> (REUSEADDR). Hence detecting the conflict is much harder and involves
>> querying sock_diag and toggling the REUSEADDR flag [1].
>>
>> b) For TCP, bind()-ing to a port within the ephemeral port range means
>> that no connecting sockets, that is those which leave it to the
>> network stack to find a free local port at connect() time, can use
>> the this port.
>>
>> IOW, the bind hash bucket tb->fastreuse will be 0 or 1, and the port
>> will be skipped during the free port search at connect() time.
>>
>> 2. Isolate the app in a dedicated netns and use the use the per-netns
>> ip_local_port_range sysctl to adjust the ephemeral port range bounds.
>>
>> The per-netns setting affects all sockets, so this approach can be used
>> only if:
>>
>> - there is just one egress IP address, or
>> - the desired egress port range is the same for all egress IP addresses
>> used by the application.
>>
>> For TCP, this approach avoids the downsides of (1). Free port search and
>> 4-tuple conflict detection is done by the network stack:
>>
>> system("sysctl -w net.ipv4.ip_local_port_range='60000 60511'")
>>
>> s = socket(AF_INET, SOCK_STREAM)
>> s.setsockopt(SOL_IP, IP_BIND_ADDRESS_NO_PORT, 1)
>> s.bind(("192.0.2.1", 0))
>> s.connect(("1.1.1.1", 53))
>> # Fails if all 4-tuples 192.0.2.1:60000-60511 -> 1.1.1.1:53 are busy
>>
>> For UDP this approach has limited applicability. Setting the
>> IP_BIND_ADDRESS_NO_PORT socket option does not result in local source
>> port being shared with other connected UDP sockets.
>>
>> Hence relying on the network stack to find a free source port, limits the
>> number of outgoing UDP flows from a single IP address down to the number
>> of available ephemeral ports.
>>
>> To put it another way, partitioning the ephemeral port range between hosts
>> using the existing Linux networking API is cumbersome.
>>
>> To address this use case, add a new socket option at the SOL_IP level,
>> named IP_LOCAL_PORT_RANGE. The new option can be used to clamp down the
>> ephemeral port range for each socket individually.
>>
>> The option can be used only to narrow down the per-netns local port
>> range. If the per-socket range lies outside of the per-netns range, the
>> latter takes precedence.
>>
>> UAPI-wise, the low and high range bounds are passed to the kernel as a pair
>> of u16 values packed into a u32. This avoids pointer passing.
>>
>> PORT_LO = 40_000
>> PORT_HI = 40_511
>>
>> s = socket(AF_INET, SOCK_STREAM)
>> v = struct.pack("I", PORT_HI << 16 | PORT_LO)
>> s.setsockopt(SOL_IP, IP_LOCAL_PORT_RANGE, v)
>> s.bind(("127.0.0.1", 0))
>> s.getsockname()
>> # Local address between ("127.0.0.1", 40_000) and ("127.0.0.1", 40_511),
>> # if there is a free port. EADDRINUSE otherwise.
>>
>> [1] https://github.com/cloudflare/cloudflare-blog/blob/232b432c1d57/2022-02-connectx/connectx.py#L116
>>
>> v1 -> v2:
>> * Fix the corner case when the per-socket range doesn't overlap with the
>> per-netns range. Fallback correctly to the per-netns range. (Kuniyuki)
>>
>> Reviewed-by: Marek Majkowski <marek@...udflare.com>
>> Signed-off-by: Jakub Sitnicki <jakub@...udflare.com>
>> ---
[...]
>> include/net/ip.h | 3 ++-
>> include/uapi/linux/in.h | 1 +
>> net/ipv4/inet_connection_sock.c | 25 +++++++++++++++++++++++--
>> net/ipv4/inet_hashtables.c | 2 +-
>> net/ipv4/ip_sockglue.c | 18 ++++++++++++++++++
>> net/ipv4/udp.c | 2 +-
>> 7 files changed, 50 insertions(+), 5 deletions(-)
>>
>> diff --git a/include/net/inet_sock.h b/include/net/inet_sock.h
>> index bf5654ce711e..51857117ac09 100644
>> --- a/include/net/inet_sock.h
>> +++ b/include/net/inet_sock.h
>> @@ -249,6 +249,10 @@ struct inet_sock {
>> __be32 mc_addr;
>> struct ip_mc_socklist __rcu *mc_list;
>> struct inet_cork_full cork;
>> + struct {
>> + __u16 lo;
>> + __u16 hi;
>> + } local_port_range;
>> };
>>
>> #define IPCORK_OPT 1 /* ip-options has been held in ipcork.opt */
>> diff --git a/include/net/ip.h b/include/net/ip.h
>> index 144bdfbb25af..c3fffaa92d6e 100644
>> --- a/include/net/ip.h
>> +++ b/include/net/ip.h
>> @@ -340,7 +340,8 @@ static inline u64 snmp_fold_field64(void __percpu *mib, int offt, size_t syncp_o
>> } \
>> }
>>
>> -void inet_get_local_port_range(struct net *net, int *low, int *high);
>> +void inet_get_local_port_range(const struct net *net, int *low, int *high);
>> +void inet_sk_get_local_port_range(const struct sock *sk, int *low, int *high);
>>
>> #ifdef CONFIG_SYSCTL
>> static inline bool inet_is_local_reserved_port(struct net *net, unsigned short port)
>> diff --git a/include/uapi/linux/in.h b/include/uapi/linux/in.h
>> index 07a4cb149305..4b7f2df66b99 100644
>> --- a/include/uapi/linux/in.h
>> +++ b/include/uapi/linux/in.h
>> @@ -162,6 +162,7 @@ struct in_addr {
>> #define MCAST_MSFILTER 48
>> #define IP_MULTICAST_ALL 49
>> #define IP_UNICAST_IF 50
>> +#define IP_LOCAL_PORT_RANGE 51
>>
>> #define MCAST_EXCLUDE 0
>> #define MCAST_INCLUDE 1
>> diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c
>> index d1f837579398..1049a9b8d152 100644
>> --- a/net/ipv4/inet_connection_sock.c
>> +++ b/net/ipv4/inet_connection_sock.c
>> @@ -117,7 +117,7 @@ bool inet_rcv_saddr_any(const struct sock *sk)
>> return !sk->sk_rcv_saddr;
>> }
>>
>> -void inet_get_local_port_range(struct net *net, int *low, int *high)
>> +void inet_get_local_port_range(const struct net *net, int *low, int *high)
>> {
>> unsigned int seq;
>>
>> @@ -130,6 +130,27 @@ void inet_get_local_port_range(struct net *net, int *low, int *high)
>> }
>> EXPORT_SYMBOL(inet_get_local_port_range);
>>
>> +void inet_sk_get_local_port_range(const struct sock *sk, int *low, int *high)
>> +{
>> + const struct inet_sock *inet = inet_sk(sk);
>> + const struct net *net = sock_net(sk);
>> + int lo, hi, sk_lo, sk_hi;
>> +
>> + inet_get_local_port_range(net, &lo, &hi);
>> +
>> + sk_lo = inet->local_port_range.lo;
>> + sk_hi = inet->local_port_range.hi;
>> +
>> + if (unlikely(sk_lo && sk_lo <= hi))
>> + lo = max(lo, sk_lo);
>> + if (unlikely(sk_hi && sk_hi >= lo))
>> + hi = min(hi, sk_hi);
>
> nit: The min of sysctl lo/hi is 1, so
>
> if (unlikely(lo <= sk_lo && sk_lo <= hi))
> lo = sk_lo;
> if (unlikely(lo <= sk_hi && sk_hi <= hi))
> hi = sk_hi;
>
> this seems cleaner.
That is much cleaner. Will apply to v3.
Powered by blists - more mailing lists