netdev - Re: [PATCH iproute2 net-next v2 3/4] ss: Buffer raw fields first, then render them as a table

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 13 Feb 2019 18:38:31 +0100
From:   Stefano Brivio <sbrivio@...hat.com>
To:     Eric Dumazet <eric.dumazet@...il.com>
Cc:     Stephen Hemminger <stephen@...workplumber.org>,
        netdev@...r.kernel.org, Sabrina Dubroca <sd@...asysnail.net>,
        David Ahern <dsahern@...il.com>
Subject: Re: [PATCH iproute2 net-next v2 3/4] ss: Buffer raw fields first,
 then render them as a table

On Wed, 13 Feb 2019 09:31:03 -0800
Eric Dumazet <eric.dumazet@...il.com> wrote:

> On 02/13/2019 12:37 AM, Stefano Brivio wrote:
> > On Tue, 12 Feb 2019 16:42:04 -0800
> > Eric Dumazet <eric.dumazet@...il.com> wrote:
> >   
> >> I do not get it.
> >>
> >> "ss -emoi " uses almost 1KB per socket.
> >>
> >> 10,000,000 sockets -> we need about 10GB of memory  ???
> >>
> >> This is a serious regression.  
> > 
> > I guess this is rather subjective: the worst case I considered back then
> > was the output of 'ss -tei0' (less than 500 bytes) for one million
> > sockets, which gives 500M of memory, which should in turn be fine on a
> > machine handling one million sockets.
> > 
> > Now, if 'ss -emoi' on 10 million sockets is an actual use case (out of
> > curiosity: how are you going to process that output? Would JSON help?),
> > I see two easy options to solve this:  
> 
> 
> ss -temoi | parser (written in shell or awk or whatever...)
> 
> This is a use case, I just got bitten because using ss command
> actually OOM my container, while trying to debug a busy GFE.
> 
> The host itself can have 10,000,000 TCP sockets, but usually sysadmin shells
> run in a container with no more than 500 MB available. 
> 
> Otherwise, it would be too easy for a buggy program to OOM the whole machine
> and have angry customers.

Ouch, I see.

> > 
> > 1. flush the output every time we reach a given buffer size (1M
> >    perhaps). This might make the resulting blocks slightly unaligned,
> >    with occasional loss of readability on lines occurring every 1k to
> >    10k sockets approximately, even though after 1k sockets column sizes
> >    won't change much (it looks anyway better than the original), and I
> >    don't expect anybody to actually scroll that output
> > 
> > 2. add a switch for unbuffered output, but then you need to remember to
> >    pass it manually, and the whole output would be as bad as the
> >    original in case you need the switch.
> > 
> > I'd rather go with 1., it's easy to implement (we already have partial
> > flushing with '--events') and it looks like a good compromise on
> > usability. Thoughts?
> >   
> 
> 1 seems fine, but a switch for 'please do not try to format' would be fine.

Let me try with 1. first then -- we already have a huge number of
switches.

> I wonder why we try to 'format' when stdout is a pipe or a regular file .

As stupid as it might sound, I just didn't think of fixing that :)

What would you suggest in that case, single whitespaces? Tabs?

-- 
Stefano