linux-kernel - Re: pipe(2), read/write, maximums and behavior.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <4e5e476b0907060506m3d57a454va94d2eafd61d73a0@mail.gmail.com>
Date:	Mon, 6 Jul 2009 14:06:17 +0200
From:	Corrado Zoccolo <czoccolo@...il.com>
To:	Linda Walsh <lkml@...nx.org>
Cc:	LKML <linux-kernel@...r.kernel.org>
Subject: Re: pipe(2), read/write, maximums and behavior.

Hi Linda,
the limit displayed by the shell is the amunt of data that the sender
can send to the pipe before it would block waiting for the receiver to
read.
In your case, you make larger transfers, so they become synchronous.

For such large transfers, you should consider using vmsplice, that
allows moving data through pipes without copying.

Corrado

On 7/6/09, Linda Walsh <lkml@...nx.org> wrote:
> I've seen a few shells claim to limit pipe sizes to 8 512Byte buffers.
> Don't know where they get this value or how they think it applies, but
> it certainly doesn't seem to apply in linux.   However, I'm not
> sure what limits do apply compared to available memory.
>
> I suppose, starting off, one might look at at a maximum of
> (Physical+Swap-resident-non-swappable mem)/2 as a top limit.
>
> A test machine I have has 8GB physical memory with a bit over 4GB
> of swap space making for about 12GB of memory.
>
> If total memory was to go toward my proglet that splits into a master
> writer and slave pipe reader, they'd have to split memory to have
> matching buffer read/write sizes. I'd "expect", (I think) at least
> a 2GB write/read to work, and possibly a 4GB write/read to work
> with alot of swap activity -- that's assuming there are no other
> restraints in dividing 12GB of address space.
>
> As it turns out -- the program dies at 2GB (the 1GB write/read works)
> but when the program tries a 2GB write & read it refuses the full write
> and the child gets less than 2GB.
>
> The master gets back that it wrote 2097148KB, though it tried to
> write 2097152KB (and the child receives the 2GB-4K buffer upon read).
>
> This is on a x86_64 machine, and unsigned long values are 8-bytes
> wide and being used with the read and write calls for lengths.
>
> Shouldn't a 2GB read/write work?  At most, together the master
> and slave would have only used 4GB for each to have a 2GB buffer.
>
> How would one determine the maximum size for 1 huge read or write
> through the pipe (from the pipe system call)?
>
> On 2GHz multi-core machines, I get about 512MB/s throughput.
>
> I attached the source file so anyone can see my methodology.
>
> you have to include "-lrt" on the gcc command line as it uses
> clock_gettime to estimate the time for the write call (the read
> call always comes back with values too small to be reasonable, so
> I don't bother printing them.
>
>
>
>


-- 
__________________________________________________________________________

dott. Corrado Zoccolo                          mailto:czoccolo@...il.com
PhD - Department of Computer Science - University of Pisa, Italy
--------------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/