linux-kernel - How to do buffered File IO event-based programs?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <4F840064.70201@teldat.de>
Date:	Tue, 10 Apr 2012 11:41:56 +0200
From:	Nils Rennebarth <nils.rennebarth@...dat.de>
To:	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: How to do buffered File IO event-based programs?

Hi,

I am writing a daemon that mostly reads and writes over network connections,
and uses a single threaded event-loop based design using epoll.

Now the daemon occasionally needs to read and write to disk, because it
gets data that is too big to keep in memory. The corresponding request
gets slowed down of course, that is ok. What I would like to avoid
however, is slowing down other client connections to the same
daemon.

As I understand it, just adding the disk-io filedescriptor to the epoll
does not work because it always is ready for reading and writing.

I could use aio, but that is cumbersome as it sets serious alingment
restrictions on the buffers that get written/get read into. Also that
way, the data will circumvent the buffer cache which is not what
I want, to the contrary: data written to a file is very likely to be
read again soon, so if there happens to be enogh free RAM available,
I would like to use it.

I could use a second worker process connected to the other via
a pipe or a unix domain socket. That does work but adds another
userspace-kernel copy plus context switch to each read and write.
This is probably irrelevant in the presence of actual disk io,
and will be the route to go in the absence of other ideas.

Do I ask something unreasonable, is my design flawed anyway, or
is there a concept/unofficial kernel patch flowing around that I
missed?

-- 

Mit freundlichen Grüßen / with kind regards

Nils Rennebarth, Software Developer

--
Funkwerk IP-Appliances GmbH
Mönchhaldenstraße 28
D-70191 Stuttgart

Tel: +49 711 900300 - 0
Fax: +49 711 900300 - 90

E-Mail: Nils.Rennebarth@...kwerk-ec.com

Location: GmbH Nuernberg, Local Court Nuernberg, HRB 25481
Managing Directors: Torsten Urban
--------------------------------
The information contained in this e-mail has been carefully researched,
but the possibility of it being inapplicable in individual cases cannot
be ruled out. We therefore regret that we cannot accept responsibility
or liability of any kind whatsoever for the correctness of the
information given. Please notify us if you discover that information is
inapplicable.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/