linux-kernel - pohmelfs: call for inclusion

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20120208193413.GA22215@ioremap.net>
Date:	Wed, 8 Feb 2012 22:34:13 +0300
From:	Evgeniy Polyakov <zbr@...emap.net>
To:	linux-kernel@...r.kernel.org
Cc:	greg@...ah.com, torvalds@...ux-foundation.org,
	akpm@...ux-foundation.org
Subject: pohmelfs: call for inclusion

Hi

I'm please to announce new and completely rewritten distributed
filesystem - POHMELFS

It went a long way from parallel NFS design which lived in
drivers/staging/pohmelfs for years effectively without usage case - that
design was dead.

New pohmelfs uses elliptics network [1] as its storage backend, which
was proved as effective distributed system. Elliptics is used in
production in Yandex search company for several years now and clusters
range from small (like 6 nodes in 3 datacenters to host 15 billions of
small files or hundred of nodes to scale to 1 Pb used for streaming).

We start to cook up 2 small clusters for production pohmelfs testing -
one of them is largest in East Europe mirror site, another one is used
for internal package storage.

Pohmelfs is just a POSIX frontend to elliptics. It supports hardlinks,
symlinks, data checksums, multiple copies and so on.
Pohmelfs uses local cache (synced on timely basis) for all operations,
and only sync (or close with sync_on_close mount option) or writeback
will flush data to remote nodes. There is also background work to flush
data to storage like commit in ext3
Directory objects are synced to the storage when they are created on
pohmelfs node, file data is being sent from cache later.

All recovery process is handled by elliptics and is performed without
pohmelfs clients every noticing that. In particular reads are always
directed to the replica with the latest data (determined according to
ellptics metadata checked at file open time).

Writes in pohmelfs can be configured to succeed when quorum commits it
or when specified in 'successful_write_count=' number of writes return
ok. This is useful when you do not care much about number of active
replicas, since you know that your data is safe. Or when you create new
storage and want to copy to single datacenter to save bandwidth.

But that's enough for advertisement. Here is the code.
I do not know whether it is a good idea to ask it for inclusion into
mainline tree right now skipping drivers/staging part. But anyway, I
post patch to remove drivers/staging/pohmelfs and add fs/pohmelfs
If it is not the way to go, I will resubmit.

1. Elliptics network
http://www.ioremap.net/projects/elliptics
http://www.elliptics.ru (russian support forum - we understand english
and will answer pohmelfs questions too)

2. Pohmelfs
http://www.ioremap.net/projects/pohmelfs (old and not yep updated site)
http://www.ioremap.net/taxonomy/term/4 (development blog section)

Signed-off-by: Evgeniy Polyakov <zbr@...emap.net>

-- 
	Evgeniy Polyakov

View attachment "pohmelfs.diff" of type "text/x-diff" (144896 bytes)

View attachment "pohmelfs-staging-remove.diff" of type "text/x-diff" (204755 bytes)