linux-ext4 - Re: No space left on device after many files creation

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4BC410AB.1030607@gmail.com>
Date:	Tue, 13 Apr 2010 10:35:23 +0400
From:	cy6erGn0m <cy6ergn0m@...il.com>
To:	tytso@....edu
CC:	Eric Sandeen <sandeen@...hat.com>, linux-ext4@...r.kernel.org
Subject: Re: No space left on device after many files creation

tytso@....edu wrote:
> On Mon, Apr 12, 2010 at 05:00:37PM -0500, Eric Sandeen wrote:
>   
>> cy6erGn0m wrote:
>>     
>>> In the past I have done the same test for ext3 and it was created ~10M
>>> files in one firectory at working filesystem (/home) and it was no
>>> issues reached.
>>>
>>> On ext4 i found inodes max size is limited as you wrote:
>>> [root@...achine sandbox]# df -i /e4
>>> Файловая система      Инодов   Испол   Своб  Исп % смонтирована на
>>> (file system/inodes/used/free/used%/mounted on)
>>> /dev/sda4               1,8M    1,8M       0  100% /e4
>>> [root@...achine sandbox]#
>>>
>>> Why inodes count limit is so small by default? Is it depends on
>>> partition size?
>>>       
>> it does depend on partition size; the inode_ratio = 16384 in mke2fs.conf
>> says 1 inode per 16k of filesystem space.
>>     
>
> 1.8 million inodes is hardly small.....  most people complain that
> we're reserving too much space for inodes.  
>
> Cy6erGn0m, What is your use case where the average file size for your
> 20G partition is smaller that 16k?
>
> 						- Ted
>   
I use filysystem as Set data structure to make fast checks from scripts.
Of course I can make long long file and test it with grep but it's will
have linear performance degradation (List), at the same time file system
has efficient tree binary structures (smth. like TreeSet). So, using
filesystem is the simplest way to do this efficient. Also I always have
timestamps for every entry. In this usecase all files are always empty.

Here is simple performance comparison: list vs tree:
cy6ergn0m@...achine /e4/sandbox/heap $ time cat ../x | grep ^610845$ ../x
610845
cat ../x  0,00s user 0,00s system 2% cpu 0,127 total
grep --color ^610845$ ../x  0,08s user 0,03s system 87% cpu 0,136 total
cy6ergn0m@...achine /e4/sandbox/heap $ time stat 610845            
  File: `610845'
  Size: 0               Blocks: 0          IO Block: 4096   пустой
обычный файл
Device: 804h/2052d      Inode: 1520242     Links: 1
Access: (0644/-rw-r--r--)  Uid: (  500/cy6ergn0m)   Gid: (  500/cy6ergn0m)
Access: 2010-04-13 10:17:58.119477521 +0400
Modify: 2010-04-13 10:17:58.119477521 +0400
Change: 2010-04-13 10:17:58.119477521 +0400
stat 610845  0,00s user 0,00s system 75% cpu 0,005 total
cy6ergn0m@...achine /e4/sandbox/heap $


I know that filesystems is not targeted for this usecase, but inodes
limit looks strange.. why they are can't be allocated dinamically? Can I
enlarge this limit on the fly?

 


Download attachment "smime.p7s" of type "application/x-pkcs7-signature" (3650 bytes)