linux-kernel - Re: [f2fs-dev] [PATCH] f2fs: compress: fix zstd data corruption

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <0d41e29e-c601-e016-e471-aed184ca4a6a@huawei.com>
Date:   Fri, 8 May 2020 10:48:39 +0800
From:   Chao Yu <yuchao0@...wei.com>
To:     Daeho Jeong <daeho43@...il.com>
CC:     <jaegeuk@...nel.org>, Daeho Jeong <daehojeong@...gle.com>,
        <linux-kernel@...r.kernel.org>,
        <linux-f2fs-devel@...ts.sourceforge.net>
Subject: Re: [f2fs-dev] [PATCH] f2fs: compress: fix zstd data corruption

Hi Daeho,

On 2020/5/8 9:28, Daeho Jeong wrote:
> Hi Chao,
> 
> IIUC, you are trying not to use ZSTD_compressBound() to save the memory
> space. Am I right?
> 
> Then, how about LZ4_compressBound() for LZ4 and lzo1x_worst_compress() for
> LZO?

Oops, it looks those limits were wrongly used...

#define LZ4_COMPRESSBOUND(isize)	(\
	(unsigned int)(isize) > (unsigned int)LZ4_MAX_INPUT_SIZE \
	? 0 \
	: (isize) + ((isize)/255) + 16)

#define lzo1x_worst_compress(x) ((x) + ((x) / 16) + 64 + 3 + 2)

Newly calculated boundary size is larger than target buffer size.

However comments on LZ4_compress_default() said:

...
 * @maxOutputSize: full or partial size of buffer 'dest'
 *	which must be already allocated
...
int LZ4_compress_default(const char *source, char *dest, int inputSize,
	int maxOutputSize, void *wrkmem);

And @out_len in lzo1x_1_compress() was passed as an output parameter to
pass length of data that compressor compressed into @out buffer.

Let me know if I missed sth.

Thannks,

> Could we save more memory space for these two cases like ZSTD?
> As you know, we are using 5 pages compression buffer for LZ4 and LZO in
> compress_log_size=2,
> and if the compressed data doesn't fit in 3 pages, it returns -EAGAIN to
> give up compressing that one.
> 
> Thanks,
> 
> 2020년 5월 8일 (금) 오전 10:17, Chao Yu <yuchao0@...wei.com>님이 작성:
> 
>> During zstd compression, ZSTD_endStream() may return non-zero value
>> because distination buffer is full, but there is still compressed data
>> remained in intermediate buffer, it means that zstd algorithm can not
>> save at last one block space, let's just writeback raw data instead of
>> compressed one, this can fix data corruption when decompressing
>> incomplete stored compression data.
>>
>> Signed-off-by: Daeho Jeong <daehojeong@...gle.com>
>> Signed-off-by: Chao Yu <yuchao0@...wei.com>
>> ---
>>  fs/f2fs/compress.c | 7 +++++++
>>  1 file changed, 7 insertions(+)
>>
>> diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c
>> index c22cc0d37369..5e4947250262 100644
>> --- a/fs/f2fs/compress.c
>> +++ b/fs/f2fs/compress.c
>> @@ -358,6 +358,13 @@ static int zstd_compress_pages(struct compress_ctx
>> *cc)
>>                 return -EIO;
>>         }
>>
>> +       /*
>> +        * there is compressed data remained in intermediate buffer due to
>> +        * no more space in cbuf.cdata
>> +        */
>> +       if (ret)
>> +               return -EAGAIN;
>> +
>>         cc->clen = outbuf.pos;
>>         return 0;
>>  }
>> --
>> 2.18.0.rc1
>>
>>
>>
>> _______________________________________________
>> Linux-f2fs-devel mailing list
>> Linux-f2fs-devel@...ts.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
>>
>