linux-kernel - Re: [PATCH v3] exfat: remove EXFAT_SB

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Wed, 12 Aug 2020 18:19:32 +0900
From:   Tetsuhiro Kohada <kohada.t2@...il.com>
To:     Sungjong Seo <sj1557.seo@...sung.com>
Cc:     kohada.tetsuhiro@...mitsubishielectric.co.jp,
        mori.takahiro@...mitsubishielectric.co.jp,
        motai.hirotaka@...mitsubishielectric.co.jp,
        'Namjae Jeon' <namjae.jeon@...sung.com>,
        linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH v3] exfat: remove EXFAT_SB_DIRTY flag

>>
>> When should VOL_DIRTY be cleared?
>>
>> The current behavior is ...
>>
>> Case of  mkdir, rmdir, rename:
>>     - set VOL_DIRTY before operation
>>     - set VOL_CLEAN after operating.
>> In async mode, it is actually written to the media after 30 seconds.
>>
>> Case of  cp, touch:
>>     - set VOL_DIRTY before operation
>>     - however, VOL_CLEAN is not called in this context.
>> VOL_CLEAN will call by sync_fs or unmount.
>>
>> I added VOL_CLEAN in last of __exfat_write_inode() and exfat_map_cluster().
>> As a result, VOL_DIRTY is cleared with cp and touch.
>> However, when copying a many files ...
>>    - Async mode: VOL_DIRTY is written to the media twice every 30 seconds.
>>    - Sync mode: Of course,  VOL_DIRTY and VOL_CLEAN to the media for each
>> file.
>>
>> Frequent writing VOL_DIRTY and VOL_CLEAN  increases the risk of boot-
>> sector curruption.
>> If the boot-sector corrupted, it causes the following serious problems  on
>> some OSs.
>>    - misjudge as unformatted
>>    - can't judge as exfat
>>    - can't repair
>>
>> I want to minimize boot sector writes, to reduce these risk.
>>
>> I looked vfat/udf implementation, which manages similar dirty information
>> on linux, and found that they ware mark-dirty at mount and cleared at
>> unmount.
>>
>> Here are some ways to clear VOL_DIRTY.
>>
>> (A) VOL_CLEAN after every write operation.
>>     :-) Ejectable at any time after a write operation.
>>     :-( Many times write to Boot-sector.
>>
>> (B) dirty at mount, clear at unmount (same as vfat/udf)
>>     :-) Write to boot-sector twice.
>>     :-( It remains dirty unless unmounted.
>>     :-( Write to boot-sector even if there is no write operation.
>>
>> (C) dirty on first write operation, clear on unmount
>>     :-) Writing to boot-sector is minimal.
>>     :-) Will not write to the boot-sector if there is no write operation.
>>     :-( It remains dirty unless unmounted.
>>
>> (D) dirty on first write operation,  clear on sync-fs/unmount
>>    :-) Writing to boot-sector can be reduced.
>>    :-) Will not write to the boot-sector if there is no write operation.
>>    :-) sync-fs makes it clean and ejectable immidiately.
>>    :-( It remains dirty unless sync-fs or unmount.
>>    :-( Frequent sync-fs will  increases writes to boot-sector.
>>
>> I think it should be (C) or(D).
>> What do you think?
>>
> 
> First of all, I'm sorry for the late reply.
> And thank you for the suggestion.

Thanks for thinking on this complicated issue.


> Most of the NAND flash devices and HDDs have wear leveling and bad sector replacement algorithms applied.
> So I think that the life of the boot sector will not be exhausted first.

I'm not too worried about the life of the boot-sector.
I'm worried about write failures caused by external factors.
(power failure/system down/vibration/etc. during writing)
They rarely occur on SD cards, but occur on many HDDs, some SSDs and USB storages by 0.1% or more.
Especially with AFT-HDD, not only boot-sector but also the following multiple sectors become unreadable.
It is not possible to completely solve this problem, as long as writing to the boot-sector.
(I think it's a exFAT's specification defect)
The only effective way to reduce this problem is to reduce writes to the boot-sector.


> Currently the volume dirty/clean policy of exfat-fs is not perfect,

Thank you for sharing the problem with you.


> but I think it behaves similarly to the policy of MS Windows.

On Windows10, the dirty flag is cleared after more than 15 seconds after all write operations are completed.
(dirty-flag is never updated during the write operation continues)


> Therefore,
> I think code improvements should be made to reduce volume flag records while maintaining the current policy.

Current policy is inconsistent.
As I wrote last mail, the problem with the current implementation is that the dirty-flag may not be cleared
after the write operation.(even if sync is enabled or disabled)
Because, some write operations clear the dirty-flag but some don't clear.
Unmount or sync command is the only way to ensure that the dirty-flag is cleared.
This has no effect on clearing the dirty-flag after a write operations, it only increases risk of destroying
the boot-sector.
  - Clear the dirty-flag after every write operation.
  - Never clear the dirty-flag after every write operation.
Unless unified to either one,  I think that sync policy cannot be consistent.

How do you think?


BR
---
etsuhiro Kohada <kohada.t2@...il.com>