linux-kernel - Re: VFS hot tracking: How to calculate data temperature?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <1352118327.2705.26.camel@menhir>
Date:	Mon, 05 Nov 2012 12:25:27 +0000
From:	Steven Whitehouse <swhiteho@...hat.com>
To:	Zhi Yong Wu <zwu.kernel@...il.com>
Cc:	Dave Chinner <david@...morbit.com>, cmm@...ibm.com,
	linux-fsdevel@...r.kernel.org, linuxram@...ux.vnet.ibm.com,
	Ben Chociej <bchociej@...il.com>,
	James Northrup <northrup.james@...il.com>,
	linux-kernel mlist <linux-kernel@...r.kernel.org>
Subject: Re: VFS hot tracking: How to calculate data temperature?

Hi,

On Mon, 2012-11-05 at 20:18 +0800, Zhi Yong Wu wrote:
> On Mon, Nov 5, 2012 at 7:57 PM, Steven Whitehouse <swhiteho@...hat.com> wrote:
> > Hi,
> >
> > On Mon, 2012-11-05 at 19:46 +0800, Zhi Yong Wu wrote:
> >> On Mon, Nov 5, 2012 at 6:33 PM, Steven Whitehouse <swhiteho@...hat.com> wrote:
> >> > Hi,
> >> >
> >> > On Mon, 2012-11-05 at 16:44 +0800, Zhi Yong Wu wrote:
> >> >> On Mon, Nov 5, 2012 at 4:28 PM, Dave Chinner <david@...morbit.com> wrote:
> >> >> > On Mon, Nov 05, 2012 at 10:35:50AM +0800, Zhi Yong Wu wrote:
> >> >> >> On Sat, Nov 3, 2012 at 5:27 AM, Mingming.cao <cmm@...ibm.com> wrote:
> >> >> >> > On Fri, 2012-11-02 at 14:38 +0800, Zhi Yong Wu wrote:
> >> >> >> >> Here also has another question.
> >> >> >> >>
> >> >> >> >> How to save the file temperature among the umount to be able to
> >> >> >> >> preserve the file tempreture after reboot?
> >> >> >> >>
> >> >> >> >> This above is the requirement from DB product.
> >> >> >> >> I thought that we can save file temperature in its inode struct, that
> >> >> >> >> is, add one new field in struct inode, then this info will be written
> >> >> >> >> to disk with inode.
> >> >> >> >>
> >> >> >> >> Any comments or ideas are appreciated, thanks.
> >> >> >> >>
> >> >> >> >>
> >> >> >> >
> >> >> >> > Maybe could save the last file temperature with extended attributes.
> >> >> >> It seems that only ext4 has the concept of extended attributes.
> >> >> >
> >> >> > All major filesystems have xattr support. They are used extensively
> >> >> > by the security and integrity subsystems, for example.
> >> >> got it, thanks.
> >> >> >
> >> >> > Saving the information might be something that is useful to certian
> >> >> > applications, but lets have the people that need that functionality
> >> >> > spell out their requirements before discussing how or what to
> >> >> > implement.  Indeed, discussion shoul dreally focus on getting the
> >> >> > core, in-memory infrastructure sorted out first before trying to
> >> >> > expand the functionality further...
> >> >> ah, but the latest patchset need some love from experienced FS guys:).......
> >> >
> >> > There is one other possible issue with saving the data into the
> >> > filesystem, which is that it may disturb what you are trying to measure.
> >> > Some filesystems (GFS2 is one) store data for small inodes in the same
> >> > block as the inode itself. So that means the accesses to the saved hot
> >> > tracking info may potentially affect the data access times too. Also
> >> > there is a very limited amount of space to expand the number of fields
> >> > in the inode, so xattr may be the only solution, depending on how much
> >> > data needs to be stored in each case.
> >> Very good analysis, two possible issues are very meanful, thanks.
> >> >
> >> > In the GFS2 case (I don't think it is unique in this) xattrs are stored
> >> > out of line and having to access them in every open means an extra block
> >> > read per inode, which again has performance implications.
> >> >
> >> > So that is not an insurmountable problem, but something to take into
> >> > account in selecting a solution,
> >> In summary, you look like preferring to xattr as its solution.
> >>
> >
> > Well, that depends on exactly how large the data to be stored is, and
> > other factors. It will add overhead to the storage/retrieval but at
> > least it is fairly generic (wrt on-disk format) so likely to be easier
> > to retrofit to existing filesystems.
> Do you have some idea with more details about how to retrofit to existing FS?:)

Well I think we've already covered the obvious ways...

> >
> > I suspect this may be one of those cases where there is no obvious right
> > answer and it is a case of selecting the least worst option, if that
> > makes sense?
> Then we can only check which solution is better via large scale
> performance test.

Indeed, and that will be to a certain extent fs dependent too,

Steve.


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/