linux-kernel - Re: fadvise DONTNEED implementation (or lack thereof)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <20101109162525.BC87.A69D9226@jp.fujitsu.com>
Date:	Tue,  9 Nov 2010 16:28:02 +0900 (JST)
From:	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>
To:	Ben Gamari <bgamari.foss@...il.com>
Cc:	kosaki.motohiro@...fujitsu.com, linux-kernel@...r.kernel.org,
	rsync@...ts.samba.org, linux-mm@...ck.org
Subject: Re: fadvise DONTNEED implementation (or lack thereof)

> I've recently been trying to track down the root cause of my server's
> persistent issue of thrashing horribly after being left inactive. It
> seems that the issue is likely my nightly backup schedule (using rsync)
> which traverses my entire 50GB home directory. I was surprised to find
> that rsync does not use fadvise to notify the kernel of its use-once
> data usage pattern.
> 
> It looks like a patch[1] was written (although never merged, it seems)
> incorporating fadvise support, but I found its implementation rather
> odd, using mincore() and FADV_DONTNEED to kick out only regions brought
> in by rsync. It seemed to me the simpler and more appropriate solution
> would be to simply flag every touched file with FADV_NOREUSE and let the
> kernel manage automatically expelling used pages.
> 
> After looking deeper into the kernel implementation[2] of fadvise() the
> reason for using DONTNEED became more apparant. It seems that the kernel
> implements NOREUSE as a noop. A little googling revealed[3] that I not
> the first person to encounter this limitation. It looks like a few
> folks[4] have discussed addressing the issue in the past, but nothing
> has happened as of 2.6.36. Are there plans to implement this
> functionality in the near future? It seems like the utility of fadvise
> is severely limited by lacking support for NOREUSE.

btw, Other OSs seems to also don't implement it.
example,

http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libc/port/gen/posix_fadvise.c

     35 /*
     36  * SUSv3 - file advisory information
     37  *
     38  * This function does nothing, but that's OK because the
     39  * Posix specification doesn't require it to do anything
     40  * other than return appropriate error numbers.
     41  *
     42  * In the future, a file system dependent fadvise() or fcntl()
     43  * interface, similar to madvise(), should be developed to enable
     44  * the kernel to optimize I/O operations based on the given advice.
     45  */
     46 
     47 /* ARGSUSED1 */
     48 int
     49 posix_fadvise(int fd, off_t offset, off_t len, int advice)
     50 {
     51 	struct stat64 statb;
     52 
     53 	switch (advice) {
     54 	case POSIX_FADV_NORMAL:
     55 	case POSIX_FADV_RANDOM:
     56 	case POSIX_FADV_SEQUENTIAL:
     57 	case POSIX_FADV_WILLNEED:
     58 	case POSIX_FADV_DONTNEED:
     59 	case POSIX_FADV_NOREUSE:
     60 		break;
     61 	default:
     62 		return (EINVAL);
     63 	}
     64 	if (len < 0)
     65 		return (EINVAL);
     66 	if (fstat64(fd, &statb) != 0)
     67 		return (EBADF);
     68 	if (S_ISFIFO(statb.st_mode))
     69 		return (ESPIPE);
     70 	return (0);
     71 }


So, I don't think application developers will use fadvise() aggressively
because we don't have a cross platform agreement of a fadvice behavior.



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/