lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAEH94LidoKSWwBnc2SV4_9hrEf7c1M3ROUbA+1FCjsZ4rqYGKQ@mail.gmail.com>
Date:	Fri, 22 Feb 2013 08:32:10 +0800
From:	Zhi Yong Wu <zwu.kernel@...il.com>
To:	linux-fsdevel@...r.kernel.org
Cc:	linux-kernel@...r.kernel.org, viro@...iv.linux.org.uk,
	david@...morbit.com, dave@...os.cz, darrick.wong@...cle.com,
	andi@...stfloor.org, hch@...radead.org,
	linuxram@...ux.vnet.ibm.com, zwu.kernel@...il.com,
	wuzhy@...ux.vnet.ibm.com
Subject: Re: [PATCH RESEND v1 00/16] vfs: hot data tracking

ping? any comments?

On Thu, Dec 20, 2012 at 10:43 PM,  <zwu.kernel@...il.com> wrote:
> From: Zhi Yong Wu <wuzhy@...ux.vnet.ibm.com>
>
> HI, guys,
>
>   This patchset has been done scalability or performance tests
> by fs_mark, ffsb and compilebench.
>   I have done the perf testing on Linux 3.7.0-rc8+ with Intel(R) Core(TM)
> i7-3770 CPU @ 3.40GHz with 8 CPUs, 16G ram and 260G disk.
>
>   Any comments or ideas are appreciated, thanks.
>
> NOTE:
>
>   The patchset can be obtained via my kernel dev git on github:
> git://github.com/wuzhy/kernel.git hot_tracking
>   If you're interested, you can also review them via
> https://github.com/wuzhy/kernel/commits/hot_tracking
>
>   For more info, please check hot_tracking.txt in Documentation
>
> Below is the perf testing report:
>
> 1. fs_mark test
>
> w/o: without hot tracking
> w/ : with hot tracking
>
>   Count     Size      FSUse%               Files/sec                     App Overhead
>
>                     w/o    w/           w/o          w/                   w/o        w/
>
>   800000     1       2      3          13756.4     32144.9              5350627    5436291
>  1600000     1       4      5           1163.4      1799.3             20848119   21708216
>  2400000     1       6      6           1360.8      1252.5              6798705    8715322
>  3200000     1       8      8           1600.1      1196.3              5751129    6013792
>  4000000     1       9      9           1071.4      1191.2             17204725   26786369
>  4800000     1      10     10           1483.5      1447.9             19555541    8383046
>  5600000     1      11     11           1457.9      1699.5              5783588   10074681
>  6400000     1      12     13           1658.8      1628.5              6992697    6185551
>  7200000     1      14     14           1662.4      1857.1              5796793   13772592
>  8000000     1      15     15           2930.0      2653.8             12431682    6152573
>  8800000     1      16     17           1630.8      1665.0              7666719   13682765
>  9600000     1      18     18           1530.3      1583.9              5823644   10171644
> 10400000     1      19     19           1437.9      1798.6             20935224    6048083
> 11200000     1      20     20           1529.0      1550.6              6647450    6003151
> 12000000     1      21     22           1558.6      1501.8             12539509   18144939
> 12800000     1      23     23           1644.2      1432.1              7074419   28101975
> 13600000     1      24     24           1753.6      1650.2              7164297   20888972
> 14400000     1      25     25           2750.0      1483.9             12756692    7441225
> 15200000     1      27     27           1551.1      1514.3              5741066    8250443
> 16000000     1      28     28           1610.8      1635.9             72193860    8545285
> 16800000     1      29     29           1646.7      1907.7              8945856   11703513
> 17600000     1      30     31           1496.6      2722.3              5858961    8989393
> 18400000     1      32     32           1457.7      1565.7             10914475   26504660
> 19200000     1      33     33           1437.6      1518.7              6708975  213303618
> 20000000     1      34     34           1825.4      1521.1              5722086   12490907
> 20800000     1      36     35           1718.4      1611.5              5873290   17942534
> 21600000     1      37     37           2152.6      1536.9            113050627    8717940
> 22400000     1      38     38           2443.7      1788.2              7398122   19834765
> 23200000     1      39     39           1518.5      1587.6              5770959   10134882
> 24000000     1      41     41           1536.8      2164.0              5751248    7214626
> 24800000     1      42     42           1576.6      2939.4              7390314    6070271
> 25600000     1      43     43           1707.4      1535.9             11075939    6052896
> 26400000     1      44     44           1522.5      1563.1             10142987   22549898
> 27200000     1      46     46           1827.4      1608.5             11613016   24828125
> 28000000     1      47     47           3420.5      1741.9              8059985   16599156
> 28800000     1      48     48           1815.5      1944.4              7847931    9043277
> 29600000     1      50     49           1650.0      1596.6              5636323    7929164
> 30400000     1      51     51           1683.7      1573.3              5766323   19369146
> 31200000     1      52     52           1610.1      1669.8              9256111    9899107
> 32000000     1      53     53           1645.2      3081.0              7855010    6057257
> 32800000     1      54     55           1835.3      3122.0              6899141    6143875
> 33600000     1      56     56           1916.8      1734.8             10271967    6049509
> 34400000     1      57     57           3119.2      1630.8             11503274   13975417
> 35200000     1      58     58           1629.2      1695.7              6827225    6214248
> 36000000     1      60     60           1636.5      1695.4             38077664   16211067
> 36800000     1      61     61           1665.2      2069.1             19948817    9358494
> 37600000     1      62     62           1734.5      1931.5             26487196    8954836
> 38400000     1      63     63           1625.8      1654.0              6649289    9131844
> 39200000     1      65     65           1778.4      1663.3             11653376    7144960
> 40000000     1      66     66           1851.0      1935.6              8164470   11288753
> 40800000     1      67     67           3171.0      3431.6             12358380    6072820
> 41600000     1      69     69           1714.3      1954.3             13765035    9364495
> 42400000     1      70     70           1591.0      1681.8             18733304    7407689
> 43200000     1      71     71           1537.2      1642.8             19534908    6163018
> 44000000     1      72     72           1630.3      1641.2             23479883   10967509
> 44800000     1      74     74           1877.5      1651.9              8174965    9484587
> 45600000     1      75     75           3322.4      1653.6             14740938    7497831
> 46400000     1      76     76           1706.9      1840.6             10348550   23296562
> 47200000     1      77     78           1837.7      2515.3             13917543   14683192
> 48000000     1      79     79           1642.6      2368.6             14365759    6080942
> 48800000     1      80     80           1827.1      1655.2              9234312    7412406
> 49600000     1      81     81           1631.0      1858.7              7543970   18610881
> 50400000     1      82     82           1560.5      1865.0             21374219    6598771
>
>
> From the above table, when the same count files with same size are created, how FS is full is
> basically same.
>
> 2. FFSB test
>                               w/o hot tracking         w/ hot tracking           ratio
>                                        v1                     v2               (v2-v1)/v1
> large_file_create
>         1 thread
>            - Trans/sec              28918.75               29014.48               +0.33%
>            - Throughput             113MB/sec              113MB/sec              +0.0%
>            - %CPU                     4.8%                    5.1%                +6.3%
>            - Trans/%CPU             602473.96              568911.37              -5.6%
>         8 threads
>            - Trans/sec              28480.37               28541.25               +0.2%
>            - Throughput             111MB/sec              111MB/sec              +0.0%
>            - %CPU                     5.6%                    5.9%                +5.4%
>            - Trans/%CPU             508578.04               483750                -4.9%
>         32 threads
>            - Trans/sec              25011.86               26992.32               +7.9%
>            - Throughput             97.7MB/sec             105MB/sec              +7.5%
>            - %CPU                     6.2%                    7.1%                +14.8%
>            - Trans/%CPU             403417.10              380173.52              -5.8%
>
> large_file_seq_read
>         1 thread
>            - Trans/sec              35303.23               34838.02               -1.3%
>            - Throughput             138MB/sec              136MB/sec              -1.4%
>            - %CPU                     5.4%                    5.4%                +0.0%
>            - Trans/%CPU             653763.52              645148.52              -1.3%
>         8 threads
>            - Trans/sec              11902.82               11205.22               -5.9%
>            - Throughput             46.5MB/sec             43.8MB/sec             -5.8%
>            - %CPU                     2.1%                    2.0%                -4.8%
>            - Trans/%CPU             566800.95                560261               -1.2%
>         32 threads
>            - Trans/sec               5068.48                5316.36               +4.9%
>            - Throughput             19.8MB/sec             20.8MB/sec             +5.1%
>            - %CPU                     0.9%                    1.0%                +11.1%
>            - Trans/%CPU             563164.45                531636               -5.6%
>
> random_write
>         1 thread
>            - Trans/sec                729.01                 738.89               +1.4%
>            - Throughput             99.7MB/sec              101MB/sec             +1.3%
>            - %CPU                     0.1%                    0.1%                +0.0%
>            - Trans/%CPU               72901                  73889                +1.4%
>         8 threads
>            - Trans/sec               714.56                  714.57               +0.0%
>            - Throughput            97.7MB/sec               97.7MB/sec            +0.0%
>            - %CPU                     0.2%                    0.2%                +0.0%
>            - Trans/%CPU               35728                  35728.5              +0.0%
>         32 threads
>            - Trans/sec               698.62                  692.59               -0.9%
>            - Throughput            95.5MB/sec               94.7MB/sec            -0.8%
>            - %CPU                     0.2%                    0.2%                +0.0%
>            - Trans/%CPU               34931                 34629.5               -0.9%
>
> random_read
>         1 thread
>            - Trans/sec               225.49                 227.03                +0.7%
>            - Throughput             902KB/sec               908KB/sec             +0.7%
>            - %CPU                     1.1%                    1.1%                +0.0%
>            - Trans/%CPU             20499.10                20639.10              +0.7%
>         8 threads
>            - Trans/sec               106.72                  105.76               -0.9%
>            - Throughput             427KB/sec               423KB/sec             -0.9%
>            - %CPU                     0.5%                    0.5%                +0.0%
>            - Trans/%CPU              2134.4                  2115.2               -0.9%
>         32 threads
>            - Trans/sec               107.44                  108.26               +0.8%
>            - Throughput             430KB/sec               433KB/sec             +0.7%
>            - %CPU                     0.5%                    0.5%                +0.0%
>            - Trans/%CPU              2148.8                  2165.2               +0.8%
>
> mail_server
>         1 thread
>            - Trans/sec               681.67                  732.66               +7.5%
>            - Throughput [read]      1.77MB/sec              1.99MB/sec            +12.4%
>            - Throughput [write]     858KB/sec               887KB/sec             +3.4%
>            - %CPU                     0.6%                    0.6%                +0.0%
>            - Trans/%CPU             11361.17                 12211                +7.5%
>         8 threads
>            - Trans/sec               630.48                  597.08               -5.3%
>            - Throughput [read]      1.64MB/sec             1.54MB/sec             -6.1%
>            - Throughput [write]     814KB/sec              784KB/sec              -3.7%
>            - %CPU                     0.6%                    0.5%                -16.7%
>            - Trans/%CPU               10508                 11941.6               +13.6%
>         32 threads
>            - Trans/sec               598.68                 566.05                -5.5%
>            - Throughput [read]      1.53MB/sec             1.5MB/sec              -2.0%
>            - Throughput [write]     804KB/sec              705KB/sec              -12.3%
>            - %CPU                     0.7%                    0.6%                -14.2%
>            - Trans/%CPU              8552.57                9434.17               +10.3%
>
> 3. Compilebench test
>
>                         w/o hot tracking        w/ hot tracking            ratio
>                                v1                     v2                 (v2-v1)/v1
> intial create              114.81 MB/s             118.32 MB/s             +3.1%
> create                     11.98 MB/s              12.26 MB/s              +2.3%
> patch                      3.61 MB/s               3.66 MB/s               +1.4%
> compile                    46.40 MB/s              48.07 MB/s              +3.6%
> clean                      126.33 MB/s             128.75 MB/s             +1.9%
> read tree                  9.93 MB/s               9.71 MB/s               -2.2%
> read compiled tree         17.19 MB/s              17.52 MB/s              +1.9%
> delete tree                12.23 seconds           11.13 seconds           -9.0%
> delete compiled tree       12.98 seconds           16.05 seconds           +26.7%
> stat tree                  7.03 seconds            5.51 seconds            -21.6%
> stat compiled tree         12.19 seconds           9.06 seconds            -25.7%
>
> Changelog:
>
>  - Solved 64 bits inode number issue. [David Sterba]
>  - Embed struct hot_type in struct file_system_type [Darrick J. Wong]
>  - Cleanup Some issues [David Sterba]
>  - Use a static hot debugfs root [Greg KH]
>  - Rewritten debugfs support based on seq_file operation. [Dave Chinner]
>  - Refactored workqueue support. [Dave Chinner]
>  - Turn some Micro into be tunable   [Zhiyong, Zheng Liu]
>        TIME_TO_KICK, and HEAT_UPDATE_DELAY
>  - Introduce hot func registering framework [Zhiyong]
>  - Remove global variable for hot tracking [Zhiyong]
>  - Add xfs hot tracking support [Dave Chinner]
>  - Add ext4 hot tracking support [Zheng Liu]
>  - Cleanedup a lot of other issues [Dave Chinner]
>  - Added memory shrinker [Dave Chinner]
>  - Converted to one workqueue to update map info periodically [Dave Chinner]
>  - Cleanedup a lot of other issues [Dave Chinner]
>  - Reduce new files and put all in fs/hot_tracking.[ch] [Dave Chinner]
>  - Add btrfs hot tracking support [Zhiyong]
>  - The first three patches can probably just be flattened into one.
>                                         [Marco Stornelli , Dave Chinner]
>
> Zhi Yong Wu (16):
>   vfs: introduce some data structures
>   vfs: add init and cleanup functions
>   vfs: add I/O frequency update function
>   vfs: add two map arrays
>   vfs: add hooks to enable hot tracking
>   vfs: add temp calculation function
>   vfs: add map info update function
>   vfs: add aging function
>   vfs: add one work queue
>   vfs: add FS hot type support
>   vfs: register one shrinker
>   vfs: add one ioctl interface
>   vfs: add debugfs support
>   proc: add two hot_track proc files
>   btrfs: add hot tracking support
>   vfs: add documentation
>
>  Documentation/filesystems/00-INDEX         |    2 +
>  Documentation/filesystems/hot_tracking.txt |  255 ++++++
>  fs/Makefile                                |    2 +-
>  fs/btrfs/ctree.h                           |    1 +
>  fs/btrfs/super.c                           |   22 +-
>  fs/compat_ioctl.c                          |    5 +
>  fs/dcache.c                                |    2 +
>  fs/direct-io.c                             |    6 +
>  fs/hot_tracking.c                          | 1345 ++++++++++++++++++++++++++++
>  fs/hot_tracking.h                          |   52 ++
>  fs/ioctl.c                                 |   74 ++
>  include/linux/fs.h                         |    5 +
>  include/linux/hot_tracking.h               |  152 ++++
>  kernel/sysctl.c                            |   14 +
>  mm/filemap.c                               |    6 +
>  mm/page-writeback.c                        |   12 +
>  mm/readahead.c                             |    7 +
>  17 files changed, 1960 insertions(+), 2 deletions(-)
>  create mode 100644 Documentation/filesystems/hot_tracking.txt
>  create mode 100644 fs/hot_tracking.c
>  create mode 100644 fs/hot_tracking.h
>  create mode 100644 include/linux/hot_tracking.h
>
> --
> 1.7.6.5
>



-- 
Regards,

Zhi Yong Wu
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists