martin@shambhala:~> date Do 21. Aug 13:27:49 CEST 2008 shambhala:~> cat /proc/version Linux version 2.6.26.2-tp42-toi-3.0-rc7a-xfs-ticket-patch (martin@shambala) (gcc version 4.3.1 (Debian 4.3.1-8) ) #1 PREEMPT Wed Aug 13 10:10:11 CEST 2008 shambhala:~> apt-show-versions | egrep "(btrfs|nilfs)" btrfs-modules-2.6.26.2-tp42-toi-3.0-rc7a-xfs-ticket-patch 0.15-1+1 installed: No available version in archive btrfs-source/lenny uptodate 0.15-1 btrfs-tools/lenny uptodate 0.15-2 nilfs2-modules-2.6.26.2-tp42-toi-3.0-rc7a-xfs-ticket-patch 2.0.4-1+1 installed: No available version in archive nilfs2-source/sid uptodate 2.0.4-1 nilfs2-tools/lenny uptodate 2.0.5-1 shambhala:~> cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 13 model name : Intel(R) Pentium(R) M processor 1.80GHz stepping : 6 cpu MHz : 600.000 cache size : 2048 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 2 wp : yes flags : fpu vme de pse tsc msr mce cx8 sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss tm pbe bts est tm2 bogomips : 1197.54 clflush size : 64 power management: martin@shambhala:~> cat /proc/mounts | tail -4 /dev/mapper/shambala-ext3 /mnt/zeit-ext3 ext3 rw,errors=continue,data=ordered 0 0 /dev/mapper/shambala-nilfs /mnt/zeit-nilfs2 nilfs2 rw 0 0 /dev/mapper/shambala-btrfs /mnt/zeit-btrfs btrfs rw 0 0 /dev/mapper/shambala-xfs /mnt/zeit-xfs xfs rw,attr2,nobarrier,logbufs=8,logbsize=256k,noquota 0 0 martin@shambhala:~> df -hT | tail -8 /dev/mapper/shambala-ext3 ext3 4,0G 137M 3,7G 4% /mnt/zeit-ext3 /dev/mapper/shambala-nilfs nilfs2 4,0G 16M 3,8G 1% /mnt/zeit-nilfs2 /dev/mapper/shambala-btrfs btrfs 4,0G 40K 4,0G 1% /mnt/zeit-btrfs /dev/mapper/shambala-xfs xfs 4,0G 4,2M 4,0G 1% /mnt/zeit-xfs shambhala:~> xfs_info /mnt/zeit-xfs meta-data=/dev/mapper/shambala-xfs isize=256 agcount=4, agsize=262144 blks = sectsz=512 attr=2 data = bsize=4096 blocks=1048576, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 log =internal bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=0 realtime =none extsz=4096 blocks=0, rtextents=0 martin@shambhala:~> cat /sys/block/sda/queue/scheduler noop anticipatory deadline [cfq] XFS without barriers, since device mapper doesn't support barrier requests (http://bugzilla.kernel.org/show_bug.cgi?id=9554): shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/ compilebench/compilebench-0.6> ./compilebench -D /mnt/zeit-xfs -i 5 -r 10 using working directory /mnt/zeit-xfs, 5 intial dirs 10 runs native unpatched native-0 222MB in 25.88 seconds (8.59 MB/s) native patched native-0 109MB in 5.82 seconds (18.84 MB/s) native patched compiled native-0 691MB in 33.69 seconds (20.53 MB/s) create dir kernel-0 222MB in 20.38 seconds (10.91 MB/s) create dir kernel-1 222MB in 27.27 seconds (8.15 MB/s) create dir kernel-2 222MB in 26.69 seconds (8.33 MB/s) create dir kernel-3 222MB in 25.17 seconds (8.83 MB/s) create dir kernel-4 222MB in 29.52 seconds (7.53 MB/s) patch dir kernel-2 109MB in 38.54 seconds (2.85 MB/s) compile dir kernel-2 691MB in 41.60 seconds (16.62 MB/s) compile dir kernel-4 680MB in 49.46 seconds (13.76 MB/s) patch dir kernel-4 691MB in 118.19 seconds (5.85 MB/s) read dir kernel-4 in 77.09 11.89 MB/s read dir kernel-3 in 30.91 7.19 MB/s create dir kernel-3116 222MB in 42.73 seconds (5.20 MB/s) clean kernel-4 691MB in 6.48 seconds (106.73 MB/s) read dir kernel-1 in 32.08 6.93 MB/s stat dir kernel-0 in 6.94 seconds run complete: ======================================================================== == intial create total runs 5 avg 8.75 MB/s (user 2.05s sys 3.72s) create total runs 1 avg 5.20 MB/s (user 2.40s sys 5.34s) patch total runs 2 avg 4.35 MB/s (user 0.83s sys 3.93s) compile total runs 2 avg 15.19 MB/s (user 0.56s sys 2.90s) clean total runs 1 avg 106.73 MB/s (user 0.07s sys 0.40s) read tree total runs 2 avg 7.06 MB/s (user 1.93s sys 3.94s) read compiled tree total runs 1 avg 11.89 MB/s (user 2.29s sys 6.22s) no runs for delete tree no runs for delete compiled tree stat tree total runs 1 avg 6.94 seconds (user 1.13s sys 0.94s) no runs for stat compiled tree With barriers on an already heavily populated filesystem - I don't have an empty one on a raw partition at hand at the moment and I for sure won't empty this one: martin@shambhala:~> df -hT | grep /home /dev/sda5 xfs 112G 104G 8,2G 93% /home shambhala:~> df -hiT | grep /home /dev/sda5 xfs 34M 751K 33M 3% /home shambhala:~> xfs_db -rx /dev/sda5 xfs_db> frag actual 726986, ideal 703687, fragmentation factor 3.20% xfs_db> quit shambhala:~> martin@shambhala:~> cat /proc/mounts | grep "/home " /dev/sda5 /home xfs rw,relatime,attr2,logbufs=8,logbsize=256k,noquota 0 0 shambhala:~> xfs_info /home meta-data=/dev/sda5 isize=256 agcount=6, agsize=4883256 blks = sectsz=512 attr=2 data = bsize=4096 blocks=29299536, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 log =internal bsize=4096 blocks=32768, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/ compilebench/compilebench-0.6> ./compilebench -D /home/martin/Zeit/compilebench -i 5 -r 10 using working directory /home/martin/Zeit/compilebench, 5 intial dirs 10 runs native unpatched native-0 222MB in 117.37 seconds (1.89 MB/s) native patched native-0 109MB in 27.46 seconds (3.99 MB/s) native patched compiled native-0 691MB in 48.03 seconds (14.40 MB/s) create dir kernel-0 222MB in 83.55 seconds (2.66 MB/s) create dir kernel-1 222MB in 86.01 seconds (2.59 MB/s) create dir kernel-2 222MB in 71.61 seconds (3.11 MB/s) create dir kernel-3 222MB in 71.73 seconds (3.10 MB/s) create dir kernel-4 222MB in 61.61 seconds (3.61 MB/s) patch dir kernel-2 109MB in 63.14 seconds (1.74 MB/s) compile dir kernel-2 691MB in 45.61 seconds (15.16 MB/s) compile dir kernel-4 680MB in 50.13 seconds (13.58 MB/s) patch dir kernel-4 691MB in 154.38 seconds (4.48 MB/s) read dir kernel-4 in 95.04 9.65 MB/s read dir kernel-3 in 49.49 4.49 MB/s create dir kernel-3116 222MB in 79.44 seconds (2.80 MB/s) clean kernel-4 691MB in 8.64 seconds (80.05 MB/s) read dir kernel-1 in 71.40 3.11 MB/s stat dir kernel-0 in 14.44 seconds run complete: ======================================================================== == intial create total runs 5 avg 3.01 MB/s (user 2.34s sys 4.30s) create total runs 1 avg 2.80 MB/s (user 2.36s sys 4.12s) patch total runs 2 avg 3.11 MB/s (user 0.91s sys 4.07s) compile total runs 2 avg 14.37 MB/s (user 0.60s sys 2.76s) clean total runs 1 avg 80.05 MB/s (user 0.09s sys 0.45s) read tree total runs 2 avg 3.80 MB/s (user 2.00s sys 4.05s) read compiled tree total runs 1 avg 9.65 MB/s (user 2.36s sys 6.42s) no runs for delete tree no runs for delete compiled tree stat tree total runs 1 avg 14.44 seconds (user 1.17s sys 1.07s) no runs for stat compiled tree shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/ compilebench/compilebench-0.6> rm -rf /home/martin/Zeit/compilebench I didn't measure it, but it took *ages* while rm -rf was mostly in D state. According to harddisk noise a lot of seeks where involved. vmstat 1 during the rm -rf: 0 0 2784 748048 20 247160 0 0 160 4628 352 1224 15 14 71 0 0 0 2784 748056 20 247308 0 0 148 3848 298 442 11 10 79 0 0 0 2784 747996 20 247428 0 0 120 3377 260 449 9 9 82 0 0 0 2784 747764 20 247580 0 0 152 4364 324 1094 20 10 70 0 1 0 2784 747452 20 247736 0 0 156 4356 279 814 15 11 74 0 0 0 2784 747408 20 247900 0 0 164 4112 360 1131 13 13 74 0 0 0 2784 747136 20 248064 0 0 164 5128 318 855 16 10 74 0 0 0 2784 746780 20 248208 0 0 144 4353 305 1066 20 12 68 0 0 0 2784 746204 20 248336 0 0 128 5388 275 966 14 11 75 0 1 0 2784 748352 20 248468 0 0 132 5384 314 1234 22 11 67 0 0 0 2784 748104 20 248604 0 0 136 4873 284 807 16 11 73 0 Same game on same productively used partition, but now without barriers: shambhala:~> mount -o remount,nobarrier /home shambhala:~> cat /proc/mounts | grep "/home " /dev/sda5 /home xfs rw,relatime,attr2,nobarrier,logbufs=8,logbsize=256k,noquota 0 0 shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/ compilebench/compilebench-0.6> mkdir /home/martin/Zeit/compilebench shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/ compilebench/compilebench-0.6> ./compilebench -D /home/martin/Zeit/compilebench -i 5 -r 10 using working directory /home/martin/Zeit/compilebench, 5 intial dirs 10 runs native unpatched native-0 222MB in 51.44 seconds (4.32 MB/s) native patched native-0 109MB in 12.69 seconds (8.64 MB/s) native patched compiled native-0 691MB in 51.75 seconds (13.36 MB/s) create dir kernel-0 222MB in 47.64 seconds (4.67 MB/s) create dir kernel-1 222MB in 53.40 seconds (4.16 MB/s) create dir kernel-2 222MB in 48.04 seconds (4.63 MB/s) create dir kernel-3 222MB in 38.26 seconds (5.81 MB/s) create dir kernel-4 222MB in 34.15 seconds (6.51 MB/s) patch dir kernel-2 109MB in 50.61 seconds (2.17 MB/s) compile dir kernel-2 691MB in 37.94 seconds (18.23 MB/s) compile dir kernel-4 680MB in 45.32 seconds (15.02 MB/s) patch dir kernel-4 691MB in 107.27 seconds (6.45 MB/s) read dir kernel-4 in 82.18 11.16 MB/s read dir kernel-3 in 42.35 5.25 MB/s create dir kernel-3116 222MB in 38.27 seconds (5.81 MB/s) clean kernel-4 691MB in 5.92 seconds (116.82 MB/s) read dir kernel-1 in 73.63 3.02 MB/s stat dir kernel-0 in 13.77 seconds run complete: ======================================================================== == intial create total runs 5 avg 5.16 MB/s (user 2.21s sys 4.23s) create total runs 1 avg 5.81 MB/s (user 2.18s sys 4.89s) patch total runs 2 avg 4.31 MB/s (user 0.90s sys 4.05s) compile total runs 2 avg 16.62 MB/s (user 0.59s sys 3.05s) clean total runs 1 avg 116.82 MB/s (user 0.09s sys 0.41s) read tree total runs 2 avg 4.14 MB/s (user 1.90s sys 4.02s) read compiled tree total runs 1 avg 11.16 MB/s (user 2.28s sys 6.36s) no runs for delete tree no runs for delete compiled tree stat tree total runs 1 avg 13.77 seconds (user 1.19s sys 1.01s) no runs for stat compiled tree Not as fast as on the clean XFS LV, but still almost everytime almost twice as fast as with barriers. shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/ compilebench/compilebench-0.6> time rm -rf /home/martin/Zeit/compilebench rm -rf /home/martin/Zeit/compilebench 0,32s user 19,19s system 15% cpu 2:09,79 total This is definately faster than before. I didn't measure exact time on first occasion, but it took ages. vmstat 1 during the rm -rf indicated much higher metadata throughput: 3 0 2780 827696 20 162492 0 0 280 11109 449 865 31 15 52 2 0 0 2780 827304 20 162816 0 0 324 6656 468 1009 57 8 7 28 2 0 2636 828992 20 163364 0 0 540 5317 350 545 30 10 30 31 2 1 2636 837488 20 164020 0 0 656 7691 394 650 39 12 0 49 0 0 2224 960360 20 164516 0 0 496 12060 420 549 13 26 56 5 0 0 2224 959988 20 164904 0 0 388 13704 425 792 16 23 61 0 0 0 2224 959864 20 165128 0 0 224 6209 363 503 12 10 78 0 1 0 2224 959376 20 165540 0 0 412 14886 392 513 12 22 66 0 Now with barriers again, but with "noop" as scheduler: shambhala:~> mount -o remount,barrier /home shambhala:~> cat /proc/mounts | grep /home /dev/sda5 /home xfs rw,relatime,attr2,logbufs=8,logbsize=256k,noquota 0 0 shambhala:~> echo "noop" >/sys/block/sda/queue/scheduler shambhala:~> cat /sys/block/sda/queue/scheduler [noop] anticipatory deadline cfq shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/ compilebench/compilebench-0.6> mkdir /home/martin/Zeit/compilebench shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/ compilebench/compilebench-0.6> ./compilebench -D /home/martin/Zeit/compilebench -i 5 -r 10 using working directory /home/martin/Zeit/compilebench, 5 intial dirs 10 runs native unpatched native-0 222MB in 97.42 seconds (2.28 MB/s) native patched native-0 109MB in 20.72 seconds (5.29 MB/s) native patched compiled native-0 691MB in 46.37 seconds (14.91 MB/s) create dir kernel-0 222MB in 84.12 seconds (2.64 MB/s) create dir kernel-1 222MB in 95.18 seconds (2.34 MB/s) create dir kernel-2 222MB in 74.57 seconds (2.98 MB/s) create dir kernel-3 222MB in 71.81 seconds (3.10 MB/s) create dir kernel-4 222MB in 64.77 seconds (3.43 MB/s) patch dir kernel-2 109MB in 81.22 seconds (1.35 MB/s) compile dir kernel-2 691MB in 41.87 seconds (16.52 MB/s) compile dir kernel-4 680MB in 50.35 seconds (13.52 MB/s) patch dir kernel-4 691MB in 151.03 seconds (4.58 MB/s) read dir kernel-4 in 82.83 11.07 MB/s read dir kernel-3 in 48.49 4.59 MB/s create dir kernel-3116 222MB in 79.43 seconds (2.80 MB/s) clean kernel-4 691MB in 15.51 seconds (44.59 MB/s) read dir kernel-1 in 75.36 2.95 MB/s stat dir kernel-0 in 14.65 seconds run complete: ======================================================================== == intial create total runs 5 avg 2.90 MB/s (user 2.35s sys 4.56s) create total runs 1 avg 2.80 MB/s (user 2.18s sys 3.92s) patch total runs 2 avg 2.96 MB/s (user 0.87s sys 4.07s) compile total runs 2 avg 15.02 MB/s (user 0.60s sys 2.73s) clean total runs 1 avg 44.59 MB/s (user 0.07s sys 0.44s) read tree total runs 2 avg 3.77 MB/s (user 2.03s sys 3.82s) read compiled tree total runs 1 avg 11.07 MB/s (user 2.29s sys 6.24s) no runs for delete tree no runs for delete compiled tree stat tree total runs 1 avg 14.65 seconds (user 1.12s sys 1.00s) no runs for stat compiled tree Some tests run a bit faster, but on cost of responsiveness to out of line I/Os (opening a new webpage in Konqueror). Some do not run faster at all. Seems that write barriers on/off make the bigger difference here. As last XFS thing: vmstat 1 during a rm -rf while switching of XFS from nobarrier to barrier: 0 0 1976 422236 1784 516840 0 0 508 17160 410 540 7 23 70 0 1 0 1976 420624 1784 517576 0 0 736 26904 539 1032 14 35 51 0 0 0 1976 419176 1784 518152 0 0 576 23842 486 1060 17 33 50 0 0 0 1976 418316 1784 518460 0 0 308 12812 317 552 6 18 76 0 2 0 1976 417392 1784 518776 0 0 316 16689 360 882 2 23 75 0 8 0 1976 432948 1784 519252 0 0 476 16710 452 630 8 39 53 0 0 0 1976 432892 1784 519392 0 0 140 4146 371 1564 14 26 60 0 0 0 1976 432628 1784 519572 0 0 180 3844 340 660 11 10 79 0 0 0 1976 432496 1784 519736 0 0 164 3852 328 534 9 8 83 0 0 0 1976 432372 1784 519920 0 0 176 4100 359 788 19 11 70 0 Its obvious, where it was switched to barrier ;) Now the other filesystems with CFQ enabled. Ext3: shambhala:~> echo "cfq" >/sys/block/sda/queue/scheduler shambhala:~> cat /sys/block/sda/queue/scheduler noop anticipatory deadline [cfq] shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/ compilebench/compilebench-0.6> ./compilebench -D /mnt/zeit-ext3 -i 5 -r 10 using working directory /mnt/zeit-ext3, 5 intial dirs 10 runs native unpatched native-0 222MB in 16.90 seconds (13.16 MB/s) native patched native-0 109MB in 4.63 seconds (23.69 MB/s) native patched compiled native-0 691MB in 39.78 seconds (17.39 MB/s) create dir kernel-0 222MB in 12.24 seconds (18.17 MB/s) create dir kernel-1 222MB in 16.71 seconds (13.31 MB/s) create dir kernel-2 222MB in 18.50 seconds (12.02 MB/s) create dir kernel-3 222MB in 18.25 seconds (12.18 MB/s) create dir kernel-4 222MB in 27.24 seconds (8.16 MB/s) patch dir kernel-2 109MB in 29.26 seconds (3.75 MB/s) compile dir kernel-2 691MB in 53.41 seconds (12.95 MB/s) compile dir kernel-4 680MB in 55.24 seconds (12.32 MB/s) patch dir kernel-4 691MB in 108.66 seconds (6.36 MB/s) read dir kernel-4 in 79.38 11.55 MB/s read dir kernel-3 in 21.65 10.27 MB/s create dir kernel-3116 222MB in 28.22 seconds (7.88 MB/s) clean kernel-4 691MB in 17.05 seconds (40.56 MB/s) read dir kernel-1 in 23.67 9.39 MB/s stat dir kernel-0 in 9.63 seconds run complete: ======================================================================== == intial create total runs 5 avg 12.77 MB/s (user 1.96s sys 3.24s) create total runs 1 avg 7.88 MB/s (user 1.57s sys 2.39s) patch total runs 2 avg 5.06 MB/s (user 0.78s sys 3.92s) compile total runs 2 avg 12.64 MB/s (user 0.54s sys 3.75s) clean total runs 1 avg 40.56 MB/s (user 0.08s sys 0.36s) read tree total runs 2 avg 9.83 MB/s (user 1.82s sys 4.32s) read compiled tree total runs 1 avg 11.55 MB/s (user 2.32s sys 7.02s) no runs for delete tree no runs for delete compiled tree stat tree total runs 1 avg 9.63 seconds (user 1.11s sys 0.89s) no runs for stat compiled tree nilfs2: shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/ compilebench/compilebench-0.6> ./compilebench -D /mnt/zeit-nilfs2 -i 5 -r 10 using working directory /mnt/zeit-nilfs2, 5 intial dirs 10 runs native unpatched native-0 222MB in 20.28 seconds (10.97 MB/s) native patched native-0 109MB in 8.83 seconds (12.42 MB/s) native patched compiled native-0 691MB in 42.44 seconds (16.30 MB/s) create dir kernel-0 222MB in 20.89 seconds (10.65 MB/s) create dir kernel-1 222MB in 21.13 seconds (10.52 MB/s) create dir kernel-2 222MB in 20.22 seconds (11.00 MB/s) create dir kernel-3 222MB in 21.60 seconds (10.30 MB/s) create dir kernel-4 222MB in 20.63 seconds (10.78 MB/s) patch dir kernel-2 109MB in 20.97 seconds (5.23 MB/s) compile dir kernel-2 691MB in 44.40 seconds (15.58 MB/s) Traceback (most recent call last): File "./compilebench", line 631, in total_runs += func(dset, rnd) File "./compilebench", line 368, in compile_one_dir mbs = run_directory(ch[0], dir, "compile dir") File "./compilebench", line 241, in run_directory fp.write(buf[:cur]) IOError: [Errno 28] No space left on device Okay, possibly due to those 11 checkpoints it stored. Seems I would need more than 4 GB for the test to complete. But enough testing for today ;). btrfs 0.15: shambhala:/home/martin/Linux/Dateisysteme/Performance-Messung/ compilebench/compilebench-0.6> ./compilebench -D /mnt/zeit-btrfs -i 5 -r 10 using working directory /mnt/zeit-btrfs, 5 intial dirs 10 runs native unpatched native-0 222MB in 13.61 seconds (16.34 MB/s) native patched native-0 109MB in 3.12 seconds (35.15 MB/s) native patched compiled native-0 691MB in 28.84 seconds (23.98 MB/s) create dir kernel-0 222MB in 10.99 seconds (20.23 MB/s) create dir kernel-1 222MB in 13.95 seconds (15.94 MB/s) create dir kernel-2 222MB in 14.99 seconds (14.83 MB/s) create dir kernel-3 222MB in 15.00 seconds (14.82 MB/s) create dir kernel-4 222MB in 16.16 seconds (13.76 MB/s) patch dir kernel-2 109MB in 30.09 seconds (3.64 MB/s) compile dir kernel-2 691MB in 58.05 seconds (11.91 MB/s) compile dir kernel-4 680MB in 55.23 seconds (12.32 MB/s) patch dir kernel-4 691MB in 134.20 seconds (5.15 MB/s) read dir kernel-4 in 108.58 8.44 MB/s read dir kernel-3 in 43.47 5.12 MB/s create dir kernel-3116 222MB in 27.81 seconds (8.00 MB/s) clean kernel-4 691MB in 17.63 seconds (39.23 MB/s) read dir kernel-1 in 70.31 3.16 MB/s stat dir kernel-0 in 32.85 seconds run complete: ======================================================================== == intial create total runs 5 avg 15.92 MB/s (user 1.06s sys 5.43s) create total runs 1 avg 8.00 MB/s (user 1.17s sys 7.41s) patch total runs 2 avg 4.40 MB/s (user 0.88s sys 10.55s) compile total runs 2 avg 12.12 MB/s (user 0.56s sys 5.34s) clean total runs 1 avg 39.23 MB/s (user 0.05s sys 2.30s) read tree total runs 2 avg 4.14 MB/s (user 1.85s sys 10.00s) read compiled tree total runs 1 avg 8.44 MB/s (user 2.19s sys 16.50s) no runs for delete tree no runs for delete compiled tree stat tree total runs 1 avg 32.85 seconds (user 1.01s sys 3.35s) no runs for stat compiled tree