The gfp_to_rank() call in the slab allocator severely impacts performance. Hence reduce it to the bone, keeping only what is needed to make the reserve work. [more AIM9 results go here] AIM9 test 2.6.21-rc5 2.6.21-rc5-slab1 CONFIG_SLAB_FAIR=y 54 tcp_test 2124.48 +/- 10.85 2137.43 +/- 9.22 12.95 55 udp_test 5204.43 +/- 45.13 5231.59 +/- 56.66 27.16 56 fifo_test 20991.42 +/- 46.71 19675.97 +/- 56.35 1315.44 57 stream_pipe 10024.16 +/- 119.88 9912.53 +/- 75.52 111.63 58 dgram_pipe 9460.18 +/- 119.50 9502.75 +/- 89.06 42.57 59 pipe_cpy 30719.81 +/- 117.01 27885.52 +/- 46.81 2834.28 2.6.21-rc5-slab2 CONFIG_SLAB_FAIR=y 54 tcp_test 2124.48 +/- 10.85 2122.80 +/- 4.70 1.68 55 udp_test 5204.43 +/- 45.13 5136.98 +/- 62.31 67.45 56 fifo_test 20991.42 +/- 46.71 19646.81 +/- 53.61 1344.60 57 stream_pipe 10024.16 +/- 119.88 9940.87 +/- 280.73 83.29 58 dgram_pipe 9460.18 +/- 119.50 9432.69 +/- 250.27 27.49 59 pipe_cpy 30719.81 +/- 117.01 27870.70 +/- 65.50 2849.10 Signed-off-by: Peter Zijlstra --- mm/internal.h | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) Index: linux-2.6-git/mm/internal.h =================================================================== --- linux-2.6-git.orig/mm/internal.h 2007-02-22 14:09:39.000000000 +0100 +++ linux-2.6-git/mm/internal.h 2007-02-22 14:24:34.000000000 +0100 @@ -105,9 +105,38 @@ static inline int alloc_flags_to_rank(in return rank; } -static inline int gfp_to_rank(gfp_t gfp_mask) +static __always_inline int gfp_to_rank(gfp_t gfp_mask) { - return alloc_flags_to_rank(gfp_to_alloc_flags(gfp_mask)); + /* + * Although correct this full version takes a ~3% performance hit + * on the network test in aim9. + * + * return alloc_flags_to_rank(gfp_to_alloc_flags(gfp_mask)); + * + * So we go cheat a little. We'll only focus on the correctness of + * rank 0. + */ + + if (likely(!(gfp_mask & __GFP_NOMEMALLOC))) { + if (gfp_mask & __GFP_EMERGENCY) + return 0; + else if (!in_irq() && (current->flags & PF_MEMALLOC)) + return 0; + /* + * We skip the TIF_MEMDIE test: + * + * if (!in_interrupt() && unlikely(test_thread_flag(TIF_MEMDIE))) + * return 0; + * + * this will force an alloc but since we are allowed the memory + * that will succeed. This will make this very rare occurence + * very expensive when under severe memory pressure, but it + * seems a valid tradeoff. + */ + } + + /* Cheat by lumping everybody else in rank 1. */ + return 1; } #endif -- - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/