Aug 06 11:38:41 * Now talking on #zcache Aug 06 11:38:41 * cameron.freenode.net sets mode +n #zcache Aug 06 11:38:41 * cameron.freenode.net sets mode +s #zcache Aug 06 12:38:15 * sjennings (~sjennings@2001:470:1f0f:87d:e46c:7cd1:4974:640b) has joined #zcache Aug 06 12:49:12 * rcj (~rcjenn@32.97.110.59) has joined #zcache Aug 06 12:50:26 * djm1 (~djm1021@inet-hqmc03-o.oracle.com) has joined #zcache Aug 06 12:50:29 Good morning Konrad, thanks for suggesting this. Aug 06 12:50:32 * darnok (~konrad@209-6-85-33.c3-0.smr-ubr2.sbo-smr.ma.cable.rcn.com) has joined #zcache Aug 06 12:50:47 hi all, first time user for freenode so I had to register, sorry to be late Aug 06 12:50:53 (this is Dan Magenheimer) Aug 06 12:51:17 djm1, I think you can be just an unregister user? Aug 06 12:51:36 rcj, Sure thinkg Aug 06 12:51:51 * darnok goes away now that the proper Konrad is in place Aug 06 12:52:08 konrad/darnok: ok, we can deal with that later, for now, "call me djm1" Aug 06 12:52:22 Sure. So I think we are all here right? Minchan couldn't make it Aug 06 12:52:38 i don't know of anyone else that was planning to come **** ENDING LOGGING AT Mon Aug 6 12:52:39 2012 **** BEGIN LOGGING AT Mon Aug 6 12:52:39 2012 Aug 06 12:52:46 Sounds good Aug 06 12:52:59 * konrad nods. Aug 06 12:53:18 Let me first start by saying thank you for being able to make it. Aug 06 12:53:42 Sometimes it takes a whole bunch of iterations to setup a call but this worked out fine. Aug 06 12:53:46 * darnok has quit (Read error: Operation timed out) Aug 06 12:54:10 And also for the interest in zcache/tmem/frontswap/cleanpage/future ideas. Aug 06 12:55:11 I think we all want all the pieces completely out of staging (irregardlesss of how internally each company uses the technologies - we have been shipping it with our kernel - UEK2) Aug 06 12:55:55 We do want it out of staging, for us we can't use it until it's out so it's a bit more critical in that way. Aug 06 12:55:55 But for right now the zcache is on the plate since that can be used in numerous ways. Aug 06 12:56:09 * djm1 makes minor correction... we are shipping frontswap and cleancache and the xen shim in UEK2, not zcache Aug 06 12:56:21 djm1, Ah thats right. Aug 06 12:56:33 * konrad puts on his bigger TODO list to address that. Aug 06 12:57:07 rcj: right, we can't use it while its in staging either Aug 06 12:57:27 UEK2 did jump the gun on frontswap though Aug 06 12:57:48 rcj, With the 'can't use it' is it more of a .. support type question? Meaning that once it gets its equal to the being almost bug-free? Aug 06 12:58:19 I think "officially" anything in staging results in a tainted kernel, though I don't know how that applies to distros Aug 06 12:58:32 Trying to parse that last sentence konrad Aug 06 12:58:35 rcj, I am trying here to figure out if the "unstaging" part that is critical to you is in terms of code quality or some other criteria. Aug 06 12:59:31 * sashal (~sasha@95-89-78-76-dynip.superkabel.de) has joined #zcache Aug 06 12:59:33 rcj, As that leads directly to what is the upmost important in zcache for you guys Aug 06 12:59:41 the distros we work with only accept mainline drivers/code, not staging Aug 06 12:59:42 konrad, quality is a different discussion. Quality has to be there. Aug 06 12:59:50 Hey Sasha. Let me paste to you on a side-channel what we said so far. Aug 06 12:59:58 konrad, thanks Aug 06 13:00:20 sashal, Hopefully your client won't ban me for pasting too fast. Aug 06 13:00:32 sashal: welcome! Aug 06 13:00:38 * djm1 is dan.magenheimer Aug 06 13:01:05 my interest is using frontswap/cleancache/zcache for in-kernel memory compression. no xen, no kvm Aug 06 13:01:07 djm1, hey! Aug 06 13:01:10 sjennings, rcj : OK, so its the 'feature' needs to be in the mainline before they will accept turning it on. Somehow I thought staging would be part of it since Fedora has some bits turned for that. Aug 06 13:02:43 sjennings, rcj , djm1 : But didn't realize the tainting part of the code? Ah yes: 2564 add_taint_module(mod, TAINT_CRAP); Aug 06 13:03:05 yes, that looks bad in the dmesg of enterprise distros Aug 06 13:03:27 * konrad nods Aug 06 13:03:57 Then it comes down to: time-line and resources Aug 06 13:04:19 konrad: well, yes, and the small matter of technical solutions too ;-) Aug 06 13:04:28 i sent out the promotion patch because we find value in zcache as it is right now Aug 06 13:05:12 it is stable and, as both Dan and I have demostrated, has measurable benefits (I/O reduction and sometime speed) Aug 06 13:05:14 sjennings, OK. so the aim for getting it out in v3.6 is .. an optimistic hope not a real I-MUST-GET-IT-NOW type. Aug 06 13:05:18 sjennings, and there are other contributors that are in same situation where they are using zcache as it exists. Aug 06 13:06:46 well, it's for 3.7 now, and the promotion patch should either work or tell us what showstoppers remain the prevent promotion Aug 06 13:07:00 sjennings: two things... I recall Dave Hansen stating fairly clearly a year ago that zcache was not near suitable for promoting... is he happy? Aug 06 13:07:25 second, have you measured zcache/non-zcache results on a recent kernel? Aug 06 13:08:13 depends on how recent your talking about. i posted my zcache performance number on v3.4 i think Aug 06 13:08:30 i consider that recent Aug 06 13:09:35 * djm1 thinks it was 3.2, not sure, but Rik Van Riel made a lot of swap subsystem improvements in the last couple of kernels... I suggest rerunning at least a couple of benchmarks (especially something swap heavy) Aug 06 13:10:46 seems like that would only make zcache, in it currently form, more useful Aug 06 13:10:46 djm1, Nothing big in v3.4: konrad@phenom:~/ssd/linux$ git log --oneline v3.4..v3.5 mm/swapfile.c Aug 06 13:10:46 9b15b81 swap: fix shmem swapping when more than 8 areas Aug 06 13:10:46 a3fe778 Merge tag 'stable/frontswap.v16-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/konrad/mm Aug 06 13:10:46 4b91355 memcg: fix/change behavior of shared anon at moving task Aug 06 13:10:46 bde05d1 shmem: replace page if mapping excludes its zone Aug 06 13:10:46 38b5faf mm: frontswap: core swap subsystem hooks and headers Aug 06 13:12:20 djm1, So I think that is OK for v3.4 perf perspective? Aug 06 13:12:36 sjennings, These weren't benchmark with other patches on top? Like the WasActive one that Dan came up with? Aug 06 13:13:25 no, i used the mainline + frontswap (before it was mainlined in v3.5) Aug 06 13:13:43 djm1, If dhansen has specific issues he'll need to speak for them in this promotion process. Aug 06 13:13:54 sjennings: you ran your benchmarks before I was at LSF/MM and I think 3.3 was released that weekend Aug 06 13:14:07 https://lkml.org/lkml/2012/4/16/449 Aug 06 13:14:23 here's one of the Rik van Riel patches Aug 06 13:16:42 sjennings: after rik's patches went in, the "non-zcache" times dropped dramatically Aug 06 13:18:11 zcache's primary benefit for me is reduced I/O Aug 06 13:18:29 just because swap on rotational media is faster now, doesn't make zcache unneeded Aug 06 13:18:57 not all linux machines use rotational swap media (i.e. phones) Aug 06 13:18:58 sjennings: believe me, I am not arguing that zcache is not needed, just that we now have a lot more work/tuning to do Aug 06 13:19:42 djm1, but there will always be interactions that drive work. Aug 06 13:20:48 djm1, having zcache in staging or mainline is orthogonal. In fact, in mainline a patch that causes a regression for zcache might get more attention. Aug 06 13:23:15 rcj: sorry, I'm not sure I understand your point Aug 06 13:23:59 I am suggesting that the benchmarks Seth and I ran and posted, if rerun today on 3.4/3.5 would look horrible Aug 06 13:24:58 how is Rik's patchset going to reduce I/O like zcache does? Aug 06 13:25:05 (correction: some of them look horrible) Aug 06 13:25:26 if anything, it increases I/O be doing more aggressive readahead Aug 06 13:25:39 djm1, You brought up Rik's patch improving swap performance without zcache. You're saying that zcache needs to improve. That's not a zcache regression. The kernel is improving in the swap area for some cases, that's great. So then we take a TODO to worrk on zcache swap performance or elevate the priority of that work. it's not a blocker. Aug 06 13:25:47 sjennings: good question... I don't know for sure... but they dramatically increase performance Aug 06 13:26:34 rcj: are you saying that zcache is useful regardless of whether it increases performance? Aug 06 13:27:29 (or are you saying you have measured it on non-rotating and it DOES increase performance there, and so you don't care if whether it increases performance on rotating?) Aug 06 13:27:56 (measured on 3.4/3.5...) Aug 06 13:27:59 djm1, I'm saying that there are other measures as Seth is pointing out. Aug 06 13:28:02 djm1, Does it matter? The goal is to have limited amount of I/Os. Aug 06 13:28:55 * sjennings what konrad said Aug 06 13:28:55 djm1, well, let me redact that. .. and not have horrible performance . Aug 06 13:28:58 konrad,rcj: to me, reduced I/O and better performance are directly correlated, unless the bottleneck is the cpu of course Aug 06 13:30:01 sjennings, ok, then please rerun your benchmarks on 3.4 or 3.5, and look at whether zcache reduced I/Os vs non-zcache Aug 06 13:31:21 djm1, You are thinking is that it does not? Or that it does reduce I/O but the perf overall goes down? Aug 06 13:31:30 i can do that, but those numbers don't negate our point. this change in behavior isn't a bug in zcache Aug 06 13:32:16 sjennings: ok, then let's start over from objectives: I think you said you want to promote zcache because it reduces I/O Aug 06 13:33:37 I am agreeing that on 3.2 it did, but I have seen on 3.4 performance for non-zcache has improved dramatically, so I believe zcache no longer reduces I/O over non-zcache Aug 06 13:34:53 i'm i correct in understand rik's patch to make swap readahead more agressive? Aug 06 13:35:12 it MAY be the case that the number of I/Os has improved dramatically for non-zcache, but the number of pages read/written for zcache is better than non-zcache Aug 06 13:35:15 *am i Aug 06 13:36:30 sjennings: yes, that is one of rik's patches, there were several and I didn't look at all of them carefully Aug 06 13:37:47 we need to come back around on this though. we want promotion. what are the absolute showstoppers to prevent this. that is bugs or hard numbers that demonstrate the zcache has no value now (since previous number indictated it did have value) Aug 06 13:38:52 sjennings, This doesn't affect cleancache and it doesn't break frontswap, it just improves other kernel code. There will always be work, but I don't see that gating mainline acceptance. Aug 06 13:38:56 sjennings, The ones that Minchin identified were cleanups and refactor of code to make it more readable. No zcache engine replacement. Aug 06 13:40:01 konrad, that sort of patch can go in anytime though. I could send out patchess to clean-up and refactor mainline code. Aug 06 13:40:07 rcj: please help me understand... you want to promote zcache because it improved performance on older kernels even if it doesn't on current kernels? Aug 06 13:41:13 sjennings: the hard numbers clearly demonstrate that the existing zcache has much less value now Aug 06 13:42:22 djm1, I don't see you presenting numbers though, only saying that it "may" affect performance. So send out performance patches. Even if Rik never sent out his patches, could zcache stand performance work? Could any part of the kernel take performance patches? I don't see this as a gate to mainline. Aug 06 13:42:53 rcj: please answer my question, you have avoided it Aug 06 13:43:36 djm1, Is your thought that the answer to the performance numbers are the zcache2 different engine? Or the different algorithm for dealing with pages? Aug 06 13:43:40 it is true I don't wish to shoot myself in the foot by publicly showing that zcache sucks now, but nor do I want to put my sign-off on promoting it while hiding the knowledge that it does Aug 06 13:45:01 konrad: yes, I've sent many emails (offlist, and one or two on) about the design flaws of the "demo" zcache... Aug 06 13:45:55 djm1, So what you are saying is that zcache2 by itself , if it was in v3.3 would show phenomal numbers, not just good. Aug 06 13:46:25 djm1, I can't answer general questions unless we're talking real numbers. The code is in public and the development should be as well. Aug 06 13:46:26 djm1, And when measuring it with v3.5, it shows good respectible numbers. Aug 06 13:46:45 konrad: sadly no, my hand was forced to release the current status Aug 06 13:47:30 djm1, So what you are saying is that you want to continue on working on thsi until you do get to that point Aug 06 13:47:53 djm1, But that might take longer than a couple of releases and one might as well just drop in the new code. Aug 06 13:47:58 rcj: ok, let me give you two numbers off the top of my head... Seth's published measurement 1700 seconds on zcache, 2000 non-zcache. Aug 06 13:48:15 New measurement on non-zcache 3.5: 800 seconds Aug 06 13:48:46 did you use my machine? Aug 06 13:48:56 they aren't comparable Aug 06 13:49:13 plus i'm talking about I/O Aug 06 13:49:29 sjennings: I am inviting you to use your machine and re-measure, or to admit you don't care what the new numbers are Aug 06 13:50:08 * konrad sighs. Aug 06 13:50:33 I was hoping to have a clear define ACTION LISTs out of this and it seems we have at least two: Aug 06 13:50:44 1). I could send out patchess to clean-up and refactor mainline code Aug 06 13:50:59 2). Run existing benchmarks against v3.5 using zcache/non-zcache Aug 06 13:51:42 doing 1 won't resolve djm1 objections though Aug 06 13:52:35 djm1, So Dan, if the numbers on sjennings show no perf degradation, or his tests on v3.5 show good numbers, I believe you would be OK with the current zcache code right? Aug 06 13:52:56 sjennings, True. Just going through the action items to jot them down. Aug 06 13:54:36 konrad: if sjennings runs the same benchmarks as he previously published on 3.5 and zcache performance always beats non-zcache performance, that is a big step in the right direction Aug 06 13:55:12 konrad: but knowing the design flaws in zcache I can fairly easily find benchmarks for which it is not the case Aug 06 13:55:20 djm1, If they show horrible perf that is tried in with the generic swap system... then what? Your drop-in does not necessarily fixes it, and it might be a swap system issue. Which is why rcj proposoal to move with the unstaging as soon as possible so that we aren't behind with this is the right thing Aug 06 13:55:52 djm1, OK, but some benchmarks are so synthethic that they bear no resemblence to real world scenario. Aug 06 13:56:58 djm1, But the design flaws discussion is something that can be addressed later too. Aug 06 13:58:17 "Lies, damn lies, and statistics", right? Seth was talking about IOPs being reduced and Dan's talking about # seconds to run kern-bench which are different, you can always find some benchmark that falls down. But that doesn't mean there isn't some value still. zcache is not a panacea but does have use-cases. Aug 06 13:58:41 rcj: Heh Aug 06 13:59:19 And with measurements you can understand the problem and address it. But the question about promotion isn't necessarily resolved that way. Aug 06 13:59:58 * rcj re-reads the second sentence and needs to restate... Aug 06 14:00:26 rcj: please re-run the same benchmarks as sjennings ran in March, measure all three: seconds, pages of I/O and number of I/Os and then let's discuss further Aug 06 14:01:27 djm1, regarding the step in the right direction comment, i need to know what evidence would be required for you to drop your objection to promotion before i go through all the trouble of producing that evidence. Aug 06 14:01:53 if i can show that zcache reduces I/O during a kernel build with a 3.5 kernel, is that enouhg Aug 06 14:02:34 sjennings: across the same range of -jN? Aug 06 14:02:39 sure Aug 06 14:03:16 number of pages of I/O or number of I/Os? (and I ask that for a very specific reason) Aug 06 14:03:50 djm1, So I am curious. If this discussion was done in v3.3 day-time - would the issue of the design of zcache be on the table? Aug 06 14:04:04 djm1, Or the rewrite or the benchmarks? Aug 06 14:05:04 konrad: good question... yes, we had this discussion offlist prior to v3.3 (and I think you were cc'ed) Aug 06 14:05:35 at that point, zcache was "good" and I was trying to make it "better", now I fear that it sucks and I am trying to make it at least "good" Aug 06 14:07:19 rcj, sjennings: I have worked in a big company for 30 years, I understand you have constraints, and I can only guess what is driving this Aug 06 14:07:51 I suspect that IBM has a contractual obligation to deliver this in some embedded system which uses SSD/NVRAM Aug 06 14:08:31 djm1, our motivation is not relevant Aug 06 14:08:41 sjennings, and the community doesn't care. Aug 06 14:08:44 so to answer sjennings question, if zcache is truly going to be used successfully by real users, no, I don't think I want to get in the way of that Aug 06 14:08:50 this code is of value to use and others Aug 06 14:09:03 s/use/us Aug 06 14:10:00 and improvements can continue to be made Aug 06 14:10:09 sjennings: restating that and ignoring what I am telling you and refusing to measure it for yourself doesn't help Aug 06 14:11:21 rcj: the community most certainly does care... had Seth and I never published *amazing* zcache numbers (compared to native), we wouldn't be having this conversation Aug 06 14:12:03 djm1, I was just saying that the community doesn't care about motivations Aug 06 14:12:10 by promoting zcache based on those numbers, it makes me feel disingenous, if not just dirty Aug 06 14:12:18 djm1, I think you made your point. You would like Seth to do some benchmarking to verify the perf numbers. Aug 06 14:12:35 djm1, And if they do not work right, then address those before promoting it. Aug 06 14:13:34 but we don't agree on what "work right" is Aug 06 14:13:43 how are we defining that Aug 06 14:14:02 i define it in reduced I/O (even if runtime is longer) Aug 06 14:15:21 which is why motivation is relevant Aug 06 14:16:04 djm1, motivation==use case I think? Aug 06 14:16:09 if this is the classic battle: "which is more important? enterprise or embedded" then let's call it that Aug 06 14:16:55 konrad: yes, but "my use case runs faster so I don't care if yours run slower, just don't use it" Aug 06 14:17:57 djm1, So your concern is with .. I can't seem to find in the log here Aug 06 14:18:22 sjennings, rcj: so IMHO zram is the right match for you then... what is it about zcache that solves your problem better than zram? Aug 06 14:19:42 djm1, Ah, speed. Workloads running faster. Aug 06 14:23:19 * djm1 goes afk Aug 06 14:26:03 djm1, Um, hope you won't be long afk Aug 06 14:27:02 i'll post numbers on the v3.5 kernel Aug 06 14:27:20 sjennings, OK. Aug 06 14:27:31 sjennings, Thank you. Aug 06 14:27:45 * sashal has quit (Excess Flood) Aug 06 14:28:05 * sashal (~sasha@95-89-78-76-dynip.superkabel.de) has joined #zcache Aug 06 14:28:21 sashal, Last thing said was: " i'll post numbers on the v3.5 kernel Aug 06 14:28:21 sjennings, OK. Aug 06 14:28:21 sjennings, Thank you." Aug 06 14:29:00 sjennings, rcj , djm1 : I thought this would be only an hour but it went for an hour and half. Yikes. Aug 06 14:29:21 sjennings, rcj : I hope it doesn't negatively impact your guys schedule for today. Aug 06 14:30:08 sjennings, rcj djm1 :I think it makes sense to one more discussion about this after Seth has the opportunity to run the numbers. Aug 06 14:30:32 sjennings, rcj djm1 sashal : Would thsi time work next week? Would a conference call be better than IRC? Aug 06 14:30:57 konrad, IRC is better, this needs to be completely public. On-list would be better. Aug 06 14:31:31 konrad, I just don't want to see barriers to people being involved in a discussion. Aug 06 14:31:55 rcj, Good point. .. I am thinking of more of removing some of the barriers by having folks in one location so to speak. Aug 06 14:32:27 konrad, I understand completely. Just trying to find a good balance. Aug 06 14:33:27 sjennings, djm1, rcj, So let me write a summary of this and also save the full log as an attachment. Aug 06 14:33:30 konrad, I'm still worried by the premise of the objection. That a non-zcache patch has improved performance somewhere else in the kernel and this poses a barrier to mainline inclusion. Aug 06 14:34:52 rcj, Right. Thought not for embedded side. The things that Andrew Morton pointed out is that he would like zcache be enabled by default. Aug 06 14:36:07 rcj, And Dan is trying to make that happen. But if the enablement of zcache does not benefit on nowadays desktops..Then it should not be enabled by default. Aug 06 14:36:37 rcj, And then it comes back to Andrew having a potential issue with zcache. So Dan is trying to make sure that this issue will not surface. Aug 06 14:37:04 rcj, But the other side of the coin is that if its used in the embedded side then it does not matter. Aug 06 14:37:51 rcj, Which I believe is what you guys are coming from. Aug 06 14:38:58 konrad, We don't know the other show-stoppers because the conversation got killed pretty early on-list. Aug 06 14:40:37 rcj, Ugh. OK. Aug 06 14:41:32 rcj, Let me see what I have in the mbox and if I can compile to something similar to what Seth did for frontswa Aug 06 14:42:08 Well, I have to head out, other things are showing up in my calendar Aug 06 14:42:48 konrad, thanks. Aug 06 14:43:13 sjennings, rcj djm1 sashal : Gotta go. I think the two TODOs are: 1). Seth run v3.5 w/ zcache and w/o zcache; 2) Konrad to rummage in his mbox to find earlier objections to zcache Aug 06 14:43:27 * djm1 returnsm apparently too late Aug 06 14:43:54 konrad, sounds good. Aug 06 14:44:00 sjennings, rcj djm1 sashal : 3) Propose to continue this on mailing list for the more broader topics, while the more fine one - on IRC. Aug 06 14:44:17 .. and also get Minchan involved here. Aug 06 14:44:44 * konrad is away Aug 06 15:01:14 * sashal has quit (Ping timeout: 240 seconds) Aug 06 15:42:06 * sashal (~sasha@95-89-78-76-dynip.superkabel.de) has joined #zcache Aug 06 16:20:00 * rcj has quit (Quit: rcj) Aug 06 17:03:01 * sjennings has quit (Quit: Leaving) Aug 07 12:17:29 * Disconnected (Connection reset by peer).