Message-ID: <3E5A0FA7E9CA944F9D5414FEC6C712205DFD172A@ORSMSX106.amr.corp.intel.com>
Date:	Fri, 1 Jul 2016 18:01:06 +0000
From:	"Yu, Fenghua" <fenghua.yu@...el.com>
To:	Borislav Petkov <bp@...e.de>, "Luck, Tony" <tony.luck@...el.com>
CC:	Thomas Gleixner <tglx@...utronix.de>, Ingo Molnar <mingo@...e.hu>,
	"Anvin, H Peter" <h.peter.anvin@...el.com>,
	Peter Zijlstra <peterz@...radead.org>,
	Stephane Eranian <eranian@...gle.com>,
	"Shankar, Ravi V" <ravi.v.shankar@...el.com>,
	Vikas Shivappa <vikas.shivappa@...ux.intel.com>,
	linux-kernel <linux-kernel@...r.kernel.org>, x86 <x86@...nel.org>
Subject: RE: [PATCH] cacheinfo: Introduce cache id

> From: Borislav Petkov [mailto:bp@...e.de]
> Sent: Friday, July 01, 2016 10:28 AM
> To: Luck, Tony <tony.luck@...el.com>
> Cc: Yu, Fenghua <fenghua.yu@...el.com>; Thomas Gleixner
> <tglx@...utronix.de>; Ingo Molnar <mingo@...e.hu>; Anvin, H Peter
> <h.peter.anvin@...el.com>; Peter Zijlstra <peterz@...radead.org>;
> Stephane Eranian <eranian@...gle.com>; Shankar, Ravi V
> <ravi.v.shankar@...el.com>; Vikas Shivappa
> <vikas.shivappa@...ux.intel.com>; linux-kernel <linux-
> kernel@...r.kernel.org>; x86 <x86@...nel.org>
> Subject: Re: [PATCH] cacheinfo: Introduce cache id
> 
> On Fri, Jul 01, 2016 at 09:50:41AM -0700, Luck, Tony wrote:
> > Here's the situation.  We have lots of (mostly) independent caches on a
> system.
> > The L3 cache (also called LLC - Last Level Cache - in some
> > documentation) is usually shared by all cpus on a physical socket. The
> > L1 and L2 caches are typically local to a core, so only shared by the
> hyperthreads on a core.
> > But I say "usually" and "typically" because the architecture doesn't
> > require that. We could have multiple separate L3 caches on a socket
> > with a subset of cpus assigned to each of them. We could have an L2
> > cache that is shared by two or more cores.
> 
> Right, so I'm presuming we don't want to make it special to the LLC but be
> generic and have any cache level have an ID.
> 
> And, since we simply call it cache, how about we drop the "node" thing and
> simply talk about cache and it having an ID. We say it is unique on the system
> and the cache with ID X is local to only thread X and the cache with ID Y is
> shared by threads Y_0, ... Y_k. And so on...
> 
> How does that sound?

That's good for me.

> 
> Struct doc will have then:
> 
>  /**
>   * struct cacheinfo - represent a cache leaf node
> + * @id: This cache's ID. ID is unique on the platform.
> 
> ?
> 
> which begs the question, does get_cache_id() give unique IDs for all caches
> in the hierarchy on the system?

Cache id is unique on the same level of cache across platform.

> 
> Because I tried this patch quickly in kvm and it was of course wrong but it
> hints at that important question:
> 
> cpu0/cache/index0/id:0b
> cpu0/cache/index1/id:0
> cpu0/cache/index2/id:0
> cpu1/cache/index0/id:1
> cpu1/cache/index1/id:1
> cpu1/cache/index2/id:1
> cpu2/cache/index0/id:2
> cpu2/cache/index1/id:2
> cpu2/cache/index2/id:2
> cpu3/cache/index0/id:3
> cpu3/cache/index1/id:3
> cpu3/cache/index2/id:3
> cpu4/cache/index0/id:4
> cpu4/cache/index1/id:4
> cpu4/cache/index2/id:4
> cpu5/cache/index0/id:5
> cpu5/cache/index1/id:5
> cpu5/cache/index2/id:5
> cpu6/cache/index0/id:6
> cpu6/cache/index1/id:6
> cpu6/cache/index2/id:6
> cpu7/cache/index0/id:7
> cpu7/cache/index1/id:7
> cpu7/cache/index2/id:7
> 
> Basically all cache indices carry the APIC ID of the core, so L1D on
> CPU0 has ID 0 and then L1I has ID 0 too and then L2 has also the same ID.

The cache id should represent shared cpus on one cache level according to SDM.

Could you check cpu#/cache/index#/shared_cpu_map or shared_cpu_list.
According to the cache id, there is NO shared cpu on each level. The shared_cpu_map
In your KVM should tell that as well, I think. I would guess all shared_cpu_map in each cache level on each cpu has itself on your KVM.

> 
> How does that look on a CAT system? Do all the different cache levels get
> different IDs?

On my Broadwell EP, the cache id info is as follows:

From index3/id, there are 2 cache ids for L3.
L3 cache id 0 has shared_cpu_map: 00,003ffff0,0003ffff
L3 cache id 1 has shared_cpu_map: ff,ffc0000f,fffc0000
Similar for L2 and L1.

Cache id is unique on the same level across platform.

#find /sys/device/system/cpu/. -name id|xargs cat
0
0
0
0
1
1
1
0
2
2
2
0
3
3
3
0
4
4
4
0
8
8
8
0
9
9
9
0
10
10
10
0
11
11
11
0
16
16
16
0
17
17
17
0
18
18
18
0
19
19
19
0
20
20
20
0
24
24
24
0
25
25
25
0
26
26
26
0
27
27
27
0
32
32
32
1
33
33
33
1
34
34
34
1
35
35
35
1
36
36
36
1
40
40
40
1
41
41
41
1
42
42
42
1
43
43
43
1
48
48
48
1
49
49
49
1
50
50
50
1
51
51
51
1
52
52
52
1
56
56
56
1
57
57
57
1
58
58
58
1
59
59
59
1
0
0
0
0
1
1
1
0
2
2
2
0
3
3
3
0
4
4
4
0
8
8
8
0
9
9
9
0
10
10
10
0
11
11
11
0
16
16
16
0
17
17
17
0
18
18
18
0
19
19
19
0
20
20
20
0
24
24
24
0
25
25
25
0
26
26
26
0
27
27
27
0
32
32
32
1
33
33
33
1
34
34
34
1
35
35
35
1
36
36
36
1
40
40
40
1
41
41
41
1
42
42
42
1
43
43
43
1
48
48
48
1
49
49
49
1
50
50
50
1
51
51
51
1
52
52
52
1
56
56
56
1
57
57
57
1
58
58
58
1
59
59
59
1

Powered by blists - more mailing lists