lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Message-ID: <20100314230121.GF6744@linux.vnet.ibm.com> Date: Sun, 14 Mar 2010 16:01:21 -0700 From: "Paul E. McKenney" <paulmck@...ux.vnet.ibm.com> To: Arnd Bergmann <arnd@...db.de> Cc: Herbert Xu <herbert@...dor.apana.org.au>, "David S. Miller" <davem@...emloft.net>, netdev@...r.kernel.org, Stephen Hemminger <shemminger@...tta.com> Subject: Re: [PATCH 6/13] bridge: Add core IGMP snooping support On Thu, Mar 11, 2010 at 07:49:52PM +0100, Arnd Bergmann wrote: > Following up on the earlier discussion, > > On Monday 08 March 2010, Arnd Bergmann wrote: > > > Arnd, would it be reasonable to extend your RCU-sparse changes to have > > > four different pointer namespaces, one for each flavor of RCU? (RCU, > > > RCU-bh, RCU-sched, and SRCU)? Always a fan of making the computer do > > > the auditing where reasonable. ;-) > > > > Yes, I guess that would be possible. I'd still leave out the rculist > > from any annotations for now, as this would get even more complex then. > > > > One consequence will be the need for new rcu_assign_pointer{,_bh,_sched} > > macros that check the address space of the first argument, otherwise > > you'd be able to stick anything in there, including non-__rcu pointers. > > I've tested this out now, see the patch below. I needed to add a number > of interfaces, but it still seems ok. Doing it for all the rculist > functions most likely would be less so. > > This is currently the head of my rcu-annotate branch of playground.git. > Paul, before I split it up and merge this with the per-subsystem patches, > can you tell me if this is what you had in mind? This looks extremely nice!!! I did note a few questions and a couple of minor change below, but the API and definitions look quite good. Search for empty lines to find them. Summary: o srcu_assign_pointer() should be defined in include/linux/srcu.h. o SRCU_INIT_POINTER() should be defined in include/linux/srcu.h. o rcu_dereference_check_sched_domain() can now rely on rcu_dereference_sched_check() to do the srcu_read_lock_held() check, so no longer needed at this level. o kvm_create_vm() should be able to use a single "buses" local variable rather than an array of them. Again, good stuff!!! Thank you for taking this on! > > > This could potentially catch the mismatched call_rcu()s, at least if the > > > rcu_head could be labeled. > > ... > > #define rcu_exchange_call(ptr, new, member, func) \ > > ({ \ > > typeof(new) old = rcu_exchange((ptr),(new)); \ > > if (old) \ > > call_rcu(&(old)->member, (func)); \ > > old; \ > > }) > > Unfortunately, this did not work out at all. Almost every user follows > a slightly different pattern for call_rcu, so I did not find a way > to match the call_rcu calls with the pointers. In particular, the functions > calling call_rcu() sometimes no longer have access to the 'old' data, > e.g. in case of synchronize_rcu. > > My current take is that static annotations won't help us here. Thank you for checking it out -- not every idea works out well in practice, I guess. ;-) Thanx, Paul > Arnd > > --- > > rcu: split up __rcu annotations > > This adds separate name spaces for the four distinct types of RCU > that we use in the kernel, namely __rcu, __rcu_bh, __rcu_sched and > __srcu. > > Signed-off-by: Arnd Bergmann <arnd@...db.de> > > --- > arch/x86/kvm/mmu.c | 6 ++-- > arch/x86/kvm/vmx.c | 2 +- > drivers/net/macvlan.c | 8 ++-- > include/linux/compiler.h | 6 ++++ > include/linux/kvm_host.h | 4 +- > include/linux/netdevice.h | 2 +- > include/linux/rcupdate.h | 68 +++++++++++++++++++++++++++++++++---------- > include/linux/srcu.h | 5 ++- > include/net/dst.h | 4 +- > include/net/llc.h | 3 +- > include/net/sock.h | 2 +- > include/trace/events/kvm.h | 4 +- > kernel/cgroup.c | 10 +++--- > kernel/perf_event.c | 8 ++-- > kernel/sched.c | 6 ++-- > kernel/sched_fair.c | 2 +- > lib/radix-tree.c | 8 ++-- > net/core/filter.c | 4 +- > net/core/sock.c | 6 ++-- > net/decnet/dn_route.c | 2 +- > net/ipv4/route.c | 60 +++++++++++++++++++------------------- > net/ipv4/tcp.c | 4 +- > net/llc/llc_core.c | 6 ++-- > net/llc/llc_input.c | 2 +- > virt/kvm/iommu.c | 4 +- > virt/kvm/kvm_main.c | 56 +++++++++++++++++++----------------- > 26 files changed, 171 insertions(+), 121 deletions(-) > > diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c > index 741373e..45877ca 100644 > --- a/arch/x86/kvm/mmu.c > +++ b/arch/x86/kvm/mmu.c > @@ -793,7 +793,7 @@ static int kvm_handle_hva(struct kvm *kvm, unsigned long hva, > int retval = 0; > struct kvm_memslots *slots; > > - slots = rcu_dereference(kvm->memslots); > + slots = srcu_dereference(kvm->memslots, &kvm->srcu); > > for (i = 0; i < slots->nmemslots; i++) { > struct kvm_memory_slot *memslot = &slots->memslots[i]; > @@ -3007,7 +3007,7 @@ unsigned int kvm_mmu_calculate_mmu_pages(struct kvm *kvm) > unsigned int nr_pages = 0; > struct kvm_memslots *slots; > > - slots = rcu_dereference(kvm->memslots); > + slots = srcu_dereference(kvm->memslots, &kvm->srcu); > for (i = 0; i < slots->nmemslots; i++) > nr_pages += slots->memslots[i].npages; > > @@ -3282,7 +3282,7 @@ static int count_rmaps(struct kvm_vcpu *vcpu) > int i, j, k, idx; > > idx = srcu_read_lock(&kvm->srcu); > - slots = rcu_dereference(kvm->memslots); > + slots = srcu_dereference(kvm->memslots, &kvm->srcu); > for (i = 0; i < KVM_MEMORY_SLOTS; ++i) { > struct kvm_memory_slot *m = &slots->memslots[i]; > struct kvm_rmap_desc *d; > diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c > index 0aec1f3..d0c82ed 100644 > --- a/arch/x86/kvm/vmx.c > +++ b/arch/x86/kvm/vmx.c > @@ -1513,7 +1513,7 @@ static gva_t rmode_tss_base(struct kvm *kvm) > struct kvm_memslots *slots; > gfn_t base_gfn; > > - slots = rcu_dereference(kvm->memslots); > + slots = srcu_dereference(kvm->memslots, &kvm->srcu); > base_gfn = slots->memslots[0].base_gfn + > slots->memslots[0].npages - 3; > return base_gfn << PAGE_SHIFT; > diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c > index 95e1bcc..b958d5a 100644 > --- a/drivers/net/macvlan.c > +++ b/drivers/net/macvlan.c > @@ -531,15 +531,15 @@ static int macvlan_port_create(struct net_device *dev) > INIT_LIST_HEAD(&port->vlans); > for (i = 0; i < MACVLAN_HASH_SIZE; i++) > INIT_HLIST_HEAD(&port->vlan_hash[i]); > - rcu_assign_pointer(dev->macvlan_port, port); > + rcu_assign_pointer_bh(dev->macvlan_port, port); > return 0; > } > > static void macvlan_port_destroy(struct net_device *dev) > { > - struct macvlan_port *port = rcu_dereference_const(dev->macvlan_port); > + struct macvlan_port *port = rcu_dereference_bh_const(dev->macvlan_port); > > - rcu_assign_pointer(dev->macvlan_port, NULL); > + rcu_assign_pointer_bh(dev->macvlan_port, NULL); > synchronize_rcu(); > kfree(port); > } > @@ -624,7 +624,7 @@ int macvlan_common_newlink(struct net *src_net, struct net_device *dev, > if (err < 0) > return err; > } > - port = rcu_dereference(lowerdev->macvlan_port); > + port = rcu_dereference_bh(lowerdev->macvlan_port); > > vlan->lowerdev = lowerdev; > vlan->dev = dev; > diff --git a/include/linux/compiler.h b/include/linux/compiler.h > index 0ab21c2..d5756d4 100644 > --- a/include/linux/compiler.h > +++ b/include/linux/compiler.h > @@ -17,6 +17,9 @@ > # define __cond_lock(x,c) ((c) ? ({ __acquire(x); 1; }) : 0) > # define __percpu __attribute__((noderef, address_space(3))) > # define __rcu __attribute__((noderef, address_space(4))) > +# define __rcu_bh __attribute__((noderef, address_space(5))) > +# define __rcu_sched __attribute__((noderef, address_space(6))) > +# define __srcu __attribute__((noderef, address_space(7))) > extern void __chk_user_ptr(const volatile void __user *); > extern void __chk_io_ptr(const volatile void __iomem *); > #else > @@ -36,6 +39,9 @@ extern void __chk_io_ptr(const volatile void __iomem *); > # define __cond_lock(x,c) (c) > # define __percpu > # define __rcu > +# define __rcu_bh > +# define __rcu_sched > +# define __srcu > #endif > > #ifdef __KERNEL__ > diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h > index 9eb0f9c..bad1787 100644 > --- a/include/linux/kvm_host.h > +++ b/include/linux/kvm_host.h > @@ -164,7 +164,7 @@ struct kvm { > raw_spinlock_t requests_lock; > struct mutex slots_lock; > struct mm_struct *mm; /* userspace tied to this vm */ > - struct kvm_memslots __rcu *memslots; > + struct kvm_memslots __srcu *memslots; > struct srcu_struct srcu; > #ifdef CONFIG_KVM_APIC_ARCHITECTURE > u32 bsp_vcpu_id; > @@ -174,7 +174,7 @@ struct kvm { > atomic_t online_vcpus; > struct list_head vm_list; > struct mutex lock; > - struct kvm_io_bus __rcu *buses[KVM_NR_BUSES]; > + struct kvm_io_bus __srcu *buses[KVM_NR_BUSES]; > #ifdef CONFIG_HAVE_KVM_EVENTFD > struct { > spinlock_t lock; > diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h > index fd7e8de..1b72188 100644 > --- a/include/linux/netdevice.h > +++ b/include/linux/netdevice.h > @@ -949,7 +949,7 @@ struct net_device { > /* bridge stuff */ > void __rcu *br_port; > /* macvlan */ > - struct macvlan_port __rcu *macvlan_port; > + struct macvlan_port __rcu_bh *macvlan_port; > /* GARP */ > struct garp_port __rcu *garp_port; > > diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h > index 03702cc..b4c6f39 100644 > --- a/include/linux/rcupdate.h > +++ b/include/linux/rcupdate.h > @@ -183,19 +183,33 @@ static inline int rcu_read_lock_sched_held(void) > * read-side critical section. It is also possible to check for > * locks being held, for example, by using lockdep_is_held(). > */ > -#define rcu_dereference_check(p, c) \ > +#define __rcu_dereference_check(p, c, space) \ > ({ \ > if (debug_locks && !(c)) \ > lockdep_rcu_dereference(__FILE__, __LINE__); \ > - rcu_dereference_raw(p); \ > + __rcu_dereference_raw(p, space); \ > }) > > + > #else /* #ifdef CONFIG_PROVE_RCU */ > > -#define rcu_dereference_check(p, c) rcu_dereference_raw(p) > +#define __rcu_dereference_check(p, c, space) \ > + __rcu_dereference_raw(p, space) > > #endif /* #else #ifdef CONFIG_PROVE_RCU */ > > +#define rcu_dereference_check(p, c) \ > + __rcu_dereference_check(p, c, __rcu) > + > +#define rcu_dereference_bh_check(p, c) \ > + __rcu_dereference_check(p, rcu_read_lock_bh_held() || (c), __rcu_bh) > + > +#define rcu_dereference_sched_check(p, c) \ > + __rcu_dereference_check(p, rcu_read_lock_sched_held() || (c), __rcu_sched) > + > +#define srcu_dereference_check(p, c) \ > + __rcu_dereference_check(p, srcu_read_lock_held() || (c), __srcu) > + > /** > * rcu_read_lock - mark the beginning of an RCU read-side critical section. > * > @@ -341,13 +355,15 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) > * exactly which pointers are protected by RCU and checks that > * the pointer is annotated as __rcu. > */ > -#define rcu_dereference_raw(p) ({ \ > +#define __rcu_dereference_raw(p, space) ({ \ > typeof(*p) *_________p1 = (typeof(*p)*__force )ACCESS_ONCE(p); \ > - (void) (((typeof (*p) __rcu *)p) == p); \ > + (void) (((typeof (*p) space *)p) == p); \ > smp_read_barrier_depends(); \ > ((typeof(*p) __force __kernel *)(_________p1)); \ > }) > > +#define rcu_dereference_raw(p) __rcu_dereference_raw(p, __rcu) > + > /** > * rcu_dereference_const - fetch an __rcu pointer outside of a > * read-side critical section. > @@ -360,18 +376,22 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) > * or in an RCU call. > */ > > -#define rcu_dereference_const(p) ({ \ > - (void) (((typeof (*p) __rcu *)p) == p); \ > +#define __rcu_dereference_const(p, space) ({ \ > + (void) (((typeof (*p) space *)p) == p); \ > ((typeof(*p) __force __kernel *)(p)); \ > }) > > +#define rcu_dereference_const(p) __rcu_dereference_const(p, __rcu) > +#define rcu_dereference_bh_const(p) __rcu_dereference_const(p, __rcu_bh) > +#define rcu_dereference_sched_const(p) __rcu_dereference_const(p, __rcu_sched) > + > /** > * rcu_dereference - fetch an RCU-protected pointer, checking for RCU > * > * Makes rcu_dereference_check() do the dirty work. > */ > #define rcu_dereference(p) \ > - rcu_dereference_check(p, rcu_read_lock_held()) > + __rcu_dereference_check(p, rcu_read_lock_held(), __rcu) > > /** > * rcu_dereference_bh - fetch an RCU-protected pointer, checking for RCU-bh > @@ -379,7 +399,7 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) > * Makes rcu_dereference_check() do the dirty work. > */ > #define rcu_dereference_bh(p) \ > - rcu_dereference_check(p, rcu_read_lock_bh_held()) > + __rcu_dereference_check(p, rcu_read_lock_bh_held(), __rcu_bh) > > /** > * rcu_dereference_sched - fetch RCU-protected pointer, checking for RCU-sched > @@ -387,7 +407,7 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) > * Makes rcu_dereference_check() do the dirty work. > */ > #define rcu_dereference_sched(p) \ > - rcu_dereference_check(p, rcu_read_lock_sched_held()) > + __rcu_dereference_check(p, rcu_read_lock_sched_held(), __rcu_sched) > > /** > * rcu_assign_pointer - assign (publicize) a pointer to a newly > @@ -402,12 +422,12 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) > * code. > */ > > -#define rcu_assign_pointer(p, v) \ > +#define __rcu_assign_pointer(p, v, space) \ > ({ \ > if (!__builtin_constant_p(v) || \ > ((v) != NULL)) \ > smp_wmb(); \ > - (p) = (typeof(*v) __force __rcu *)(v); \ > + (p) = (typeof(*v) __force space *)(v); \ > }) > > /** > @@ -415,10 +435,17 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) > * without barriers. > * Using this is almost always a bug. > */ > -#define __rcu_assign_pointer(p, v) \ > - ({ \ > - (p) = (typeof(*v) __force __rcu *)(v); \ > - }) > +#define rcu_assign_pointer(p, v) \ > + __rcu_assign_pointer(p, v, __rcu) > + > +#define rcu_assign_pointer_bh(p, v) \ > + __rcu_assign_pointer(p, v, __rcu_bh) > + > +#define rcu_assign_pointer_sched(p, v) \ > + __rcu_assign_pointer(p, v, __rcu_sched) > + > +#define srcu_assign_pointer(p, v) \ > + __rcu_assign_pointer(p, v, __srcu) For consistency, the definition of srcu_assign_pointer() should go into include/linux/srcu.h. > /** > * RCU_INIT_POINTER - initialize an RCU protected member > @@ -427,6 +454,15 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) > #define RCU_INIT_POINTER(p, v) \ > p = (typeof(*v) __force __rcu *)(v) > > +#define RCU_INIT_POINTER_BH(p, v) \ > + p = (typeof(*v) __force __rcu_bh *)(v) > + > +#define RCU_INIT_POINTER_SCHED(p, v) \ > + p = (typeof(*v) __force __rcu_sched *)(v) > + > +#define SRCU_INIT_POINTER(p, v) \ > + p = (typeof(*v) __force __srcu *)(v) > + For consistency, the definition of SRCU_INIT_POINTER() should go into include/linux/srcu.h. > /* Infrastructure to implement the synchronize_() primitives. */ > > struct rcu_synchronize { > diff --git a/include/linux/srcu.h b/include/linux/srcu.h > index 4d5ecb2..feaf661 100644 > --- a/include/linux/srcu.h > +++ b/include/linux/srcu.h > @@ -111,7 +111,10 @@ static inline int srcu_read_lock_held(struct srcu_struct *sp) > * Makes rcu_dereference_check() do the dirty work. > */ > #define srcu_dereference(p, sp) \ > - rcu_dereference_check(p, srcu_read_lock_held(sp)) > + __rcu_dereference_check(p, srcu_read_lock_held(sp), __srcu) > + > +#define srcu_dereference_const(p) \ > + __rcu_dereference_const(p, __srcu) > > /** > * srcu_read_lock - register a new reader for an SRCU-protected structure. > diff --git a/include/net/dst.h b/include/net/dst.h > index 5f839aa..bbeaba2 100644 > --- a/include/net/dst.h > +++ b/include/net/dst.h > @@ -94,9 +94,9 @@ struct dst_entry { > unsigned long lastuse; > union { > struct dst_entry *next; > - struct rtable __rcu *rt_next; > + struct rtable __rcu_bh *rt_next; > struct rt6_info *rt6_next; > - struct dn_route *dn_next; > + struct dn_route __rcu_bh *dn_next; > }; > }; > > diff --git a/include/net/llc.h b/include/net/llc.h > index 8299cb2..5700082 100644 > --- a/include/net/llc.h > +++ b/include/net/llc.h > @@ -59,7 +59,8 @@ struct llc_sap { > int (* rcv_func)(struct sk_buff *skb, > struct net_device *dev, > struct packet_type *pt, > - struct net_device *orig_dev) __rcu; > + struct net_device *orig_dev) > + __rcu_bh; > struct llc_addr laddr; > struct list_head node; > spinlock_t sk_lock; > diff --git a/include/net/sock.h b/include/net/sock.h > index e07cd78..66d5e09 100644 > --- a/include/net/sock.h > +++ b/include/net/sock.h > @@ -290,7 +290,7 @@ struct sock { > struct ucred sk_peercred; > long sk_rcvtimeo; > long sk_sndtimeo; > - struct sk_filter __rcu *sk_filter; > + struct sk_filter __rcu_bh *sk_filter; > void *sk_protinfo; > struct timer_list sk_timer; > ktime_t sk_stamp; > diff --git a/kernel/cgroup.c b/kernel/cgroup.c > index fc45694..db3e502 100644 > --- a/kernel/cgroup.c > +++ b/kernel/cgroup.c > @@ -1392,7 +1392,7 @@ static int cgroup_get_sb(struct file_system_type *fs_type, > root_count++; > > sb->s_root->d_fsdata = root_cgrp; > - __rcu_assign_pointer(root->top_cgroup.dentry, sb->s_root); > + rcu_assign_pointer(root->top_cgroup.dentry, sb->s_root); > > /* Link the top cgroup in this hierarchy into all > * the css_set objects */ > @@ -3243,7 +3243,7 @@ int __init cgroup_init_early(void) > css_set_count = 1; > init_cgroup_root(&rootnode); > root_count = 1; > - __rcu_assign_pointer(init_task.cgroups, &init_css_set); > + rcu_assign_pointer(init_task.cgroups, &init_css_set); > > init_css_set_link.cg = &init_css_set; > init_css_set_link.cgrp = dummytop; > @@ -3551,7 +3551,7 @@ void cgroup_exit(struct task_struct *tsk, int run_callbacks) > /* Reassign the task to the init_css_set. */ > task_lock(tsk); > cg = rcu_dereference_const(tsk->cgroups); > - __rcu_assign_pointer(tsk->cgroups, &init_css_set); > + rcu_assign_pointer(tsk->cgroups, &init_css_set); > task_unlock(tsk); > if (cg) > put_css_set_taskexit(cg); > @@ -3959,8 +3959,8 @@ static int __init cgroup_subsys_init_idr(struct cgroup_subsys *ss) > return PTR_ERR(newid); > > newid->stack[0] = newid->id; > - __rcu_assign_pointer(newid->css, rootcss); > - __rcu_assign_pointer(rootcss->id, newid); > + rcu_assign_pointer(newid->css, rootcss); > + rcu_assign_pointer(rootcss->id, newid); > return 0; > } > > diff --git a/kernel/perf_event.c b/kernel/perf_event.c > index ac8bcbd..e1b65b2 100644 > --- a/kernel/perf_event.c > +++ b/kernel/perf_event.c > @@ -1223,8 +1223,8 @@ void perf_event_task_sched_out(struct task_struct *task, > * XXX do we need a memory barrier of sorts > * wrt to rcu_dereference() of perf_event_ctxp > */ > - __rcu_assign_pointer(task->perf_event_ctxp, next_ctx); > - __rcu_assign_pointer(next->perf_event_ctxp, ctx); > + rcu_assign_pointer(task->perf_event_ctxp, next_ctx); > + rcu_assign_pointer(next->perf_event_ctxp, ctx); > ctx->task = next; > next_ctx->task = task; > do_switch = 0; > @@ -5376,10 +5376,10 @@ int perf_event_init_task(struct task_struct *child) > */ > cloned_ctx = rcu_dereference(parent_ctx->parent_ctx); > if (cloned_ctx) { > - __rcu_assign_pointer(child_ctx->parent_ctx, cloned_ctx); > + rcu_assign_pointer(child_ctx->parent_ctx, cloned_ctx); > child_ctx->parent_gen = parent_ctx->parent_gen; > } else { > - __rcu_assign_pointer(child_ctx->parent_ctx, parent_ctx); > + rcu_assign_pointer(child_ctx->parent_ctx, parent_ctx); > child_ctx->parent_gen = parent_ctx->generation; > } > get_ctx(rcu_dereference_const(child_ctx->parent_ctx)); > diff --git a/kernel/sched.c b/kernel/sched.c > index 05fd61e..83744d6 100644 > --- a/kernel/sched.c > +++ b/kernel/sched.c > @@ -528,7 +528,7 @@ struct rq { > > #ifdef CONFIG_SMP > struct root_domain *rd; > - struct sched_domain __rcu *sd; > + struct sched_domain __rcu_sched *sd; > > unsigned char idle_at_tick; > /* For active balancing */ > @@ -603,7 +603,7 @@ static inline int cpu_of(struct rq *rq) > } > > #define rcu_dereference_check_sched_domain(p) \ > - rcu_dereference_check((p), \ > + rcu_dereference_sched_check((p), \ > rcu_read_lock_sched_held() || \ > lockdep_is_held(&sched_domains_mutex)) Given your definition, the "rcu_read_lock_sched_held() || \" should now be able to be deleted, correct? > > @@ -6323,7 +6323,7 @@ cpu_attach_domain(struct sched_domain *sd, struct root_domain *rd, int cpu) > sched_domain_debug(sd, cpu); > > rq_attach_root(rq, rd); > - rcu_assign_pointer(rq->sd, sd); > + rcu_assign_pointer_sched(rq->sd, sd); > } > > /* cpus with isolated domains */ > diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c > index 3e1fd96..5a5ea2c 100644 > --- a/kernel/sched_fair.c > +++ b/kernel/sched_fair.c > @@ -3476,7 +3476,7 @@ static void run_rebalance_domains(struct softirq_action *h) > > static inline int on_null_domain(int cpu) > { > - return !rcu_dereference(cpu_rq(cpu)->sd); > + return !rcu_dereference_sched(cpu_rq(cpu)->sd); > } > > /* > diff --git a/lib/radix-tree.c b/lib/radix-tree.c > index f6ae74c..4c6f149 100644 > --- a/lib/radix-tree.c > +++ b/lib/radix-tree.c > @@ -264,7 +264,7 @@ static int radix_tree_extend(struct radix_tree_root *root, unsigned long index) > return -ENOMEM; > > /* Increase the height. */ > - __rcu_assign_pointer(node->slots[0], > + rcu_assign_pointer(node->slots[0], > radix_tree_indirect_to_ptr(rcu_dereference_const(root->rnode))); > > /* Propagate the aggregated tag info into the new root */ > @@ -1090,7 +1090,7 @@ static inline void radix_tree_shrink(struct radix_tree_root *root) > newptr = rcu_dereference_const(to_free->slots[0]); > if (root->height > 1) > newptr = radix_tree_ptr_to_indirect(newptr); > - __rcu_assign_pointer(root->rnode, newptr); > + rcu_assign_pointer(root->rnode, newptr); > root->height--; > radix_tree_node_free(to_free); > } > @@ -1125,7 +1125,7 @@ void *radix_tree_delete(struct radix_tree_root *root, unsigned long index) > slot = rcu_dereference_const(root->rnode); > if (height == 0) { > root_tag_clear_all(root); > - __rcu_assign_pointer(root->rnode, NULL); > + rcu_assign_pointer(root->rnode, NULL); > goto out; > } > slot = radix_tree_indirect_to_ptr(slot); > @@ -1183,7 +1183,7 @@ void *radix_tree_delete(struct radix_tree_root *root, unsigned long index) > } > root_tag_clear_all(root); > root->height = 0; > - __rcu_assign_pointer(root->rnode, NULL); > + rcu_assign_pointer(root->rnode, NULL); > if (to_free) > radix_tree_node_free(to_free); > > diff --git a/net/core/filter.c b/net/core/filter.c > index d38ef7f..b88675b 100644 > --- a/net/core/filter.c > +++ b/net/core/filter.c > @@ -522,7 +522,7 @@ int sk_attach_filter(struct sock_fprog *fprog, struct sock *sk) > > rcu_read_lock_bh(); > old_fp = rcu_dereference_bh(sk->sk_filter); > - rcu_assign_pointer(sk->sk_filter, fp); > + rcu_assign_pointer_bh(sk->sk_filter, fp); > rcu_read_unlock_bh(); > > if (old_fp) > @@ -539,7 +539,7 @@ int sk_detach_filter(struct sock *sk) > rcu_read_lock_bh(); > filter = rcu_dereference_bh(sk->sk_filter); > if (filter) { > - rcu_assign_pointer(sk->sk_filter, NULL); > + rcu_assign_pointer_bh(sk->sk_filter, NULL); > sk_filter_delayed_uncharge(sk, filter); > ret = 0; > } > diff --git a/net/core/sock.c b/net/core/sock.c > index 74242e2..8549387 100644 > --- a/net/core/sock.c > +++ b/net/core/sock.c > @@ -1073,11 +1073,11 @@ static void __sk_free(struct sock *sk) > if (sk->sk_destruct) > sk->sk_destruct(sk); > > - filter = rcu_dereference_check(sk->sk_filter, > + filter = rcu_dereference_bh_check(sk->sk_filter, > atomic_read(&sk->sk_wmem_alloc) == 0); > if (filter) { > sk_filter_uncharge(sk, filter); > - rcu_assign_pointer(sk->sk_filter, NULL); > + rcu_assign_pointer_bh(sk->sk_filter, NULL); > } > > sock_disable_timestamp(sk, SOCK_TIMESTAMP); > @@ -1167,7 +1167,7 @@ struct sock *sk_clone(const struct sock *sk, const gfp_t priority) > sock_reset_flag(newsk, SOCK_DONE); > skb_queue_head_init(&newsk->sk_error_queue); > > - filter = rcu_dereference_const(newsk->sk_filter); > + filter = rcu_dereference_bh_const(newsk->sk_filter); > if (filter != NULL) > sk_filter_charge(newsk, filter); > > diff --git a/net/decnet/dn_route.c b/net/decnet/dn_route.c > index a7bf03c..22ec1d1 100644 > --- a/net/decnet/dn_route.c > +++ b/net/decnet/dn_route.c > @@ -92,7 +92,7 @@ > > struct dn_rt_hash_bucket > { > - struct dn_route *chain; > + struct dn_route __rcu_bh *chain; > spinlock_t lock; > }; > > diff --git a/net/ipv4/route.c b/net/ipv4/route.c > index 37bf0d9..99cef80 100644 > --- a/net/ipv4/route.c > +++ b/net/ipv4/route.c > @@ -200,7 +200,7 @@ const __u8 ip_tos2prio[16] = { > */ > > struct rt_hash_bucket { > - struct rtable __rcu *chain; > + struct rtable __rcu_bh *chain; > }; > > #if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK) || \ > @@ -731,26 +731,26 @@ static void rt_do_flush(int process_context) > spin_lock_bh(rt_hash_lock_addr(i)); > #ifdef CONFIG_NET_NS > { > - struct rtable __rcu ** prev; > + struct rtable __rcu_bh ** prev; > struct rtable * p; > > - rth = rcu_dereference_const(rt_hash_table[i].chain); > + rth = rcu_dereference_bh(rt_hash_table[i].chain); > > /* defer releasing the head of the list after spin_unlock */ > - for (tail = rth; tail; tail = rcu_dereference_const(tail->u.dst.rt_next)) > + for (tail = rth; tail; tail = rcu_dereference_bh(tail->u.dst.rt_next)) > if (!rt_is_expired(tail)) > break; > if (rth != tail) > - __rcu_assign_pointer(rt_hash_table[i].chain, tail); > + rcu_assign_pointer_bh(rt_hash_table[i].chain, tail); > > /* call rt_free on entries after the tail requiring flush */ > prev = &rt_hash_table[i].chain; > - for (p = rcu_dereference_const(*prev); p; p = next) { > - next = rcu_dereference_const(p->u.dst.rt_next); > + for (p = rcu_dereference_bh(*prev); p; p = next) { > + next = rcu_dereference_bh(p->u.dst.rt_next); > if (!rt_is_expired(p)) { > prev = &p->u.dst.rt_next; > } else { > - __rcu_assign_pointer(*prev, next); > + rcu_assign_pointer_bh(*prev, next); > rt_free(p); > } > } > @@ -763,7 +763,7 @@ static void rt_do_flush(int process_context) > spin_unlock_bh(rt_hash_lock_addr(i)); > > for (; rth != tail; rth = next) { > - next = rcu_dereference_const(rth->u.dst.rt_next); > + next = rcu_dereference_bh(rth->u.dst.rt_next); > rt_free(rth); > } > } > @@ -785,7 +785,7 @@ static void rt_check_expire(void) > static unsigned int rover; > unsigned int i = rover, goal; > struct rtable *rth, *aux; > - struct rtable __rcu **rthp; > + struct rtable __rcu_bh **rthp; > unsigned long samples = 0; > unsigned long sum = 0, sum2 = 0; > unsigned long delta; > @@ -815,8 +815,8 @@ static void rt_check_expire(void) > continue; > length = 0; > spin_lock_bh(rt_hash_lock_addr(i)); > - while ((rth = rcu_dereference_const(*rthp)) != NULL) { > - prefetch(rcu_dereference_const(rth->u.dst.rt_next)); > + while ((rth = rcu_dereference_bh(*rthp)) != NULL) { > + prefetch(rcu_dereference_bh(rth->u.dst.rt_next)); > if (rt_is_expired(rth)) { > *rthp = rth->u.dst.rt_next; > rt_free(rth); > @@ -836,14 +836,14 @@ nofree: > * attributes don't unfairly skew > * the length computation > */ > - for (aux = rcu_dereference_const(rt_hash_table[i].chain);;) { > + for (aux = rcu_dereference_bh(rt_hash_table[i].chain);;) { > if (aux == rth) { > length += ONE; > break; > } > if (compare_hash_inputs(&aux->fl, &rth->fl)) > break; > - aux = rcu_dereference_const(aux->u.dst.rt_next); > + aux = rcu_dereference_bh(aux->u.dst.rt_next); > } > continue; > } > @@ -959,7 +959,7 @@ static int rt_garbage_collect(struct dst_ops *ops) > static int rover; > static int equilibrium; > struct rtable *rth; > - struct rtable __rcu **rthp; > + struct rtable __rcu_bh **rthp; > unsigned long now = jiffies; > int goal; > > @@ -1012,7 +1012,7 @@ static int rt_garbage_collect(struct dst_ops *ops) > k = (k + 1) & rt_hash_mask; > rthp = &rt_hash_table[k].chain; > spin_lock_bh(rt_hash_lock_addr(k)); > - while ((rth = rcu_dereference_const(*rthp)) != NULL) { > + while ((rth = rcu_dereference_bh(*rthp)) != NULL) { > if (!rt_is_expired(rth) && > !rt_may_expire(rth, tmo, expire)) { > tmo >>= 1; > @@ -1079,10 +1079,10 @@ static int rt_intern_hash(unsigned hash, struct rtable *rt, > struct rtable **rp, struct sk_buff *skb) > { > struct rtable *rth; > - struct rtable __rcu **rthp; > + struct rtable __rcu_bh **rthp; > unsigned long now; > struct rtable *cand; > - struct rtable __rcu **candp; > + struct rtable __rcu_bh **candp; > u32 min_score; > int chain_length; > int attempts = !in_softirq(); > @@ -1129,7 +1129,7 @@ restart: > rthp = &rt_hash_table[hash].chain; > > spin_lock_bh(rt_hash_lock_addr(hash)); > - while ((rth = rcu_dereference_const(*rthp)) != NULL) { > + while ((rth = rcu_dereference_bh(*rthp)) != NULL) { > if (rt_is_expired(rth)) { > *rthp = rth->u.dst.rt_next; > rt_free(rth); > @@ -1143,13 +1143,13 @@ restart: > * must be visible to another weakly ordered CPU before > * the insertion at the start of the hash chain. > */ > - rcu_assign_pointer(rth->u.dst.rt_next, > + rcu_assign_pointer_bh(rth->u.dst.rt_next, > rt_hash_table[hash].chain); > /* > * Since lookup is lockfree, the update writes > * must be ordered for consistency on SMP. > */ > - rcu_assign_pointer(rt_hash_table[hash].chain, rth); > + rcu_assign_pointer_bh(rt_hash_table[hash].chain, rth); > > dst_use(&rth->u.dst, now); > spin_unlock_bh(rt_hash_lock_addr(hash)); > @@ -1252,7 +1252,7 @@ restart: > * previous writes to rt are comitted to memory > * before making rt visible to other CPUS. > */ > - rcu_assign_pointer(rt_hash_table[hash].chain, rt); > + rcu_assign_pointer_bh(rt_hash_table[hash].chain, rt); > > spin_unlock_bh(rt_hash_lock_addr(hash)); > > @@ -1325,13 +1325,13 @@ void __ip_select_ident(struct iphdr *iph, struct dst_entry *dst, int more) > > static void rt_del(unsigned hash, struct rtable *rt) > { > - struct rtable __rcu **rthp; > + struct rtable __rcu_bh **rthp; > struct rtable *aux; > > rthp = &rt_hash_table[hash].chain; > spin_lock_bh(rt_hash_lock_addr(hash)); > ip_rt_put(rt); > - while ((aux = rcu_dereference_const(*rthp)) != NULL) { > + while ((aux = rcu_dereference_bh(*rthp)) != NULL) { > if (aux == rt || rt_is_expired(aux)) { > *rthp = aux->u.dst.rt_next; > rt_free(aux); > @@ -1348,7 +1348,7 @@ void ip_rt_redirect(__be32 old_gw, __be32 daddr, __be32 new_gw, > int i, k; > struct in_device *in_dev = in_dev_get(dev); > struct rtable *rth; > - struct rtable __rcu **rthp; > + struct rtable __rcu_bh **rthp; > __be32 skeys[2] = { saddr, 0 }; > int ikeys[2] = { dev->ifindex, 0 }; > struct netevent_redirect netevent; > @@ -1384,7 +1384,7 @@ void ip_rt_redirect(__be32 old_gw, __be32 daddr, __be32 new_gw, > rthp=&rt_hash_table[hash].chain; > > rcu_read_lock(); > - while ((rth = rcu_dereference(*rthp)) != NULL) { > + while ((rth = rcu_dereference_bh(*rthp)) != NULL) { > struct rtable *rt; > > if (rth->fl.fl4_dst != daddr || > @@ -1646,8 +1646,8 @@ unsigned short ip_rt_frag_needed(struct net *net, struct iphdr *iph, > rt_genid(net)); > > rcu_read_lock(); > - for (rth = rcu_dereference(rt_hash_table[hash].chain); rth; > - rth = rcu_dereference(rth->u.dst.rt_next)) { > + for (rth = rcu_dereference_bh(rt_hash_table[hash].chain); rth; > + rth = rcu_dereference_bh(rth->u.dst.rt_next)) { > unsigned short mtu = new_mtu; > > if (rth->fl.fl4_dst != daddr || > @@ -2287,8 +2287,8 @@ int ip_route_input(struct sk_buff *skb, __be32 daddr, __be32 saddr, > hash = rt_hash(daddr, saddr, iif, rt_genid(net)); > > rcu_read_lock(); > - for (rth = rcu_dereference(rt_hash_table[hash].chain); rth; > - rth = rcu_dereference(rth->u.dst.rt_next)) { > + for (rth = rcu_dereference_bh(rt_hash_table[hash].chain); rth; > + rth = rcu_dereference_bh(rth->u.dst.rt_next)) { > if (((rth->fl.fl4_dst ^ daddr) | > (rth->fl.fl4_src ^ saddr) | > (rth->fl.iif ^ iif) | > diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c > index d8ce05b..003d54f 100644 > --- a/net/ipv4/tcp.c > +++ b/net/ipv4/tcp.c > @@ -3252,9 +3252,9 @@ void __init tcp_init(void) > memset(&tcp_secret_two.secrets[0], 0, sizeof(tcp_secret_two.secrets)); > tcp_secret_one.expires = jiffy; /* past due */ > tcp_secret_two.expires = jiffy; /* past due */ > - __rcu_assign_pointer(tcp_secret_generating, &tcp_secret_one); > + rcu_assign_pointer(tcp_secret_generating, &tcp_secret_one); > tcp_secret_primary = &tcp_secret_one; > - __rcu_assign_pointer(tcp_secret_retiring, &tcp_secret_two); > + rcu_assign_pointer(tcp_secret_retiring, &tcp_secret_two); > tcp_secret_secondary = &tcp_secret_two; > } > > diff --git a/net/llc/llc_core.c b/net/llc/llc_core.c > index ed7f424..8696677 100644 > --- a/net/llc/llc_core.c > +++ b/net/llc/llc_core.c > @@ -50,7 +50,7 @@ static struct llc_sap *__llc_sap_find(unsigned char sap_value) > { > struct llc_sap* sap; > > - list_for_each_entry(sap, &llc_sap_list, node) > + list_for_each_entry_rcu(sap, &llc_sap_list, node) > if (sap->laddr.lsap == sap_value) > goto out; > sap = NULL; > @@ -103,7 +103,7 @@ struct llc_sap *llc_sap_open(unsigned char lsap, > if (!sap) > goto out; > sap->laddr.lsap = lsap; > - rcu_assign_pointer(sap->rcv_func, func); > + rcu_assign_pointer_bh(sap->rcv_func, func); > list_add_tail_rcu(&sap->node, &llc_sap_list); > out: > spin_unlock_bh(&llc_sap_list_lock); > @@ -127,7 +127,7 @@ void llc_sap_close(struct llc_sap *sap) > list_del_rcu(&sap->node); > spin_unlock_bh(&llc_sap_list_lock); > > - synchronize_rcu(); > + synchronize_rcu_bh(); > > kfree(sap); > } > diff --git a/net/llc/llc_input.c b/net/llc/llc_input.c > index 57ad974..b775530 100644 > --- a/net/llc/llc_input.c > +++ b/net/llc/llc_input.c > @@ -179,7 +179,7 @@ int llc_rcv(struct sk_buff *skb, struct net_device *dev, > * First the upper layer protocols that don't need the full > * LLC functionality > */ > - rcv = rcu_dereference(sap->rcv_func); > + rcv = rcu_dereference_bh(sap->rcv_func); > if (rcv) { > struct sk_buff *cskb = skb_clone(skb, GFP_ATOMIC); > if (cskb) > diff --git a/virt/kvm/iommu.c b/virt/kvm/iommu.c > index 80fd3ad..2ba7048 100644 > --- a/virt/kvm/iommu.c > +++ b/virt/kvm/iommu.c > @@ -78,7 +78,7 @@ static int kvm_iommu_map_memslots(struct kvm *kvm) > int i, r = 0; > struct kvm_memslots *slots; > > - slots = rcu_dereference(kvm->memslots); > + slots = srcu_dereference(kvm->memslots, &kvm->srcu); > > for (i = 0; i < slots->nmemslots; i++) { > r = kvm_iommu_map_pages(kvm, &slots->memslots[i]); > @@ -217,7 +217,7 @@ static int kvm_iommu_unmap_memslots(struct kvm *kvm) > int i; > struct kvm_memslots *slots; > > - slots = rcu_dereference(kvm->memslots); > + slots = srcu_dereference(kvm->memslots, &kvm->srcu); > > for (i = 0; i < slots->nmemslots; i++) { > kvm_iommu_put_pages(kvm, slots->memslots[i].base_gfn, > diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c > index 548f925..ae28c71 100644 > --- a/virt/kvm/kvm_main.c > +++ b/virt/kvm/kvm_main.c > @@ -372,6 +372,8 @@ static struct kvm *kvm_create_vm(void) > { > int r = 0, i; > struct kvm *kvm = kvm_arch_create_vm(); > + struct kvm_io_bus *buses[KVM_NR_BUSES]; > + struct kvm_memslots *memslots; This one looks like more that simply an RCU change... OK, I get it -- you are creating these temporaries in order to avoid overflowing the line. Never mind!!! ;-) > if (IS_ERR(kvm)) > goto out; > @@ -386,14 +388,15 @@ static struct kvm *kvm_create_vm(void) > #endif > > r = -ENOMEM; > - kvm->memslots = kzalloc(sizeof(struct kvm_memslots), GFP_KERNEL); > + memslots = kzalloc(sizeof(struct kvm_memslots), GFP_KERNEL); > + srcu_assign_pointer(kvm->memslots, memslots); > if (!kvm->memslots) > goto out_err; > if (init_srcu_struct(&kvm->srcu)) > goto out_err; > for (i = 0; i < KVM_NR_BUSES; i++) { > - kvm->buses[i] = kzalloc(sizeof(struct kvm_io_bus), > - GFP_KERNEL); > + buses[i] = kzalloc(sizeof(struct kvm_io_bus), GFP_KERNEL); > + srcu_assign_pointer(kvm->buses[i], buses[i]); But why do you need an array for "buses" instead of only one variable? > if (!kvm->buses[i]) { > cleanup_srcu_struct(&kvm->srcu); > goto out_err; > @@ -428,8 +431,8 @@ out_err: > hardware_disable_all(); > out_err_nodisable: > for (i = 0; i < KVM_NR_BUSES; i++) > - kfree(kvm->buses[i]); > - kfree(kvm->memslots); > + kfree(buses[i]); OK, I see what you are trying to do. But why not free all the non-NULL ones from the kvm-> structure, and then use a single "buses" rather than an array of them? Perhaps running "i" down from whereever the earlier loop left it, in case it is difficult to zero the underlying kvm-> structure? Just trying to save a bit of stack space... > + kfree(memslots); > kfree(kvm); > return ERR_PTR(r); > } > @@ -464,12 +467,12 @@ static void kvm_free_physmem_slot(struct kvm_memory_slot *free, > void kvm_free_physmem(struct kvm *kvm) > { > int i; > - struct kvm_memslots *slots = kvm->memslots; > + struct kvm_memslots *slots = srcu_dereference_const(kvm->memslots); > > for (i = 0; i < slots->nmemslots; ++i) > kvm_free_physmem_slot(&slots->memslots[i], NULL); > > - kfree(kvm->memslots); > + kfree(slots); > } > > static void kvm_destroy_vm(struct kvm *kvm) > @@ -483,7 +486,7 @@ static void kvm_destroy_vm(struct kvm *kvm) > spin_unlock(&kvm_lock); > kvm_free_irq_routing(kvm); > for (i = 0; i < KVM_NR_BUSES; i++) > - kvm_io_bus_destroy(kvm->buses[i]); > + kvm_io_bus_destroy(srcu_dereference_const(kvm->buses[i])); > kvm_coalesced_mmio_free(kvm); > #if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER) > mmu_notifier_unregister(&kvm->mmu_notifier, kvm->mm); > @@ -552,7 +555,8 @@ int __kvm_set_memory_region(struct kvm *kvm, > if (mem->guest_phys_addr + mem->memory_size < mem->guest_phys_addr) > goto out; > > - memslot = &kvm->memslots->memslots[mem->slot]; > + old_memslots = srcu_dereference(kvm->memslots, &kvm->srcu); > + memslot = &old_memslots->memslots[mem->slot]; > base_gfn = mem->guest_phys_addr >> PAGE_SHIFT; > npages = mem->memory_size >> PAGE_SHIFT; > > @@ -573,7 +577,7 @@ int __kvm_set_memory_region(struct kvm *kvm, > /* Check for overlaps */ > r = -EEXIST; > for (i = 0; i < KVM_MEMORY_SLOTS; ++i) { > - struct kvm_memory_slot *s = &kvm->memslots->memslots[i]; > + struct kvm_memory_slot *s = &old_memslots->memslots[i]; > > if (s == memslot || !s->npages) > continue; > @@ -669,13 +673,13 @@ skip_lpage: > slots = kzalloc(sizeof(struct kvm_memslots), GFP_KERNEL); > if (!slots) > goto out_free; > - memcpy(slots, kvm->memslots, sizeof(struct kvm_memslots)); > + old_memslots = srcu_dereference_const(kvm->memslots); > + memcpy(slots, old_memslots, sizeof(struct kvm_memslots)); > if (mem->slot >= slots->nmemslots) > slots->nmemslots = mem->slot + 1; > slots->memslots[mem->slot].flags |= KVM_MEMSLOT_INVALID; > > - old_memslots = kvm->memslots; > - rcu_assign_pointer(kvm->memslots, slots); > + srcu_assign_pointer(kvm->memslots, slots); > synchronize_srcu_expedited(&kvm->srcu); > /* From this point no new shadow pages pointing to a deleted > * memslot will be created. > @@ -705,7 +709,8 @@ skip_lpage: > slots = kzalloc(sizeof(struct kvm_memslots), GFP_KERNEL); > if (!slots) > goto out_free; > - memcpy(slots, kvm->memslots, sizeof(struct kvm_memslots)); > + old_memslots = srcu_dereference_const(kvm->memslots); > + memcpy(slots, old_memslots, sizeof(struct kvm_memslots)); > if (mem->slot >= slots->nmemslots) > slots->nmemslots = mem->slot + 1; > > @@ -718,8 +723,7 @@ skip_lpage: > } > > slots->memslots[mem->slot] = new; > - old_memslots = kvm->memslots; > - rcu_assign_pointer(kvm->memslots, slots); > + srcu_assign_pointer(kvm->memslots, slots); > synchronize_srcu_expedited(&kvm->srcu); > > kvm_arch_commit_memory_region(kvm, mem, old, user_alloc); > @@ -775,7 +779,7 @@ int kvm_get_dirty_log(struct kvm *kvm, > if (log->slot >= KVM_MEMORY_SLOTS) > goto out; > > - memslot = &kvm->memslots->memslots[log->slot]; > + memslot = &srcu_dereference(kvm->memslots, &kvm->srcu)->memslots[log->slot]; > r = -ENOENT; > if (!memslot->dirty_bitmap) > goto out; > @@ -829,7 +833,7 @@ EXPORT_SYMBOL_GPL(kvm_is_error_hva); > struct kvm_memory_slot *gfn_to_memslot_unaliased(struct kvm *kvm, gfn_t gfn) > { > int i; > - struct kvm_memslots *slots = rcu_dereference(kvm->memslots); > + struct kvm_memslots *slots = srcu_dereference(kvm->memslots, &kvm->srcu); > > for (i = 0; i < slots->nmemslots; ++i) { > struct kvm_memory_slot *memslot = &slots->memslots[i]; > @@ -851,7 +855,7 @@ struct kvm_memory_slot *gfn_to_memslot(struct kvm *kvm, gfn_t gfn) > int kvm_is_visible_gfn(struct kvm *kvm, gfn_t gfn) > { > int i; > - struct kvm_memslots *slots = rcu_dereference(kvm->memslots); > + struct kvm_memslots *slots = srcu_dereference(kvm->memslots, &kvm->srcu); > > gfn = unalias_gfn_instantiation(kvm, gfn); > for (i = 0; i < KVM_MEMORY_SLOTS; ++i) { > @@ -895,7 +899,7 @@ out: > int memslot_id(struct kvm *kvm, gfn_t gfn) > { > int i; > - struct kvm_memslots *slots = rcu_dereference(kvm->memslots); > + struct kvm_memslots *slots = srcu_dereference(kvm->memslots, &kvm->srcu); > struct kvm_memory_slot *memslot = NULL; > > gfn = unalias_gfn(kvm, gfn); > @@ -1984,7 +1988,7 @@ int kvm_io_bus_write(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr, > int len, const void *val) > { > int i; > - struct kvm_io_bus *bus = rcu_dereference(kvm->buses[bus_idx]); > + struct kvm_io_bus *bus = srcu_dereference(kvm->buses[bus_idx], &kvm->srcu); > for (i = 0; i < bus->dev_count; i++) > if (!kvm_iodevice_write(bus->devs[i], addr, len, val)) > return 0; > @@ -1996,7 +2000,7 @@ int kvm_io_bus_read(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t addr, > int len, void *val) > { > int i; > - struct kvm_io_bus *bus = rcu_dereference(kvm->buses[bus_idx]); > + struct kvm_io_bus *bus = srcu_dereference(kvm->buses[bus_idx], &kvm->srcu); > > for (i = 0; i < bus->dev_count; i++) > if (!kvm_iodevice_read(bus->devs[i], addr, len, val)) > @@ -2010,7 +2014,7 @@ int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx, > { > struct kvm_io_bus *new_bus, *bus; > > - bus = kvm->buses[bus_idx]; > + bus = srcu_dereference_const(kvm->buses[bus_idx]); > if (bus->dev_count > NR_IOBUS_DEVS-1) > return -ENOSPC; > > @@ -2019,7 +2023,7 @@ int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx, > return -ENOMEM; > memcpy(new_bus, bus, sizeof(struct kvm_io_bus)); > new_bus->devs[new_bus->dev_count++] = dev; > - rcu_assign_pointer(kvm->buses[bus_idx], new_bus); > + srcu_assign_pointer(kvm->buses[bus_idx], new_bus); > synchronize_srcu_expedited(&kvm->srcu); > kfree(bus); > > @@ -2037,7 +2041,7 @@ int kvm_io_bus_unregister_dev(struct kvm *kvm, enum kvm_bus bus_idx, > if (!new_bus) > return -ENOMEM; > > - bus = kvm->buses[bus_idx]; > + bus = srcu_dereference_const(kvm->buses[bus_idx]); > memcpy(new_bus, bus, sizeof(struct kvm_io_bus)); > > r = -ENOENT; > @@ -2053,7 +2057,7 @@ int kvm_io_bus_unregister_dev(struct kvm *kvm, enum kvm_bus bus_idx, > return r; > } > > - rcu_assign_pointer(kvm->buses[bus_idx], new_bus); > + srcu_assign_pointer(kvm->buses[bus_idx], new_bus); > synchronize_srcu_expedited(&kvm->srcu); > kfree(bus); > return r; > -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists