Message-ID: <20211129151325.GA135990@fuller.cnet>
Date:   Mon, 29 Nov 2021 12:13:25 -0300
From:   Marcelo Tosatti <mtosatti@...hat.com>
To:     Frederic Weisbecker <frederic@...nel.org>
Cc:     linux-kernel@...r.kernel.org, Nitesh Lal <nilal@...hat.com>,
        Nicolas Saenz Julienne <nsaenzju@...hat.com>,
        Christoph Lameter <cl@...ux.com>,
        Juri Lelli <juri.lelli@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Alex Belits <abelits@...its.com>, Peter Xu <peterx@...hat.com>,
        Thomas Gleixner <tglx@...utronix.de>,
        Daniel Bristot de Oliveira <bristot@...hat.com>
Subject: Re: [patch v7 02/10] add prctl task isolation prctl docs and samples

On Tue, Nov 23, 2021 at 01:36:20PM +0100, Frederic Weisbecker wrote:
> On Fri, Nov 12, 2021 at 09:35:33AM -0300, Marcelo Tosatti wrote:
> > +**PR_ISOL_CFG_SET**:
> > +
> > +        Set task isolation configuration.
> > +        The general format is::
> > +
> > +                prctl(PR_ISOL_CFG_SET, what, arg3, arg4, arg5);
> > +
> > +        The 'what' argument specifies what to configure. Possible values are:
> > +
> > +        - ``I_CFG_FEAT``:
> > +
> > +                Set configuration of task isolation features. 'arg3' specifies
> > +                the feature. Possible values are:
> > +
> > +                - ``ISOL_F_QUIESCE``:
> > +
> > +                        If arg4 is QUIESCE_CONTROL, set the control structure
> > +                        for quiescing of background kernel activities, from
> > +                        the location pointed to by ``(int *)arg5``::
> > +
> > +                         struct task_isol_quiesce_control {
> > +                                __u64 flags;
> > +                                __u64 quiesce_mask;
> > +                                __u64 quiesce_oneshot_mask;
> > +                                __u64 pad[5];
> > +                         };
> > +
> > +                        Where:
> > +
> > +                        *flags*: Additional flags (should be zero).
> > +
> > +                        *quiesce_mask*: A bitmask containing which kernel
> > +                        activities to quiesce.
> > +
> > +                        *quiesce_oneshot_mask*: A bitmask indicating which kernel
> > +                        activities should behave in oneshot mode, that is, quiescing
> > +                        will happen on return from prctl(PR_ISOL_ACTIVATE_SET), but not
> > +                        on return of subsequent system calls. The corresponding bit(s)
> > +                        must also be set at quiesce_mask.
> > +
> > +                        *pad*: Additional space for future enhancements.
> > +
> > +                        For quiesce_mask (and quiesce_oneshot_mask), possible bit sets are:
> > +
> > +                        - ``ISOL_F_QUIESCE_VMSTATS``
> > +
> > +                        VM statistics are maintained in per-CPU counters to
> > +                        improve performance. When a CPU modifies a VM statistic,
> > +                        this modification is kept in the per-CPU counter.
> > +                        Certain activities require a global count, which
> > +                        involves requesting each CPU to flush its local counters
> > +                        to the global VM counters.
> > +
> > +                        This flush is implemented via a workqueue item, which
> > +                        might schedule a workqueue on isolated CPUs.
> > +
> > +                        To avoid this interruption, task isolation can be
> > +                        configured to, upon return from system calls, synchronize
> > +                        the per-CPU counters to global counters, thus avoiding
> > +                        the interruption.
> 
> Sorry I know this is already v7 but we really don't want to screw up this interface.

No problem.

> What would be more simple and flexible for individual features to quiesce:
> 
>    arg3 = ISOL_F_QUIESCE
>    arg4 = which feature to quiesce (eg: ISOL_F_QUIESCE_VMSTATS)

arg4 is QUIESCE_CONTROL today so one can expand the interface
(by adding new commands).

>    arg5 =
> 
>        struct task_isol_quiesce_control {
>            __u64 flags; //with ONESHOT as the first and only possible flag for now
>            __u64 pad[5];
>        };

So your idea is to allow expansion at this level ? Say while adding
a new

ISOL_F_QUIESCE_NEWITEM

one can add

	struct task_isol_quiesce_control_newitem {
		__u64 flags;
		__u64 pad[5];
	};

And add new fields to "struct task_isol_quiesce_control_newitem".

One downside of this suggestion is that for use-cases of the task_isol_computation.c type,
(see patch 2 "add prctl task isolation prctl docs and samples"), this might need
multiple system calls for each enable/disable cycle. Is that OK?

See more below.

> This way we can really do a finegrained control over each feature to quiesce.

With the patchset above, one can add new values to arg4 
(at the same level of QUIESCE_CONTROL). Your suggestion does not save
room for that.

One could add new values to the same space of I_CFG_FEAT:

          prctl(PR_ISOL_CFG_SET, I_CFG_FEAT_xxx, ...);

But that sounds awkward.

Or change the current ioctl to:

          prctl(PR_ISOL_CFG, I_CFG_FEAT_CONTROL, ...);

Which makes it less awkward.

What do you say?

--- 

And then, what about keeping the bitmaps with enabled/one-shot mode
per feature per bit (to avoid multiple system calls)
but having (in the future) additional per-quiesce instance commands ?

Powered by blists - more mailing lists