Hello,

On Mon, Nov 15, 2021 at 08:31:22PM +0100, Michal Koutný wrote:

Now to the constraints and partition setups. I think it's useful to have a model with which the implementation can be compared with. I tried to condense some "simple rules" from the descriptions you posted in v8 plus your response to my remarks in v7 [2]. These should only be the "validity conditions", not "transition conditions".

FWIW, my opinion is pretty much in line with Michal's in this regard. Other than that, everything looks pretty good to me.

Thanks.

On 11/15/21 14:31, Michal Koutný wrote:

Hello.

On Mon, Oct 18, 2021 at 10:36:18AM -0400, Waiman Long longman@redhat.com wrote:

...

• When set to "isolated", the CPUs in that partition root will
• be in an isolated state without any load balancing from the
• scheduler. Tasks in such a partition must be explicitly bound
• to each individual CPU.

This sounds reasonable but it seems to have some usability issues as was raised in another thread [1]. (I could only think of the workaround of single-cpu cgroup leaves + CLONE_INTO_CGROUP.)

It can be a problem when one is trying to move from one cgroup to another cgroup with non-overlapping cpus laterally. However, if a task is initially from a parent cgroup with affinity mask that include cpus in the isolated child cgroup, I believe it should be able to move to the isolated child cgroup without problem. Otherwise, it is a bug that needs to be fixed.

TL;DR Do whatever you find suitable but (re)consider sticking to the delegation principle (making hotplug and ancestor changes equal).

Now to the constraints and partition setups. I think it's useful to have a model with which the implementation can be compared with. I tried to condense some "simple rules" from the descriptions you posted in v8 plus your response to my remarks in v7 [2]. These should only be the "validity conditions", not "transition conditions".

## Validity conditions

For simplification, there's a condition called 'degraded' that tells whether a cpuset can host tasks (with the given config) that expands to two predicates:

degraded := cpus.internal_effective == ø && has_tasks valid_root := !degraded && cpus_exclusive && parent.valid_root (valid_member := !degraded)

with a helping predicate cpus_exclusive := cpus not shared by a sibling

The effective CPUs basically combine configured+available CPUs

cpus.internal_effective := (cpus ∩ parent.cpus ∩ online_cpus) - passed

where passed := union of children cpus whose partition is not member

Finally, to handle the degraded cpusets gracefully, we define

if (!degraded) cpus.effective := cpus.internal_effective else cpus.effective := parent.cpus.effective

(In cases when there's no parent, we replace its cpus with online_cpus.)

---

I'll try applying these conditions to your description.

...

• "cpuset.cpus" must always be set up first before enabling
• partition.

This is just a transition condition.

...

   Unlike "member" whose "cpuset.cpus.effective" can

• contain CPUs not in "cpuset.cpus", this can never happen with a
• valid partition root. In other words, "cpuset.cpus.effective"
• is always a subset of "cpuset.cpus" for a valid partition root.

IIUC this refers to the cgroup that is 'degraded'. (The consequences for a valid partition root follow from valid_root definition above.)

...

• When a parent partition root cannot exclusively grant any of
• the CPUs specified in "cpuset.cpus", "cpuset.cpus.effective"
• becomes empty.

This sounds too strict to me, perhaps you meant 'cannot grant _all_ of the CPUs'?

I think the wording may be confusing. What I meant is none of the requested cpu can be granted. So if there is at least one granted, the effective cpus won't be empty.

...

   If there are tasks in the partition root, the

• partition root becomes invalid and "cpuset.cpus.effective"
• is reset to that of the nearest non-empty ancestor.

This is captured in the definition of 'degraded'.

...

•    Note that a task cannot be moved to a croup with empty
  

•    "cpuset.cpus.effective".
  

A transition condition. (Makes sense.)

[With the validity conditions above, it's possible to have 'valid_root' with empty cpus (hence also empty cpus.internal_effective) if there are no tasks in there. The transition conditions so far prevented this corner case.]

...

• There are additional constraints on where a partition root can
• be enabled ("root" or "isolated"). It can only be enabled in
• a cgroup if all the following conditions are met.

I think the enablement (aka rewriting cpuset.cpus.partition) could be always possible but it'd result in "root invalid (...)" if the resulting config doesn't meet the validity condition.

...

• 1. The "cpuset.cpus" is non-empty and exclusive, i.e. they are

•   not shared by any of its siblings.
  

The emptiness here is a judgement call (in my formulation of the conditions it seemed simpler to allow empty cpus.internal_effective with no tasks).

There are more constraints in enabling a partition. Once it is enabled, there will be less constraints to maintain its validity.

...

• 2. The parent cgroup is a valid partition root.

Captured in the valid_root definition.

...

• 3. The "cpuset.cpus" is a subset of parent's "cpuset.cpus".

This is unnecessary strictness. Allow such config, cpus.internal_effective still can't be more than parent's cpuset.cpus. (Or do you have a reason to discard such configs?)

...

• 4. There is no child cgroups with cpuset enabled. This avoids

•   cpu migrations of multiple cgroups simultaneously which can
  

•   be problematic.
  

A transition condition (i.e. not relevant to validity conditions).

...

• Once becoming a partition root, changes to "cpuset.cpus"
• is generally allowed as long as the cpu list is exclusive,
• non-empty and is a superset of children's cpu lists.

Any changes should be allowed otherwise it denies the delegation principle of v2 (IOW a parent should be able to preempt CPUs given to chilren previously and not be denied because of them).

(If the change results in failed validity condition the cgroup of course cannot be be a valid_root anymore.)

...

•    The constraints of a valid partition root are as follows:
  

•    1) The parent cgroup is a valid partition root.
  

•    2) "cpuset.cpus.effective" is a subset of "cpuset.cpus"
  

•    3) "cpuset.cpus.effective" is non-empty when there are tasks
  

•       in the partition.
  

(This seem to miss the sibling exclusivity condition.) Here I'd simply paste the "Validity conditions" specified above instead.

You currently cannot make change to cpuset.cpus that violates the cpu exclusivity rule. The above constraints will not disallow you to make the change. They just affect the validity of the partition root.

...

•    Changing a partition root to "member" is always allowed.
  

•    If there are child partition roots underneath it, however,
  

•    they will be forced to be switched back to "member" too and
  

•    lose their partitions. So care must be taken to double check
  

•    for this condition before disabling a partition root.
  

(Or is this how delegation is intended?) However, AFAICS, parent still can't remove cpuset.cpus even when the child is a "member". Otherwise, I agree with the back-switch.

There are only 2 possibilities here. Either we force the child partitions to be become members or invalid partition root. The purpose of invalid partition root is actually a transient state which can be recovered in some way to make the partition again. However, changing a parent partition root to member breaks the possibility of recovering later. That is why I think it is more sensible to force those child partitions to become members.

...

• Setting a cgroup to a valid partition root will take the CPUs
• away from the effective CPUs of the parent partition.

Captured in the definition of cpus.internal_effective.

...

• A valid parent partition may distribute out all its CPUs to
• its child partitions as long as it is not the root cgroup as
• we need some house-keeping CPUs in the root cgroup.

This actually applies to any root partition that's supposed to host tasks. (IOW, 'valid_root' cannot be 'degraded'.)

...

• An invalid partition is not a real partition even though some
• internal states may still be kept.

Tautology? (Or new definition of "real".)

...

• An invalid partition root can be reverted back to a real
• partition root if none of the constraints of a valid partition

•    root are violated.
  

Yes. (Also tautological.)

Anyway, as I said above, I just tried to formulate the model for clearer understanding and the implementation may introduce transition constraints but it'd be good to always have the simple rules to tell what's a valid root in the tree and what's not.

Thanks for analyzing each statements for their validity. I will try to improve it to make it easier to understand.

Cheers, Longman

On 11/16/21 12:54, Michal Koutný wrote:

On Mon, Nov 15, 2021 at 04:10:29PM -0500, Waiman Long longman@redhat.com wrote:

...

...

On Mon, Oct 18, 2021 at 10:36:18AM -0400, Waiman Long longman@redhat.com wrote:

...

• scheduler. Tasks in such a partition must be explicitly bound
• to each individual CPU.

[...]

It can be a problem when one is trying to move from one cgroup to another cgroup with non-overlapping cpus laterally. However, if a task is initially from a parent cgroup with affinity mask that include cpus in the isolated child cgroup, I believe it should be able to move to the isolated child cgroup without problem. Otherwise, it is a bug that needs to be fixed.

app_root cpuset.cpus=0-3 `- non_rt cpuset.cpus=0-1 cpuset.cpus.partition=member `- rt cpuset.cpus=2-3 cpuset.cpus.partition=isolated

The app_root would have cpuset.cpus.effective=0-1 so even the task in app_root can't sched_setaffinity() to cpus 2-3. But AFAICS, the migration calls set_cpus_allowed_ptr() anyway, so the task in the isolated partition needn't to bind explicitly with sched_setaffinity(). (It'd have two cpus available, so one more sched_setaffinity() or migration into a single-cpu list is desirable.)

All in all, I think the behavior is OK and the explicit binding of tasks in an isolated cpuset is optional (not a must as worded currently).

...

I think the wording may be confusing. What I meant is none of the requested cpu can be granted. So if there is at least one granted, the effective cpus won't be empty.

Ack.

...

You currently cannot make change to cpuset.cpus that violates the cpu exclusivity rule. The above constraints will not disallow you to make the change. They just affect the validity of the partition root.

Sibling exclusivity should be a validity condition regardless of whether transition is allowed or not. (At least it looks simpler to me.)

...

...

...

•    Changing a partition root to "member" is always allowed.
  

•    If there are child partition roots underneath it, however,
  

•    they will be forced to be switched back to "member" too and
  

•    lose their partitions. So care must be taken to double check
  

•    for this condition before disabling a partition root.
  

(Or is this how delegation is intended?) However, AFAICS, parent still can't remove cpuset.cpus even when the child is a "member". Otherwise, I agree with the back-switch.

There are only 2 possibilities here. Either we force the child partitions to be become members or invalid partition root.

My point here was mostly about preempting the cpus (as a v2 specific feature). (I'm rather indifferent whether children turn into invalid roots or members.)

Below is my latest iterations of the cpuset.cpus.partition documentation. If there is no objection or other suggestion for improvement, I am going to send out another iteration of the patch series with the updated documentation.

Cheers, Longman

--------------------------------------------------------------

cpuset.cpus.partition     A read-write single value file which exists on non-root     cpuset-enabled cgroups.  This flag is owned by the parent cgroup     and is not delegatable.

It accepts only the following input values when written to.

========    ================================       "member"    Non-root member of a partition       "root"    Partition root       "isolated"    Partition root without load balancing       ========    ================================

The root cgroup is always a partition root and its state     cannot be changed.  All other non-root cgroups start out as     "member".

When set to "root", the current cgroup is the root of a new     partition or scheduling domain that comprises itself and     all its descendants except those that are separate partition     roots themselves and their descendants.

The value shown in "cpuset.cpus.effective" of a partition root is     the CPUs that the parent partition root can dedicate to the new     partition root.  They are subtracted from "cpuset.cpus.effective"     of the parent and may be different from "cpuset.cpus"

When set to "isolated", the CPUs in that partition root will     be in an isolated state without any load balancing from the     scheduler.  Tasks placed in such a partition with multiple     CPUs should be carefully distributed and bound to each of the     individual CPUs for optimal performance.

A partition root ("root" or "isolated") can be in one of the     two possible states - valid or invalid.  An invalid partition     root is in a degraded state where some state information are     retained, but behaves more like a "member".

On read, the "cpuset.cpus.partition" file can show the following     values.

======================    ==============================       "member"            Non-root member of a partition       "root"            Partition root       "isolated"            Partition root without load balancing       "root invalid (<reason>)"    Invalid partition root       ======================    ==============================

In the case of an invalid partition root, a descriptive string on     why the partition is invalid is included within parentheses.

Almost all possible state transitions among "member", valid     and invalid partition roots are allowed except from "member"     to invalid partition root.

Before the "member" to partition root transition can happen,     the following conditions must be met or the transition will     not be allowed.

1) The "cpuset.cpus" is non-empty and exclusive, i.e. they are        not shared by any of its siblings.     2) The parent cgroup is a valid partition root.     3) The "cpuset.cpus" is a subset of parent's "cpuset.cpus".     4) There is no child cgroups with cpuset enabled.  This avoids        cpu migrations of multiple cgroups simultaneously which can        be problematic.

Once becoming a partition root, the following two rules restrict     what changes can be made to "cpuset.cpus".

1) The value must be exclusive.     2) If child cpusets exist, the value must be a superset of what        are defined in the child cpusets.

The second rule applies even for "member". Other changes to     "cpuset.cpus" that do not violate the above rules are always     allowed.

External events like hotplug or inappropriate changes to     "cpuset.cpus" can cause a valid partition root to become invalid.     Besides the constraints on changing "cpuset.cpus" listed above,     the other conditions required to maintain the validity of a     partition root are as follows:

1) The parent cgroup is a valid partition root.     2) If "cpuset.cpus.effective" is empty, the partition must have        no task associated with it. Otherwise, the partition becomes        invalid and "cpuset.cpus.effective" will fall back to that        of the nearest non-empty ancestor.

A corollary of a valid partition root is that     "cpuset.cpu.effective" is always a subset of "cpuset.cpus".     Note that a task cannot be moved to a cgroup with empty     "cpuset.cpus.effective".

Changing a partition root (valid or invalid) to "member" is     always allowed.  If there are child partition roots underneath     it, however, they will be forced to be switched back to "member"     too and lose their partitions. So care must be taken to double     check for this condition before disabling a partition root.

A valid parent partition may distribute out all its CPUs to     its child partitions as long as it is not the root cgroup and     there is no task associated with it.

An invalid partition root can be reverted back to a valid one     if none of the validity constraints of a valid partition root     are violated.

Poll and inotify events are triggered whenever the state of     "cpuset.cpus.partition" changes.  That includes changes caused by     write to "cpuset.cpus.partition", cpu hotplug and other changes     that make the partition invalid.  This will allow user space     agents to monitor unexpected changes to "cpuset.cpus.partition"     without the need to do continuous polling.

On 11/30/21 12:11, Tejun Heo wrote:

Hello, Waiman.

On Tue, Nov 30, 2021 at 10:35:19AM -0500, Waiman Long wrote:

...

On read, the "cpuset.cpus.partition" file can show the following     values.

======================    ==============================       "member"            Non-root member of a partition       "root"            Partition root       "isolated"            Partition root without load balancing       "root invalid (<reason>)"    Invalid partition root       ======================    ==============================

What happens if an isolated domain becomes invalid and then valid again due to cpu hotplug? Does it go "root invalid" and then back to "isolated"?

Yes, the current code allow recovering from an invalid state. In this particular case, the transition will be "isolated" --> "root invalid" --> "isolated".

...

...

Before the "member" to partition root transition can happen,     the following conditions must be met or the transition will     not be allowed.

1) The "cpuset.cpus" is non-empty and exclusive, i.e. they are        not shared by any of its siblings.     2) The parent cgroup is a valid partition root.     3) The "cpuset.cpus" is a subset of parent's "cpuset.cpus".     4) There is no child cgroups with cpuset enabled.  This avoids        cpu migrations of multiple cgroups simultaneously which can        be problematic.

So, I still have a hard time justifying the above restrictions. 1) can be broken through hotplug anyway. 2) can be broken by the parent switching to member. 3) would mean that we'd need to restrict parent's config changes depending on what children are doing. 4) is more understandable but it's an implementation detail that we can address in the future.

The initial transition to a partition root has a higher barrier. Once it becomes a partition root. Some restrictions are relaxed.

...

Once becoming a partition root, the following two rules restrict     what changes can be made to "cpuset.cpus".

1) The value must be exclusive.     2) If child cpusets exist, the value must be a superset of what        are defined in the child cpusets.

The second rule applies even for "member". Other changes to     "cpuset.cpus" that do not violate the above rules are always     allowed.

While it isn't necessarily tied to this series, it's a big no-no to restrict what a parent can do depending on what its descendants are doing. A cgroup higher up in the hierarchy should be able to change configuration however it sees fit as deligation breaks down otherwise.

Maybe you can argue that cpuset is special and shouldn't be subject to such convention but I can't see strong enough justifications especially given that most of these restrictions can be broken by hotplug operations anyway and thus need code to handle those situations.

These are all pre-existing restrictions before the introduction of partition. These are checks done in validate_change(). I am just saying out loud the existing behavior. If you think that needs to be changed, I am fine with that. However, it will be a separate patch as it is not a behavior that is introduced by this series.

...

Changing a partition root (valid or invalid) to "member" is     always allowed.  If there are child partition roots underneath     it, however, they will be forced to be switched back to "member"     too and lose their partitions. So care must be taken to double     check for this condition before disabling a partition root.

Wouldn't it make more sense for them to retain their configuration and turn invalid? Why is this special?

Once an invalid partition is changed to "member", there is no way for a child invalid partition root to recover and become valid again. There is why I force them to become "member" also. I am OK if you believe it is better to keep them in the invalid state forever until we explicitly changed them to "member" eventually.

...

A valid parent partition may distribute out all its CPUs to     its child partitions as long as it is not the root cgroup and     there is no task associated with it.

A valid parent partition which isn't root never has tasks in them to begin with.

I believe there is some corner cases where it is possible to put task in an intermediate partition. That is why I put down this statement.

Cheers, Longman

On 12/1/21 09:13, Michal Koutný wrote:

On Tue, Nov 30, 2021 at 10:56:34PM -0500, Waiman Long longman@redhat.com wrote:

...

...

...

A valid parent partition may distribute out all its CPUs to     its child partitions as long as it is not the root cgroup and     there is no task associated with it.

A valid parent partition which isn't root never has tasks in them to begin with.

I believe there is some corner cases where it is possible to put task in an intermediate partition. That is why I put down this statement.

Just mind the threads -- cpuset controller is threaded and having tasks in inner cgroup nodes is a real scenario. I wouldn't consider it a corner case.

[ Actually, the paragraph could IMO be simplified:

Right, I shouldn't say corner cases. Having task in an intermediate partition is possible depending on event sequence. I am aware that there are code in the cpuset code to prevent that, but it didn't block all cases.

...

A valid parent partition may distribute out all its CPUs to  its child partitions as long as there is no task associated with it.

Assuming there's always at least one kernel thread in the root cgroup that can't be migrated anyway.]

I am aware of that. That is why I said root cgroup must have at least one cpu in its "cpuset.cpus.effective".

Cheers, Longman

On 12/1/21 11:39, Tejun Heo wrote:

On Wed, Dec 01, 2021 at 09:56:21AM -0500, Waiman Long wrote:

...

Right, I shouldn't say corner cases. Having task in an intermediate partition is possible depending on event sequence. I am aware that there are code in the cpuset code to prevent that, but it didn't block all cases.

...

...

A valid parent partition may distribute out all its CPUs to  its child partitions as long as there is no task associated with it.

Assuming there's always at least one kernel thread in the root cgroup that can't be migrated anyway.]

I am aware of that. That is why I said root cgroup must have at least one cpu in its "cpuset.cpus.effective".

In that case, let's explicitly describe that condition.

Yes, I will. Only non-root cgroup can distribute out all its CPUs. I thought I said that in the documentation, maybe it is very clear.

Cheers, Longman

Hello, Waiman.

On Tue, Nov 30, 2021 at 10:56:34PM -0500, Waiman Long wrote:

...

What happens if an isolated domain becomes invalid and then valid again due to cpu hotplug? Does it go "root invalid" and then back to "isolated"?

Yes, the current code allow recovering from an invalid state. In this particular case, the transition will be "isolated" --> "root invalid" --> "isolated".

Wouldn't it be clearer if it became "isolated invalid"?

...

While it isn't necessarily tied to this series, it's a big no-no to restrict what a parent can do depending on what its descendants are doing. A cgroup higher up in the hierarchy should be able to change configuration however it sees fit as deligation breaks down otherwise.

Maybe you can argue that cpuset is special and shouldn't be subject to such convention but I can't see strong enough justifications especially given that most of these restrictions can be broken by hotplug operations anyway and thus need code to handle those situations.

These are all pre-existing restrictions before the introduction of partition. These are checks done in validate_change(). I am just saying out loud the existing behavior. If you think that needs to be changed, I am fine with that. However, it will be a separate patch as it is not a behavior that is introduced by this series.

I see. It looks more problematic now with the addtion of the state transition error reporting, more possible state transitions and, well, actual documentation.

Once an invalid partition is changed to "member", there is no way for a child invalid partition root to recover and become valid again. There is why I force them to become "member" also. I am OK if you believe it is better to keep them in the invalid state forever until we explicitly changed them to "member" eventually.

That's because we don't allow turning a cgroup with descendants into a partition, right?

So, when we were first adding the partition support, the thinking was that as it's pretty niche anyway, we can take some aberrations and restrictions, but I don't think it's a good direction to be building up on top of those like this and would much prefer to clean up the rules and restrictions. I know that this has been going on for quite a while and am sorry that am coming back to the same issue repeatedly which isn't necessarily caused by the proposed change. What do you think?

Thanks.

On 12/1/21 13:05, Waiman Long wrote:

On 12/1/21 11:46, Tejun Heo wrote:

...

So, when we were first adding the partition support, the thinking was that as it's pretty niche anyway, we can take some aberrations and restrictions, but I don't think it's a good direction to be building up on top of those like this and would much prefer to clean up the rules and restrictions. I know that this has been going on for quite a while and am sorry that am coming back to the same issue repeatedly which isn't necessarily caused by the proposed change. What do you think?

I think I can relax some of the restrictions, but probably not all of them at this time. We can certainly working on removing as much restriction and limitations as possible in future update to the partition code.

I would say that partition is a cpuset feature that only a minority of users may ever need to use. What I don't want to do is to make the partition feature as general and accommodating as possible and then some of them become dead code that people never use. It won't break binary compatibility to relax or remove limitations in the future. However, imposing new limitation or restriction in the future may not be possible. So I would like to see new use cases evolve that require us to remove the limitations. If that happens, I am happy to update the code to accommodate the new use cases.

For the current use cases of partition that I am aware of, the current limitations as documented will not be a problem for those use cases.

The document below is my latest draft of the document. There are several major changes from the earlier draft:

1) The limitation that "cpuset.cpus" has to be a superset of child's "cpuset.cpus" has been removed as a new patch to remove that limitation will be added.

2) The initial transition from "member" to partition root now requires that "cpuset.cpus" overlap with that of the parent's "cpuset.cpus" instead of being a superset.

3) Now read back of "cpuset.cpus.partition" may return "isolated invalid".

For the transition back to "member", I haven't changed the current wording of forcing child partition roots to become "member" yet. If you think keeping them as invalid partition root is better, I can made that change too.

Please let me know what other changes you would like to see.

Cheers, Longman

----------------------------------------------------------------------------------------

cpuset.cpus.partition     A read-write single value file which exists on non-root     cpuset-enabled cgroups.  This flag is owned by the parent cgroup     and is not delegatable.

It accepts only the following input values when written to.

========    ================================       "member"    Non-root member of a partition       "root"    Partition root       "isolated"    Partition root without load balancing       ========    ================================

The root cgroup is always a partition root and its state     cannot be changed.  All other non-root cgroups start out as     "member".

When set to "root", the current cgroup is the root of a new     partition or scheduling domain that comprises itself and     all its descendants except those that are separate partition     roots themselves and their descendants.

The value shown in "cpuset.cpus.effective" of a partition root is     the CPUs that the parent partition root can dedicate to the new     partition root.  They are subtracted from "cpuset.cpus.effective"     of the parent and may be different from "cpuset.cpus"

When set to "isolated", the CPUs in that partition root will     be in an isolated state without any load balancing from the     scheduler.  Tasks placed in such a partition with multiple     CPUs should be carefully distributed and bound to each of the     individual CPUs for optimal performance.

A partition root ("root" or "isolated") can be in one of the     two possible states - valid or invalid.  An invalid partition     root is in a degraded state where some state information are     retained, but behaves more like a "member".

On read, the "cpuset.cpus.partition" file can show the following     values.

======================    ==============================       "member"            Non-root member of a partition       "root"            Partition root       "isolated"            Partition root without load balancing       "root invalid (<reason>)"    Invalid partition root       "isolated invalid (<reason>)"    Invalid isolated partition root       ======================    ==============================

In the case of an invalid partition root, a descriptive string on     why the partition is invalid is included within parentheses.

Almost all possible state transitions among "member", valid     and invalid partition roots are allowed except from "member"     to invalid partition root.

Before the "member" to partition root transition can happen,     the following conditions must be met or the transition will     not be allowed.

1) The "cpuset.cpus" is non-empty and exclusive, i.e. they are        not shared by any of its siblings.     2) The parent cgroup is a valid partition root.     3) The "cpuset.cpus" must contain at least one of the CPUs from        parent's "cpuset.cpus", i.e. they overlap.     4) There is no child cgroups with cpuset enabled.  This avoids        cpu migrations of multiple cgroups simultaneously which can        be problematic.

Once becoming a partition root, the only rule restricting     changes made to "cpuset.cpus" is the exclusivity rule where     none of the siblings of a partition root can share CPUs with     it.

External events like hotplug or inappropriate changes to     "cpuset.cpus" can cause a valid partition root to become invalid.     Besides the exclusivity rule listed above, the other conditions     required to maintain the validity of a partition root are     as follows:

1) The parent cgroup is a valid partition root.     2) If "cpuset.cpus.effective" is empty, the partition must have        no task associated with it. Otherwise, the partition becomes        invalid and "cpuset.cpus.effective" will fall back to that        of the nearest non-empty ancestor.

A corollary of a valid partition root is that     "cpuset.cpus.effective" is always a subset of "cpuset.cpus".     Note that a task cannot be moved to a cgroup with empty     "cpuset.cpus.effective".

Changing a partition root (valid or invalid) to "member" is     always allowed.  If there are child partition roots underneath     it, however, they will be forced to be switched back to "member"     too and lose their partitions. So care must be taken to double     check for this condition before disabling a partition root.

A valid non-root parent partition may distribute out all its CPUs     to its child partitions when there is no task associated with it.

An invalid partition root can be reverted back to a valid one     if none of the validity constraints of a valid partition root     are violated.

Poll and inotify events are triggered whenever the state of     "cpuset.cpus.partition" changes.  That includes changes caused by     write to "cpuset.cpus.partition", cpu hotplug and other changes     that make the partition invalid.  This will allow user space     agents to monitor unexpected changes to "cpuset.cpus.partition"     without the need to do continuous polling.

On 12/3/21 13:25, Michal Koutný wrote:

Hello Longman.

On Wed, Dec 01, 2021 at 08:28:09PM -0500, Waiman Long longman@redhat.com wrote:

...

1. The limitation that "cpuset.cpus" has to be a superset of child's

"cpuset.cpus" has been removed as a new patch to remove that limitation will be added.

Superb!

...

2. The initial transition from "member" to partition root now requires that

"cpuset.cpus" overlap with that of the parent's "cpuset.cpus" instead of being a superset.

That's sensible.

...

For the transition back to "member", I haven't changed the current wording of forcing child partition roots to become "member" yet. If you think keeping them as invalid partition root is better, I can made that change too.

I wrote I was indifferent about this in a previous mail but when I think about it now, switching to invalid root is perhaps better than switching to member since it'd effectively mean that modifications of the parent config propagate (permanently) also to a descendant config, which is an undesired v1-ism.

That makes sense. I will keep those child partitions in an invalid state then.

...

Please let me know what other changes you would like to see.

I hope my remarks below are just clarifications and not substantial changes. Besides that I find your new draft good. Thanks!

...

[...]     An invalid partition root can be reverted back to a valid one     if none of the validity constraints of a valid partition root     are violated.

s/can be/will be/

(I understand the intention is to make it asynchronously and automatically, i.e. without writing into the affected descendant(s) cpuset.partition again.)

Yes, that will be automatic and the partition will become valid again if other events cause changes that unbreak the validity constraints.

...

Poll and inotify events are triggered whenever the state of     "cpuset.cpus.partition" changes.  That includes changes caused by     write to "cpuset.cpus.partition", cpu hotplug and other changes     that make the partition invalid.

-> that change validity status

(In accordance with the comment above.)

Cheers, Longman

On Wed, Nov 10, 2021 at 12:13:57PM +0100, Felix Moessbauer wrote:

Hi Weiman,

...

v8:

• Reorganize the patch series and rationalize the features and constraints of a partition.
• Update patch descriptions and documentation accordingly.

v7:

• Simplify the documentation patch (patch 5) as suggested by Tejun.
• Fix a typo in patch 2 and improper commit log in patch 3.

v6:

• Remove duplicated tmpmask from update_prstate() which should fix the frame size too large problem reported by kernel test robot.

This patchset makes four enhancements to the cpuset v2 code.

Patch 1: Enable partition with no task to have empty cpuset.cpus.effective.

Patch 2: Refining the features and constraints of a cpuset partition clarifying what changes are allowed.

Patch 3: Add a new partition state "isolated" to create a partition root without load balancing. This is for handling intermitten workloads that have a strict low latency requirement.

I just tested this patch-series and can confirm that it works on 5.15.0-rc7-rt15 (PREEMT_RT).

However, I was not able to see any latency improvements when using cpuset.cpus.partition=isolated. The test was performed with jitterdebugger on CPUs 1-3 and the following cmdline: rcu_nocbs=1-4 nohz_full=1-4 irqaffinity=0,5-6,11 intel_pstate=disable On the other cpus, stress-ng was executed to generate load.

enum hk_flags { HK_FLAG_TIMER = 1, HK_FLAG_RCU = (1 << 1), HK_FLAG_MISC = (1 << 2), HK_FLAG_SCHED = (1 << 3), HK_FLAG_TICK = (1 << 4), HK_FLAG_DOMAIN = (1 << 5), HK_FLAG_WQ = (1 << 6), HK_FLAG_MANAGED_IRQ = (1 << 7), HK_FLAG_KTHREAD = (1 << 8), };

static int __init housekeeping_nohz_full_setup(char *str) { unsigned int flags;

flags = HK_FLAG_TICK | HK_FLAG_WQ | HK_FLAG_TIMER | HK_FLAG_RCU | HK_FLAG_MISC | HK_FLAG_KTHREAD;

return housekeeping_setup(str, flags); } __setup("nohz_full=", housekeeping_nohz_full_setup);

So HK_FLAG_SCHED and HK_FLAG_MANAGED_IRQ are unset in your configuration. Perhaps they are affecting your latency numbers?

This tool might be handy to see what is the reason for the latency source:

https://github.com/xzpeter/rt-trace-bpf

./rt-trace-bcc.py -c isolated-cpu

Just some more general notes:

Even with this new "isolated" type, it is still very tricky to get a similar behavior as with isolcpus (as long as I don't miss something here):

Consider an RT application that consists of a non-rt thread that should be floating and a rt-thread that should be placed in the isolated domain. This requires cgroup.type=threaded on both cgroups and changes to the application (threads have to be born in non-rt group and moved to rt-group).

Theoretically, this could be done externally, but in case the application sets the affinity mask manually, you run into a timing issue (setting affinities to CPUs outside the current cpuset.cpus results in EINVAL).

Best regards, Felix Moessbauer Siemens AG

...

Patch 4: Enable the "cpuset.cpus.partition" file to show the reason that causes invalid partition like "root invalid (No cpu available due to hotplug)".

Patch 5 updates the cgroup-v2.rst file accordingly. Patch 6 adds a new cpuset test to test the new cpuset partition code.

-----Original Message----- From: Michal Koutný mkoutny@suse.com Sent: Wednesday, November 10, 2021 2:57 PM To: Moessbauer, Felix (T RDA IOT SES-DE) felix.moessbauer@siemens.com Cc: longman@redhat.com; akpm@linux-foundation.org; cgroups@vger.kernel.org; corbet@lwn.net; frederic@kernel.org; guro@fb.com; hannes@cmpxchg.org; juri.lelli@redhat.com; linux-doc@vger.kernel.org; linux- kernel@vger.kernel.org; linux-kselftest@vger.kernel.org; lizefan.x@bytedance.com; mtosatti@redhat.com; pauld@redhat.com; peterz@infradead.org; shuah@kernel.org; tj@kernel.org; Kiszka, Jan (T RDA IOT) jan.kiszka@siemens.com; Schild, Henning (T RDA IOT SES-DE) henning.schild@siemens.com Subject: Re: [PATCH v8 0/6] cgroup/cpuset: Add new cpuset partition type & empty effecitve cpus

Hello.

On Wed, Nov 10, 2021 at 12:13:57PM +0100, Felix Moessbauer felix.moessbauer@siemens.com wrote:

...

However, I was not able to see any latency improvements when using cpuset.cpus.partition=isolated.

Interesting. What was the baseline against which you compared it (isolcpus, no cpusets,...)?

For this test, I just compared both settings cpuset.cpus.partition=isolated|root. There, I did not see a significant difference (but I know, RT tuning depends on a ton of things).

...

The test was performed with jitterdebugger on CPUs 1-3 and the following

cmdline:

...

rcu_nocbs=1-4 nohz_full=1-4 irqaffinity=0,5-6,11 intel_pstate=disable On the other cpus, stress-ng was executed to generate load. [...]

...

This requires cgroup.type=threaded on both cgroups and changes to the application (threads have to be born in non-rt group and moved to rt-group).

But even with isolcpus the application would need to set affinity of threads to the selected CPUs (cf cgroup migrating). Do I miss anything?

Yes, that's true. But there are two differences (given that you use isolcpus): 1. the application only has to set the affinity for rt threads. Threads that do not explicitly set the affinity are automatically excluded from the isolated cores. Even common rt test applications like jitterdebugger do not pin their non-rt threads. 2. Threads can be started on non-rt CPUs and then bound to a specific rt CPU. This binding can be specified before thread creation via pthread_create. By that, you can make sure that at no point in time a thread has a "forbidden" CPU in its affinities.

With cgroup2, you cannot guarantee the second aspect, as thread creation and moving to a cgroup is not an atomic operation. Also - please correct me if I'm wrong - you first have to create a thread before moving it into a group. At creation time, you cannot set the final affinity mask (as you create it in the non-rt group and there the CPU is not in the cpuset.cpus). Once you move the thread to the rt cgroup, it has a default mask and by that can be executed on other rt cores.

Best regards, Felix

Thanks, Michal

On Wed, Nov 10, 2021 at 01:10:20PM -0300, Marcelo Tosatti wrote:

On Wed, Nov 10, 2021 at 03:21:54PM +0000, Moessbauer, Felix wrote:

...

...

-----Original Message----- From: Michal Koutný mkoutny@suse.com Sent: Wednesday, November 10, 2021 2:57 PM To: Moessbauer, Felix (T RDA IOT SES-DE) felix.moessbauer@siemens.com Cc: longman@redhat.com; akpm@linux-foundation.org; cgroups@vger.kernel.org; corbet@lwn.net; frederic@kernel.org; guro@fb.com; hannes@cmpxchg.org; juri.lelli@redhat.com; linux-doc@vger.kernel.org; linux- kernel@vger.kernel.org; linux-kselftest@vger.kernel.org; lizefan.x@bytedance.com; mtosatti@redhat.com; pauld@redhat.com; peterz@infradead.org; shuah@kernel.org; tj@kernel.org; Kiszka, Jan (T RDA IOT) jan.kiszka@siemens.com; Schild, Henning (T RDA IOT SES-DE) henning.schild@siemens.com Subject: Re: [PATCH v8 0/6] cgroup/cpuset: Add new cpuset partition type & empty effecitve cpus

Hello.

On Wed, Nov 10, 2021 at 12:13:57PM +0100, Felix Moessbauer felix.moessbauer@siemens.com wrote:

...

However, I was not able to see any latency improvements when using cpuset.cpus.partition=isolated.

Interesting. What was the baseline against which you compared it (isolcpus, no cpusets,...)?

For this test, I just compared both settings cpuset.cpus.partition=isolated|root. There, I did not see a significant difference (but I know, RT tuning depends on a ton of things).

...

...

The test was performed with jitterdebugger on CPUs 1-3 and the following

cmdline:

...

rcu_nocbs=1-4 nohz_full=1-4 irqaffinity=0,5-6,11 intel_pstate=disable On the other cpus, stress-ng was executed to generate load. [...]

...

This requires cgroup.type=threaded on both cgroups and changes to the application (threads have to be born in non-rt group and moved to rt-group).

But even with isolcpus the application would need to set affinity of threads to the selected CPUs (cf cgroup migrating). Do I miss anything?

Yes, that's true. But there are two differences (given that you use isolcpus):

1. the application only has to set the affinity for rt threads.

Threads that do not explicitly set the affinity are automatically excluded from the isolated cores. Even common rt test applications like jitterdebugger do not pin their non-rt threads. 2. Threads can be started on non-rt CPUs and then bound to a specific rt CPU. This binding can be specified before thread creation via pthread_create. By that, you can make sure that at no point in time a thread has a "forbidden" CPU in its affinities.

With cgroup2, you cannot guarantee the second aspect, as thread creation and moving to a cgroup is not an atomic operation. Also - please correct me if I'm wrong - you first have to create a thread before moving it into a group. At creation time, you cannot set the final affinity mask (as you create it in the non-rt group and there the CPU is not in the cpuset.cpus). Once you move the thread to the rt cgroup, it has a default mask and by that can be executed on other rt cores.

man clone3:

   CLONE_NEWCGROUP (since Linux 4.6)
          Create  the  process  in  a  new cgroup namespace.  If this flag is not set, then (as with fork(2)) the
          process is created in the same cgroup namespaces as the calling process.

          For further information on cgroup namespaces, see cgroup_namespaces(7).

          Only a privileged process (CAP_SYS_ADMIN) can employ CLONE_NEWCGROUP.

Err, CLONE_INTO_CGROUP.

On Wed, Nov 10, 2021 at 05:15:41PM +0100, Jan Kiszka wrote:

On 10.11.21 17:10, Marcelo Tosatti wrote:

...

On Wed, Nov 10, 2021 at 03:21:54PM +0000, Moessbauer, Felix wrote:

...

...

-----Original Message----- From: Michal Koutný mkoutny@suse.com Sent: Wednesday, November 10, 2021 2:57 PM To: Moessbauer, Felix (T RDA IOT SES-DE) felix.moessbauer@siemens.com Cc: longman@redhat.com; akpm@linux-foundation.org; cgroups@vger.kernel.org; corbet@lwn.net; frederic@kernel.org; guro@fb.com; hannes@cmpxchg.org; juri.lelli@redhat.com; linux-doc@vger.kernel.org; linux- kernel@vger.kernel.org; linux-kselftest@vger.kernel.org; lizefan.x@bytedance.com; mtosatti@redhat.com; pauld@redhat.com; peterz@infradead.org; shuah@kernel.org; tj@kernel.org; Kiszka, Jan (T RDA IOT) jan.kiszka@siemens.com; Schild, Henning (T RDA IOT SES-DE) henning.schild@siemens.com Subject: Re: [PATCH v8 0/6] cgroup/cpuset: Add new cpuset partition type & empty effecitve cpus

Hello.

On Wed, Nov 10, 2021 at 12:13:57PM +0100, Felix Moessbauer felix.moessbauer@siemens.com wrote:

...

However, I was not able to see any latency improvements when using cpuset.cpus.partition=isolated.

Interesting. What was the baseline against which you compared it (isolcpus, no cpusets,...)?

For this test, I just compared both settings cpuset.cpus.partition=isolated|root. There, I did not see a significant difference (but I know, RT tuning depends on a ton of things).

...

...

The test was performed with jitterdebugger on CPUs 1-3 and the following

cmdline:

...

rcu_nocbs=1-4 nohz_full=1-4 irqaffinity=0,5-6,11 intel_pstate=disable On the other cpus, stress-ng was executed to generate load. [...]

...

This requires cgroup.type=threaded on both cgroups and changes to the application (threads have to be born in non-rt group and moved to rt-group).

But even with isolcpus the application would need to set affinity of threads to the selected CPUs (cf cgroup migrating). Do I miss anything?

Yes, that's true. But there are two differences (given that you use isolcpus):

1. the application only has to set the affinity for rt threads.

Threads that do not explicitly set the affinity are automatically excluded from the isolated cores. Even common rt test applications like jitterdebugger do not pin their non-rt threads. 2. Threads can be started on non-rt CPUs and then bound to a specific rt CPU. This binding can be specified before thread creation via pthread_create. By that, you can make sure that at no point in time a thread has a "forbidden" CPU in its affinities.

With cgroup2, you cannot guarantee the second aspect, as thread creation and moving to a cgroup is not an atomic operation. Also - please correct me if I'm wrong - you first have to create a thread before moving it into a group. At creation time, you cannot set the final affinity mask (as you create it in the non-rt group and there the CPU is not in the cpuset.cpus). Once you move the thread to the rt cgroup, it has a default mask and by that can be executed on other rt cores.

man clone3:

   CLONE_NEWCGROUP (since Linux 4.6)
          Create  the  process  in  a  new cgroup namespace.  If this flag is not set, then (as with fork(2)) the
          process is created in the same cgroup namespaces as the calling process.

          For further information on cgroup namespaces, see cgroup_namespaces(7).

          Only a privileged process (CAP_SYS_ADMIN) can employ CLONE_NEWCGROUP.

Is there pthread_attr_setcgroup_np()?

Jan

Don't know... Waiman?

On Wed, Nov 10, 2021 at 05:15:41PM +0100, Jan Kiszka jan.kiszka@siemens.com wrote:

Is there pthread_attr_setcgroup_np()?

If I'm not mistaken the 'p' in pthreads stands for POSIX and cgroups are Linux specific so you won't find that (unless you implement that yourself). ¯_(ツ)_/¯

Michal

On Wed, Nov 10, 2021 at 03:21:54PM +0000, "Moessbauer, Felix" felix.moessbauer@siemens.com wrote:

2. Threads can be started on non-rt CPUs and then bound to a specific rt CPU.

This binding can be specified before thread creation via pthread_create. By that, you can make sure that at no point in time a thread has a "forbidden" CPU in its affinities.

It should boil down to some clone$version(2) and sched_setaffinity(2) calls, so strictly speaking even with pthread_create(3) the thread is shortly running with the parent's affinity.

With cgroup2, you cannot guarantee the second aspect, as thread creation and moving to a cgroup is not an atomic operation.

As suggested by others, CLONE_INTO_CGROUP (into cpuset cgroup) can actually "hide" the migration into the clone3() call.

At creation time, you cannot set the final affinity mask (as you create it in the non-rt group and there the CPU is not in the cpuset.cpus). Once you move the thread to the rt cgroup, it has a default mask and by that can be executed on other rt cores.

Good point. Perhaps you could work this around by having another level of (non-root partition) cpuset cgroups for individual CPUs? (Maybe there's more clever approach, this is just first to come into my mind.)

Michal