This isn't in the fastpath, it's in the rebalancing logic.
the reality is much more complex unfortunately. C and P states hang together tightly, and even C state on one core impacts other cores' performance, just like P state selection on one core impacts other cores.
(at least for x86, we should really stop talking as if the OS picks the "frequency", that's just not the case anymore)
I agree, the reality is very complex. But we should go back and analyse what problem we are trying to solve, what each framework is trying to address.
When viewed separately from the scheduler, cpufreq and cpuidle governors do the right thing. But they both base their action on the CPU load (balance) decided by the scheduler and it's the latter that we are trying to adjust (and we are still debating what the right approach is).
Since such information seems too complex to be moved into the scheduler, why don't we get cpufreq in charge of restricting the load balancing to certain CPUs? It already tracks the load/idle time to (gradually) change the P state. Depending on the governor/policy, it could decide that (for
(btw in case you missed it, for Intel HW we no longer use cpufreq anymore)
Cpuidle I think for now can stay the same, gradually entering deeper sleep states. It could be later unified with cpufreq if there are any benefits. In deciding the load balancing restrictions, maybe cpufreq should be aware of C-state latencies.
on the Intel side, we're likely to merge the Intel idle driver and P state driver in the near future fwiw. We'll keep using cpuidle framework (since it doesn't do all that much other than provide a nice hook for the idle loop), but we likely will make a hw specific selection logic there.
I do agree the scheduler needs to get integrated a bit better, in that it has some better knowledge, and to be honest, we likely need to switch from giving tasks credit for "time consumed" to giving them credit for something like "cycles consumed" or "instructions executed" or a mix thereof. So that a task that runs on a slower CPU (for either policy choice reasons or due to hardware capabilities), it gets charged less than when it runs fast.