On 28 June 2016 at 15:58, Leo Yan leo.yan@linaro.org wrote:
On Mon, Jun 27, 2016 at 10:56:57PM +0200, Vincent Guittot wrote:
On 23 June 2016 at 15:43, Leo Yan leo.yan@linaro.org wrote:
Current code calculates avg_load as below: sgs->avg_load = (sgs->group_load*SCHED_CAPACITY_SCALE) / sgs->group_capacity
Let's see below scenario for cluster level average load calculation:
The little cluster have 4 CPUs and 2 tasks running with 100% utilization (nice = 0); if little cluster capacity is 400, then get little cluster avg_load = (2 * 1024 * 1024) / (400 * 4) = 1310.
On the other hand, if big cluster have 4 CPUs and 4 tasks running with 100% utilization (nice = 0); if big cluster capacity is 1024, then get big cluster avg_load = (4 * 1024 * 1024) / (1024 * 4) = 1024.
So finally scheduler considers little cluster has higher load and this obviously doesn't make sense due big cluster has 4 CPUs been running but little cluster actually have 2 CPUs are idle.
This perfectly makes sense, it's all about how much computation capacity has to be shared between load so even if 2 little cpu are idle, the tasks on big cluster have more compute capacity than those on little cluster.
I'm not sure if I have done enough homework for this :) My understanding for load value is the task requirement for CPU computation capacity, the range is [0..1024]. So when reach 1024
load value is as an enhancement of task's weight that was used before in the scheduler. The task's weight is now pondered by the runnable time of the task so a task with a short runnable time will not make a CPU seen as heavily loaded just because of thsi short high prio task. This load is then used to ensure a fair share of the CPUs compute capacity between the tasks The range is not [0..1024] but [0..88761]
meaning all CPU capacity has been consumed. avg_load is the average
Only the utilization of the cpu means that all CPU capacity has been consumed but not the load which is used for fair time sharing between tasks
value for sched_group's all CPUs.
And avg_load is a singal for CPU capacity consuming but not for task. Please free correct me if I misunderstand for this.
I gave the example which is not quite typical. Let's see more practical example:
Big cluster has 4 CPUs with 6 runnable tasks, group_load = 3547; so big cluster avg_load = (3547 * 1024) / 4096 = 886;
Little cluster has 4 CPUs with 2 runnable tasks, group_load = 1639;, so little cluster avg_load = (1639 * 1024 / 1603 = 1046;
So how we calculate imbalance load between these two clusters? In current code, even big cluster is overloaded but it will not migrate tasks to little cluser:
In this case we use the fact the big cluster is overloaded but not little cluster and the load_per_task value In mainline kernel, this is not handle correctly but I thought that this was handled by EAS code; There is this code below in calculate_imbalance that matches with your condition
/* * Busiest group is overloaded, local is not, use the spare * cycles to maximize throughput */ if (busiest->group_type == group_overloaded && local->group_type <= group_misfit_task) { env->imbalance = busiest->load_per_task; return; }
7788 /* 7789 * If the local group is busier than the selected busiest group 7790 * don't try and pull any tasks. 7791 */ 7792 if (local->avg_load >= busiest->avg_load) 7793 goto out_balanced;
We can't use this avg_load into account only when big cluster becomes overloaded whereas little cpu has some idle cpus.
Even we can add one extra checking for if big cluster is overloaded, then question is how many imbalance load should be migrated from big cluster to little cluster? For upper case, we can see avg_load value is pointless. So this is why I tried to write this patch to change avg_load to represent the concept for "CPU capacity consuming ratio".
So this is caused by the load value is not scaled by capacity, so it will get wrong average load value by divide capacity value. So just need simply divide CPU number (group->group_weight). So for upper case we can get little cluster avg_load = (2 * 1024) / 4 = 512; big cluster avg_load = (4 * 1024) / 4 = 1024.
Signed-off-by: Leo Yan leo.yan@linaro.org
kernel/sched/fair.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a6eef88..353520d 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5475,7 +5475,7 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, }
/* Adjust by relative CPU capacity of the group */
avg_load = (avg_load * SCHED_CAPACITY_SCALE) / group->sgc->capacity;
avg_load = avg_load / group->group_weight; if (local_group) { this_load = avg_load;@@ -7250,7 +7250,7 @@ static inline void update_sg_lb_stats(struct lb_env *env,
/* Adjust by relative CPU capacity of the group */ sgs->group_capacity = group->sgc->capacity;
sgs->avg_load = (sgs->group_load*SCHED_CAPACITY_SCALE) / sgs->group_capacity;
sgs->avg_load = sgs->group_load / group->group_weight; if (sgs->sum_nr_running) sgs->load_per_task = sgs->sum_weighted_load / sgs->sum_nr_running;-- 1.9.1