Currently, there can't be multiple instances of single governor_type. If we have a multi-package system, where we have multiple instances of struct policy (per package), we can't have multiple instances of same governor. i.e. We can't have multiple instances of ondemand governor for multiple packages.

Governors directory in sysfs is created at /sys/devices/system/cpu/cpufreq/ governor-name/. Which again reflects that there can be only one instance of a governor_type in the system.

This is a bottleneck for multicluster system, where we want different packages to use same governor type, but with different tunables.

This patch is inclined towards providing this infrastructure. Because we are required to allocate governor's resources dynamically now, we must do it at policy creation and end. And so got CPUFREQ_GOV_POLICY_INIT/EXIT.

Signed-off-by: Viresh Kumar viresh.kumar@linaro.org Signed-off-by: Rafael J. Wysocki rafael.j.wysocki@intel.com --- drivers/cpufreq/cpufreq.c | 21 ++++++++++++++++++--- include/linux/cpufreq.h | 9 ++++++--- 2 files changed, 24 insertions(+), 6 deletions(-)

diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index c5996fe..08df7a1 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -1070,6 +1070,8 @@ static int __cpufreq_remove_dev(struct device *dev, struct subsys_interface *sif

/* If cpu is last user of policy, free policy */ if (cpus == 1) { + __cpufreq_governor(data, CPUFREQ_GOV_POLICY_EXIT); + lock_policy_rwsem_read(cpu); kobj = &data->kobj; cmp = &data->kobj_unregister; @@ -1651,7 +1653,7 @@ EXPORT_SYMBOL(cpufreq_get_policy); static int __cpufreq_set_policy(struct cpufreq_policy *data, struct cpufreq_policy *policy) { - int ret = 0; + int ret = 0, failed = 1;

pr_debug("setting new policy for CPU %u: %u - %u kHz\n", policy->cpu, policy->min, policy->max); @@ -1705,18 +1707,31 @@ static int __cpufreq_set_policy(struct cpufreq_policy *data, pr_debug("governor switch\n");

/* end old governor */ - if (data->governor) + if (data->governor) { __cpufreq_governor(data, CPUFREQ_GOV_STOP); + __cpufreq_governor(data, + CPUFREQ_GOV_POLICY_EXIT); + }

/* start new governor */ data->governor = policy->governor; - if (__cpufreq_governor(data, CPUFREQ_GOV_START)) { + if (!__cpufreq_governor(data, CPUFREQ_GOV_POLICY_INIT)) { + if (!__cpufreq_governor(data, CPUFREQ_GOV_START)) + failed = 0; + else + __cpufreq_governor(data, + CPUFREQ_GOV_POLICY_EXIT); + } + + if (failed) { /* new governor failed, so re-start old one */ pr_debug("starting governor %s failed\n", data->governor->name); if (old_gov) { data->governor = old_gov; __cpufreq_governor(data, + CPUFREQ_GOV_POLICY_INIT); + __cpufreq_governor(data, CPUFREQ_GOV_START); } ret = -EINVAL; diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h index 1382866..848b38b 100644 --- a/include/linux/cpufreq.h +++ b/include/linux/cpufreq.h @@ -106,6 +106,7 @@ struct cpufreq_policy { * governors are used */ unsigned int policy; /* see above */ struct cpufreq_governor *governor; /* see below */ + void *governor_data;

struct work_struct update; /* if update_policy() needs to be * called, but you're in IRQ context */ @@ -178,9 +179,11 @@ static inline unsigned long cpufreq_scale(unsigned long old, u_int div, u_int mu * CPUFREQ GOVERNORS * *********************************************************************/

-#define CPUFREQ_GOV_START 1 -#define CPUFREQ_GOV_STOP 2 -#define CPUFREQ_GOV_LIMITS 3 +#define CPUFREQ_GOV_START 1 +#define CPUFREQ_GOV_STOP 2 +#define CPUFREQ_GOV_LIMITS 3 +#define CPUFREQ_GOV_POLICY_INIT 4 +#define CPUFREQ_GOV_POLICY_EXIT 5

struct cpufreq_governor { char name[CPUFREQ_NAME_LEN];

__cpufreq_governor() must be called with a correct policy->cpus mask. In __cpufreq_remove_dev() we initially clear policy->cpus with cpumask_clear_cpu() and then call __cpufreq_governor(policy, CPUFREQ_GOV_POLICY_EXIT). If the governor is doing some per-cpu stuff in EXIT callback, this can create uncertain behavior.

Generic governors in drivers/cpufreq/ doesn't do any per-cpu stuff in EXIT callback and so we don't face any issues currently. But its better to keep the code clean, so we don't face any issues in future.

Now, we call cpumask_clear_cpu() only when multiple cpus are managed by policy.

Signed-off-by: Viresh Kumar viresh.kumar@linaro.org Signed-off-by: Rafael J. Wysocki rafael.j.wysocki@intel.com --- drivers/cpufreq/cpufreq.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index 85963fc..0072b46 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -1036,7 +1036,9 @@ static int __cpufreq_remove_dev(struct device *dev, struct subsys_interface *sif

WARN_ON(lock_policy_rwsem_write(cpu)); cpus = cpumask_weight(data->cpus); - cpumask_clear_cpu(cpu, data->cpus); + + if (cpus > 1) + cpumask_clear_cpu(cpu, data->cpus); unlock_policy_rwsem_write(cpu);

if (cpu != data->cpu) {

There are two types of INIT/EXIT activities that we need to do for governors: - Done only once per governor (doesn't depend how many instances of the governor there are). eg: cpufreq_register_notifier() for conservative governor. - Done per governor instance, eg: sysfs_{create|remove}_group().

There were some corner cases where current code isn't able to handle them separately and so failing for some test cases.

We use two separate variables now for keeping track of above two requirements. - governor->initialized for first one - dbs_data->usage_count for per governor instance

Signed-off-by: Viresh Kumar viresh.kumar@linaro.org Signed-off-by: Rafael J. Wysocki rafael.j.wysocki@intel.com --- drivers/cpufreq/cpufreq_governor.c | 11 +++++++---- drivers/cpufreq/cpufreq_governor.h | 1 + 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_governor.c b/drivers/cpufreq/cpufreq_governor.c index 26fbb72..38f959b 100644 --- a/drivers/cpufreq/cpufreq_governor.c +++ b/drivers/cpufreq/cpufreq_governor.c @@ -250,6 +250,7 @@ int cpufreq_governor_dbs(struct cpufreq_policy *policy, if (have_governor_per_policy()) { WARN_ON(dbs_data); } else if (dbs_data) { + dbs_data->usage_count++; policy->governor_data = dbs_data; return 0; } @@ -261,6 +262,7 @@ int cpufreq_governor_dbs(struct cpufreq_policy *policy, }

dbs_data->cdata = cdata; + dbs_data->usage_count = 1; rc = cdata->init(dbs_data); if (rc) { pr_err("%s: POLICY_INIT: init() failed\n", __func__); @@ -289,7 +291,8 @@ int cpufreq_governor_dbs(struct cpufreq_policy *policy, set_sampling_rate(dbs_data, max(dbs_data->min_sampling_rate, latency * LATENCY_MULTIPLIER));

- if (dbs_data->cdata->governor == GOV_CONSERVATIVE) { + if ((cdata->governor == GOV_CONSERVATIVE) && + (!policy->governor->initialized)) { struct cs_ops *cs_ops = dbs_data->cdata->gov_ops;

cpufreq_register_notifier(cs_ops->notifier_block, @@ -301,12 +304,12 @@ int cpufreq_governor_dbs(struct cpufreq_policy *policy,

return 0; case CPUFREQ_GOV_POLICY_EXIT: - if ((policy->governor->initialized == 1) || - have_governor_per_policy()) { + if (!--dbs_data->usage_count) { sysfs_remove_group(get_governor_parent_kobj(policy), get_sysfs_attr(dbs_data));

- if (dbs_data->cdata->governor == GOV_CONSERVATIVE) { + if ((dbs_data->cdata->governor == GOV_CONSERVATIVE) && + (policy->governor->initialized == 1)) { struct cs_ops *cs_ops = dbs_data->cdata->gov_ops;

cpufreq_unregister_notifier(cs_ops->notifier_block, diff --git a/drivers/cpufreq/cpufreq_governor.h b/drivers/cpufreq/cpufreq_governor.h index 3570d09..8998d3d 100644 --- a/drivers/cpufreq/cpufreq_governor.h +++ b/drivers/cpufreq/cpufreq_governor.h @@ -212,6 +212,7 @@ struct common_dbs_data { struct dbs_data { struct common_dbs_data *cdata; unsigned int min_sampling_rate; + int usage_count; void *tuners;

/* dbs_mutex protects dbs_enable in governor start/stop */

From: Stratos Karafotis stratosk@semaphore.gr

Currently we always calculate the CPU iowait time and add it to idle time. If we are in ondemand and we use io_is_busy, we re-calculate iowait time and we subtract it from idle time.

With this patch iowait time is calculated only when necessary avoiding the double call to get_cpu_iowait_time_us. We use a parameter in function get_cpu_idle_time to distinguish when the iowait time will be added to idle time or not, without the need of keeping the prev_io_wait.

Signed-off-by: Stratos Karafotis stratosk@semaphore.gr Acked-by: Viresh Kumar <viresh.kumar@linaro.,org> Signed-off-by: Rafael J. Wysocki rafael.j.wysocki@intel.com --- drivers/cpufreq/cpufreq_conservative.c | 2 +- drivers/cpufreq/cpufreq_governor.c | 48 +++++++++++++--------------------- drivers/cpufreq/cpufreq_governor.h | 3 +-- drivers/cpufreq/cpufreq_ondemand.c | 11 +++++++- 4 files changed, 30 insertions(+), 34 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_conservative.c b/drivers/cpufreq/cpufreq_conservative.c index 98b4946..2465e63 100644 --- a/drivers/cpufreq/cpufreq_conservative.c +++ b/drivers/cpufreq/cpufreq_conservative.c @@ -238,7 +238,7 @@ static ssize_t store_ignore_nice(struct dbs_data *dbs_data, const char *buf, struct cs_cpu_dbs_info_s *dbs_info; dbs_info = &per_cpu(cs_cpu_dbs_info, j); dbs_info->cdbs.prev_cpu_idle = get_cpu_idle_time(j, - &dbs_info->cdbs.prev_cpu_wall); + &dbs_info->cdbs.prev_cpu_wall, 0); if (cs_tuners->ignore_nice) dbs_info->cdbs.prev_cpu_nice = kcpustat_cpu(j).cpustat[CPUTIME_NICE]; diff --git a/drivers/cpufreq/cpufreq_governor.c b/drivers/cpufreq/cpufreq_governor.c index 38f959b..13ca17a 100644 --- a/drivers/cpufreq/cpufreq_governor.c +++ b/drivers/cpufreq/cpufreq_governor.c @@ -67,13 +67,13 @@ static inline u64 get_cpu_idle_time_jiffy(unsigned int cpu, u64 *wall) return cputime_to_usecs(idle_time); }

-u64 get_cpu_idle_time(unsigned int cpu, u64 *wall) +u64 get_cpu_idle_time(unsigned int cpu, u64 *wall, int io_busy) { - u64 idle_time = get_cpu_idle_time_us(cpu, NULL); + u64 idle_time = get_cpu_idle_time_us(cpu, io_busy ? wall : NULL);

if (idle_time == -1ULL) return get_cpu_idle_time_jiffy(cpu, wall); - else + else if (!io_busy) idle_time += get_cpu_iowait_time_us(cpu, wall);

return idle_time; @@ -100,13 +100,22 @@ void dbs_check_cpu(struct dbs_data *dbs_data, int cpu) /* Get Absolute Load (in terms of freq for ondemand gov) */ for_each_cpu(j, policy->cpus) { struct cpu_dbs_common_info *j_cdbs; - u64 cur_wall_time, cur_idle_time, cur_iowait_time; - unsigned int idle_time, wall_time, iowait_time; + u64 cur_wall_time, cur_idle_time; + unsigned int idle_time, wall_time; unsigned int load; + int io_busy = 0;

j_cdbs = dbs_data->cdata->get_cpu_cdbs(j);

- cur_idle_time = get_cpu_idle_time(j, &cur_wall_time); + /* + * For the purpose of ondemand, waiting for disk IO is + * an indication that you're performance critical, and + * not that the system is actually idle. So do not add + * the iowait time to the cpu idle time. + */ + if (dbs_data->cdata->governor == GOV_ONDEMAND) + io_busy = od_tuners->io_is_busy; + cur_idle_time = get_cpu_idle_time(j, &cur_wall_time, io_busy);

wall_time = (unsigned int) (cur_wall_time - j_cdbs->prev_cpu_wall); @@ -134,29 +143,6 @@ void dbs_check_cpu(struct dbs_data *dbs_data, int cpu) idle_time += jiffies_to_usecs(cur_nice_jiffies); }

- if (dbs_data->cdata->governor == GOV_ONDEMAND) { - struct od_cpu_dbs_info_s *od_j_dbs_info = - dbs_data->cdata->get_cpu_dbs_info_s(cpu); - - cur_iowait_time = get_cpu_iowait_time_us(j, - &cur_wall_time); - if (cur_iowait_time == -1ULL) - cur_iowait_time = 0; - - iowait_time = (unsigned int) (cur_iowait_time - - od_j_dbs_info->prev_cpu_iowait); - od_j_dbs_info->prev_cpu_iowait = cur_iowait_time; - - /* - * For the purpose of ondemand, waiting for disk IO is - * an indication that you're performance critical, and - * not that the system is actually idle. So subtract the - * iowait time from the cpu idle time. - */ - if (od_tuners->io_is_busy && idle_time >= iowait_time) - idle_time -= iowait_time; - } - if (unlikely(!wall_time || wall_time < idle_time)) continue;

@@ -236,6 +222,7 @@ int cpufreq_governor_dbs(struct cpufreq_policy *policy, struct cs_dbs_tuners *cs_tuners = NULL; struct cpu_dbs_common_info *cpu_cdbs; unsigned int sampling_rate, latency, ignore_nice, j, cpu = policy->cpu; + int io_busy = 0; int rc;

if (have_governor_per_policy()) @@ -338,6 +325,7 @@ int cpufreq_governor_dbs(struct cpufreq_policy *policy, sampling_rate = od_tuners->sampling_rate; ignore_nice = od_tuners->ignore_nice; od_ops = dbs_data->cdata->gov_ops; + io_busy = od_tuners->io_is_busy; }

switch (event) { @@ -354,7 +342,7 @@ int cpufreq_governor_dbs(struct cpufreq_policy *policy, j_cdbs->cpu = j; j_cdbs->cur_policy = policy; j_cdbs->prev_cpu_idle = get_cpu_idle_time(j, - &j_cdbs->prev_cpu_wall); + &j_cdbs->prev_cpu_wall, io_busy); if (ignore_nice) j_cdbs->prev_cpu_nice = kcpustat_cpu(j).cpustat[CPUTIME_NICE]; diff --git a/drivers/cpufreq/cpufreq_governor.h b/drivers/cpufreq/cpufreq_governor.h index 8998d3d..eae3163 100644 --- a/drivers/cpufreq/cpufreq_governor.h +++ b/drivers/cpufreq/cpufreq_governor.h @@ -148,7 +148,6 @@ struct cpu_dbs_common_info {

struct od_cpu_dbs_info_s { struct cpu_dbs_common_info cdbs; - u64 prev_cpu_iowait; struct cpufreq_frequency_table *freq_table; unsigned int freq_lo; unsigned int freq_lo_jiffies; @@ -257,7 +256,7 @@ static ssize_t show_sampling_rate_min_gov_pol \ return sprintf(buf, "%u\n", dbs_data->min_sampling_rate); \ }

-u64 get_cpu_idle_time(unsigned int cpu, u64 *wall); +u64 get_cpu_idle_time(unsigned int cpu, u64 *wall, int io_busy); void dbs_check_cpu(struct dbs_data *dbs_data, int cpu); bool need_load_eval(struct cpu_dbs_common_info *cdbs, unsigned int sampling_rate); diff --git a/drivers/cpufreq/cpufreq_ondemand.c b/drivers/cpufreq/cpufreq_ondemand.c index 15e80ee..8500147 100644 --- a/drivers/cpufreq/cpufreq_ondemand.c +++ b/drivers/cpufreq/cpufreq_ondemand.c @@ -339,11 +339,20 @@ static ssize_t store_io_is_busy(struct dbs_data *dbs_data, const char *buf, struct od_dbs_tuners *od_tuners = dbs_data->tuners; unsigned int input; int ret; + unsigned int j;

ret = sscanf(buf, "%u", &input); if (ret != 1) return -EINVAL; od_tuners->io_is_busy = !!input; + + /* we need to re-evaluate prev_cpu_idle */ + for_each_online_cpu(j) { + struct od_cpu_dbs_info_s *dbs_info = &per_cpu(od_cpu_dbs_info, + j); + dbs_info->cdbs.prev_cpu_idle = get_cpu_idle_time(j, + &dbs_info->cdbs.prev_cpu_wall, od_tuners->io_is_busy); + } return count; }

@@ -414,7 +423,7 @@ static ssize_t store_ignore_nice(struct dbs_data *dbs_data, const char *buf, struct od_cpu_dbs_info_s *dbs_info; dbs_info = &per_cpu(od_cpu_dbs_info, j); dbs_info->cdbs.prev_cpu_idle = get_cpu_idle_time(j, - &dbs_info->cdbs.prev_cpu_wall); + &dbs_info->cdbs.prev_cpu_wall, od_tuners->io_is_busy); if (od_tuners->ignore_nice) dbs_info->cdbs.prev_cpu_nice = kcpustat_cpu(j).cpustat[CPUTIME_NICE];

get_governor_parent_kobj() can be used by any governor, generic cpufreq governors or platform specific ones and so must be present in cpufreq.c instead of cpufreq_governor.c.

This patch moves it to cpufreq.c. This also adds EXPORT_SYMBOL_GPL(get_governor_parent_kobj) so that modules can use this function too.

Signed-off-by: Viresh Kumar viresh.kumar@linaro.org --- drivers/cpufreq/cpufreq.c | 9 +++++++++ drivers/cpufreq/cpufreq_governor.c | 8 -------- include/linux/cpufreq.h | 1 + 3 files changed, 10 insertions(+), 8 deletions(-)

diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index d476bb8..763fc8e1 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -134,6 +134,15 @@ bool have_governor_per_policy(void) } EXPORT_SYMBOL_GPL(have_governor_per_policy);

+struct kobject *get_governor_parent_kobj(struct cpufreq_policy *policy) +{ + if (have_governor_per_policy()) + return &policy->kobj; + else + return cpufreq_global_kobject; +} +EXPORT_SYMBOL_GPL(get_governor_parent_kobj); + static struct cpufreq_policy *__cpufreq_cpu_get(unsigned int cpu, bool sysfs) { struct cpufreq_policy *data; diff --git a/drivers/cpufreq/cpufreq_governor.c b/drivers/cpufreq/cpufreq_governor.c index 13ca17a..39ea9e5 100644 --- a/drivers/cpufreq/cpufreq_governor.c +++ b/drivers/cpufreq/cpufreq_governor.c @@ -29,14 +29,6 @@

#include "cpufreq_governor.h"

-static struct kobject *get_governor_parent_kobj(struct cpufreq_policy *policy) -{ - if (have_governor_per_policy()) - return &policy->kobj; - else - return cpufreq_global_kobject; -} - static struct attribute_group *get_sysfs_attr(struct dbs_data *dbs_data) { if (have_governor_per_policy()) diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h index 9fd6bce..3f53526 100644 --- a/include/linux/cpufreq.h +++ b/include/linux/cpufreq.h @@ -340,6 +340,7 @@ const char *cpufreq_get_current_driver(void); int cpufreq_get_policy(struct cpufreq_policy *policy, unsigned int cpu); int cpufreq_update_policy(unsigned int cpu); bool have_governor_per_policy(void); +struct kobject *get_governor_parent_kobj(struct cpufreq_policy *policy);

#ifdef CONFIG_CPU_FREQ /* query the current CPU frequency (in kHz). If zero, cpufreq couldn't detect it */

Governors other than ondemand and conservative can also use get_cpu_idle_time() and they aren't required to compile cpufreq_governor.c. So, move these independent routines to cpufreq.c instead.

Signed-off-by: Viresh Kumar viresh.kumar@linaro.org --- drivers/cpufreq/cpufreq.c | 38 ++++++++++++++++++++++++++++++++++++++ drivers/cpufreq/cpufreq_governor.c | 36 ------------------------------------ drivers/cpufreq/cpufreq_governor.h | 1 - include/linux/cpufreq.h | 1 + 4 files changed, 39 insertions(+), 37 deletions(-)

diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index cdde8cb..ea9f755 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -17,7 +17,9 @@

#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt

+#include <asm/cputime.h> #include <linux/kernel.h> +#include <linux/kernel_stat.h> #include <linux/module.h> #include <linux/init.h> #include <linux/notifier.h> @@ -25,6 +27,7 @@ #include <linux/delay.h> #include <linux/interrupt.h> #include <linux/spinlock.h> +#include <linux/tick.h> #include <linux/device.h> #include <linux/slab.h> #include <linux/cpu.h> @@ -143,6 +146,41 @@ struct kobject *get_governor_parent_kobj(struct cpufreq_policy *policy) } EXPORT_SYMBOL_GPL(get_governor_parent_kobj);

+static inline u64 get_cpu_idle_time_jiffy(unsigned int cpu, u64 *wall) +{ + u64 idle_time; + u64 cur_wall_time; + u64 busy_time; + + cur_wall_time = jiffies64_to_cputime64(get_jiffies_64()); + + busy_time = kcpustat_cpu(cpu).cpustat[CPUTIME_USER]; + busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_SYSTEM]; + busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_IRQ]; + busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_SOFTIRQ]; + busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_STEAL]; + busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_NICE]; + + idle_time = cur_wall_time - busy_time; + if (wall) + *wall = cputime_to_usecs(cur_wall_time); + + return cputime_to_usecs(idle_time); +} + +u64 get_cpu_idle_time(unsigned int cpu, u64 *wall, int io_busy) +{ + u64 idle_time = get_cpu_idle_time_us(cpu, io_busy ? wall : NULL); + + if (idle_time == -1ULL) + return get_cpu_idle_time_jiffy(cpu, wall); + else if (!io_busy) + idle_time += get_cpu_iowait_time_us(cpu, wall); + + return idle_time; +} +EXPORT_SYMBOL_GPL(get_cpu_idle_time); + static struct cpufreq_policy *__cpufreq_cpu_get(unsigned int cpu, bool sysfs) { struct cpufreq_policy *data; diff --git a/drivers/cpufreq/cpufreq_governor.c b/drivers/cpufreq/cpufreq_governor.c index 39ea9e5..6bb0b11 100644 --- a/drivers/cpufreq/cpufreq_governor.c +++ b/drivers/cpufreq/cpufreq_governor.c @@ -23,7 +23,6 @@ #include <linux/kernel_stat.h> #include <linux/mutex.h> #include <linux/slab.h> -#include <linux/tick.h> #include <linux/types.h> #include <linux/workqueue.h>

@@ -37,41 +36,6 @@ static struct attribute_group *get_sysfs_attr(struct dbs_data *dbs_data) return dbs_data->cdata->attr_group_gov_sys; }

-static inline u64 get_cpu_idle_time_jiffy(unsigned int cpu, u64 *wall) -{ - u64 idle_time; - u64 cur_wall_time; - u64 busy_time; - - cur_wall_time = jiffies64_to_cputime64(get_jiffies_64()); - - busy_time = kcpustat_cpu(cpu).cpustat[CPUTIME_USER]; - busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_SYSTEM]; - busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_IRQ]; - busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_SOFTIRQ]; - busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_STEAL]; - busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_NICE]; - - idle_time = cur_wall_time - busy_time; - if (wall) - *wall = cputime_to_usecs(cur_wall_time); - - return cputime_to_usecs(idle_time); -} - -u64 get_cpu_idle_time(unsigned int cpu, u64 *wall, int io_busy) -{ - u64 idle_time = get_cpu_idle_time_us(cpu, io_busy ? wall : NULL); - - if (idle_time == -1ULL) - return get_cpu_idle_time_jiffy(cpu, wall); - else if (!io_busy) - idle_time += get_cpu_iowait_time_us(cpu, wall); - - return idle_time; -} -EXPORT_SYMBOL_GPL(get_cpu_idle_time); - void dbs_check_cpu(struct dbs_data *dbs_data, int cpu) { struct cpu_dbs_common_info *cdbs = dbs_data->cdata->get_cpu_cdbs(cpu); diff --git a/drivers/cpufreq/cpufreq_governor.h b/drivers/cpufreq/cpufreq_governor.h index eae3163..eca7762 100644 --- a/drivers/cpufreq/cpufreq_governor.h +++ b/drivers/cpufreq/cpufreq_governor.h @@ -256,7 +256,6 @@ static ssize_t show_sampling_rate_min_gov_pol \ return sprintf(buf, "%u\n", dbs_data->min_sampling_rate); \ }

-u64 get_cpu_idle_time(unsigned int cpu, u64 *wall, int io_busy); void dbs_check_cpu(struct dbs_data *dbs_data, int cpu); bool need_load_eval(struct cpu_dbs_common_info *cdbs, unsigned int sampling_rate); diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h index 3f53526..2839e49 100644 --- a/include/linux/cpufreq.h +++ b/include/linux/cpufreq.h @@ -337,6 +337,7 @@ const char *cpufreq_get_current_driver(void); /********************************************************************* * CPUFREQ 2.6. INTERFACE * *********************************************************************/ +u64 get_cpu_idle_time(unsigned int cpu, u64 *wall, int io_busy); int cpufreq_get_policy(struct cpufreq_policy *policy, unsigned int cpu); int cpufreq_update_policy(unsigned int cpu); bool have_governor_per_policy(void);

Cpufreq no longer calls governor callback for offlined cpus. i.e. All policy->cpus are guaranteed to be online. Hence we don't need explicit check to see if cpu is online or not.

Signed-off-by: Viresh Kumar viresh.kumar@linaro.org --- drivers/cpufreq/cpufreq_interactive.c | 3 --- 1 file changed, 3 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c index 4ed2fed..baa5638 100644 --- a/drivers/cpufreq/cpufreq_interactive.c +++ b/drivers/cpufreq/cpufreq_interactive.c @@ -992,9 +992,6 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy,

switch (event) { case CPUFREQ_GOV_START: - if (!cpu_online(policy->cpu)) - return -EINVAL; - mutex_lock(&gov_lock);

freq_table =

If we have a multi-package system, where we have multiple instances of struct policy (per package), currently we can't have multiple instances of same governor. i.e. We can't have multiple instances of Interactive governor for multiple packages.

This is a bottleneck for multicluster system, where we want different packages to use Interactive governor, but with different tunables.

This patch uses the infrastructure provided by earlier patches pushed in Mainline in v3.10-rc1/rc2 and implements per policy instances of Interactive governor.

Signed-off-by: Viresh Kumar viresh.kumar@linaro.org --- drivers/cpufreq/cpufreq_interactive.c | 615 ++++++++++++++++++++-------------- 1 file changed, 371 insertions(+), 244 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c index 8821873..05733ea 100644 --- a/drivers/cpufreq/cpufreq_interactive.c +++ b/drivers/cpufreq/cpufreq_interactive.c @@ -30,12 +30,11 @@ #include <linux/workqueue.h> #include <linux/kthread.h> #include <linux/slab.h> +#include "cpufreq_governor.h"

#define CREATE_TRACE_POINTS #include <trace/events/cpufreq_interactive.h>

-static int active_count; - struct cpufreq_interactive_cpuinfo { struct timer_list cpu_timer; struct timer_list cpu_slack_timer; @@ -62,110 +61,131 @@ static cpumask_t speedchange_cpumask; static spinlock_t speedchange_cpumask_lock; static struct mutex gov_lock;

-/* Hi speed to bump to from lo speed when load burst (default max) */ -static unsigned int hispeed_freq; - -/* Go to hi speed when CPU load at or above this value. */ -#define DEFAULT_GO_HISPEED_LOAD 99 -static unsigned long go_hispeed_load = DEFAULT_GO_HISPEED_LOAD; - -/* Target load. Lower values result in higher CPU speeds. */ #define DEFAULT_TARGET_LOAD 90 static unsigned int default_target_loads[] = {DEFAULT_TARGET_LOAD}; -static spinlock_t target_loads_lock; -static unsigned int *target_loads = default_target_loads; -static int ntarget_loads = ARRAY_SIZE(default_target_loads);

-/* - * The minimum amount of time to spend at a frequency before we can ramp down. - */ -#define DEFAULT_MIN_SAMPLE_TIME (80 * USEC_PER_MSEC) -static unsigned long min_sample_time = DEFAULT_MIN_SAMPLE_TIME; - -/* - * The sample rate of the timer used to increase frequency - */ #define DEFAULT_TIMER_RATE (20 * USEC_PER_MSEC) -static unsigned long timer_rate = DEFAULT_TIMER_RATE; - -/* - * Wait this long before raising speed above hispeed, by default a single - * timer interval. - */ #define DEFAULT_ABOVE_HISPEED_DELAY DEFAULT_TIMER_RATE static unsigned int default_above_hispeed_delay[] = { DEFAULT_ABOVE_HISPEED_DELAY }; -static spinlock_t above_hispeed_delay_lock; -static unsigned int *above_hispeed_delay = default_above_hispeed_delay; -static int nabove_hispeed_delay = ARRAY_SIZE(default_above_hispeed_delay);

-/* Non-zero means indefinite speed boost active */ -static int boost_val; -/* Duration of a boot pulse in usecs */ -static int boostpulse_duration_val = DEFAULT_MIN_SAMPLE_TIME; -/* End time of boost pulse in ktime converted to usecs */ -static u64 boostpulse_endtime; +struct cpufreq_interactive_tunables { + int usage_count;

-/* - * Max additional time to wait in idle, beyond timer_rate, at speeds above - * minimum before wakeup to reduce speed, or -1 if unnecessary. - */ + /* Hi speed to bump to from lo speed when load burst (default max) */ + unsigned int hispeed_freq; + + /* Go to hi speed when CPU load at or above this value. */ +#define DEFAULT_GO_HISPEED_LOAD 99 + unsigned long go_hispeed_load; + + /* Target load. Lower values result in higher CPU speeds. */ + spinlock_t target_loads_lock; + unsigned int *target_loads; + int ntarget_loads; + + /* + * The minimum amount of time to spend at a frequency before we can ramp + * down. + */ +#define DEFAULT_MIN_SAMPLE_TIME (80 * USEC_PER_MSEC) + unsigned long min_sample_time; + + /* + * The sample rate of the timer used to increase frequency + */ + unsigned long timer_rate; + + /* + * Wait this long before raising speed above hispeed, by default a + * single timer interval. + */ + spinlock_t above_hispeed_delay_lock; + unsigned int *above_hispeed_delay; + int nabove_hispeed_delay; + + /* Non-zero means indefinite speed boost active */ + int boost_val; + /* Duration of a boot pulse in usecs */ + int boostpulse_duration_val; + /* End time of boost pulse in ktime converted to usecs */ + u64 boostpulse_endtime; + + /* + * Max additional time to wait in idle, beyond timer_rate, at speeds + * above minimum before wakeup to reduce speed, or -1 if unnecessary. + */ #define DEFAULT_TIMER_SLACK (4 * DEFAULT_TIMER_RATE) -static int timer_slack_val = DEFAULT_TIMER_SLACK; + int timer_slack_val; + + bool io_is_busy; +};

-static bool io_is_busy; +/* For cases where we have single governor instance for system */ +struct cpufreq_interactive_tunables *common_tunables; + +static struct attribute_group *get_sysfs_attr(void);

static void cpufreq_interactive_timer_resched( struct cpufreq_interactive_cpuinfo *pcpu) { - unsigned long expires = jiffies + usecs_to_jiffies(timer_rate); + struct cpufreq_interactive_tunables *tunables = + pcpu->policy->governor_data; + unsigned long expires = jiffies + + usecs_to_jiffies(tunables->timer_rate); unsigned long flags;

mod_timer_pinned(&pcpu->cpu_timer, expires); - if (timer_slack_val >= 0 && pcpu->target_freq > pcpu->policy->min) { - expires += usecs_to_jiffies(timer_slack_val); + if (tunables->timer_slack_val >= 0 && + pcpu->target_freq > pcpu->policy->min) { + expires += usecs_to_jiffies(tunables->timer_slack_val); mod_timer_pinned(&pcpu->cpu_slack_timer, expires); }

spin_lock_irqsave(&pcpu->load_lock, flags); pcpu->time_in_idle = get_cpu_idle_time(smp_processor_id(), - &pcpu->time_in_idle_timestamp, io_is_busy); + &pcpu->time_in_idle_timestamp, + tunables->io_is_busy); pcpu->cputime_speedadj = 0; pcpu->cputime_speedadj_timestamp = pcpu->time_in_idle_timestamp; spin_unlock_irqrestore(&pcpu->load_lock, flags); }

-static unsigned int freq_to_above_hispeed_delay(unsigned int freq) +static unsigned int freq_to_above_hispeed_delay( + struct cpufreq_interactive_tunables *tunables, + unsigned int freq) { int i; unsigned int ret; unsigned long flags;

- spin_lock_irqsave(&above_hispeed_delay_lock, flags); + spin_lock_irqsave(&tunables->above_hispeed_delay_lock, flags);

- for (i = 0; i < nabove_hispeed_delay - 1 && - freq >= above_hispeed_delay[i+1]; i += 2) + for (i = 0; i < tunables->nabove_hispeed_delay - 1 && + freq >= tunables->above_hispeed_delay[i+1]; i += 2) ;

- ret = above_hispeed_delay[i]; - spin_unlock_irqrestore(&above_hispeed_delay_lock, flags); + ret = tunables->above_hispeed_delay[i]; + spin_unlock_irqrestore(&tunables->above_hispeed_delay_lock, flags); return ret; }

-static unsigned int freq_to_targetload(unsigned int freq) +static unsigned int freq_to_targetload(struct cpufreq_interactive_tunables + *tunables, unsigned int freq) { int i; unsigned int ret; unsigned long flags;

- spin_lock_irqsave(&target_loads_lock, flags); + spin_lock_irqsave(&tunables->target_loads_lock, flags);

- for (i = 0; i < ntarget_loads - 1 && freq >= target_loads[i+1]; i += 2) + for (i = 0; i < tunables->ntarget_loads - 1 && + freq >= tunables->target_loads[i+1]; i += 2) ;

- ret = target_loads[i]; - spin_unlock_irqrestore(&target_loads_lock, flags); + ret = tunables->target_loads[i]; + spin_unlock_irqrestore(&tunables->target_loads_lock, flags); return ret; }

@@ -174,9 +194,8 @@ static unsigned int freq_to_targetload(unsigned int freq) * choose_freq() will find the minimum frequency that does not exceed its * target load given the current load. */ - -static unsigned int choose_freq( - struct cpufreq_interactive_cpuinfo *pcpu, unsigned int loadadjfreq) +static unsigned int choose_freq(struct cpufreq_interactive_cpuinfo *pcpu, + unsigned int loadadjfreq) { unsigned int freq = pcpu->policy->cur; unsigned int prevfreq, freqmin, freqmax; @@ -188,7 +207,7 @@ static unsigned int choose_freq(

do { prevfreq = freq; - tl = freq_to_targetload(freq); + tl = freq_to_targetload(pcpu->policy->governor_data, freq);

/* * Find the lowest frequency where the computed load is less @@ -260,13 +279,15 @@ static unsigned int choose_freq( static u64 update_load(int cpu) { struct cpufreq_interactive_cpuinfo *pcpu = &per_cpu(cpuinfo, cpu); + struct cpufreq_interactive_tunables *tunables = + pcpu->policy->governor_data; u64 now; u64 now_idle; unsigned int delta_idle; unsigned int delta_time; u64 active_time;

- now_idle = get_cpu_idle_time(cpu, &now, io_is_busy); + now_idle = get_cpu_idle_time(cpu, &now, tunables->io_is_busy); delta_idle = (unsigned int)(now_idle - pcpu->time_in_idle); delta_time = (unsigned int)(now - pcpu->time_in_idle_timestamp); active_time = delta_time - delta_idle; @@ -285,6 +306,8 @@ static void cpufreq_interactive_timer(unsigned long data) int cpu_load; struct cpufreq_interactive_cpuinfo *pcpu = &per_cpu(cpuinfo, data); + struct cpufreq_interactive_tunables *tunables = + pcpu->policy->governor_data; unsigned int new_freq; unsigned int loadadjfreq; unsigned int index; @@ -308,25 +331,25 @@ static void cpufreq_interactive_timer(unsigned long data) do_div(cputime_speedadj, delta_time); loadadjfreq = (unsigned int)cputime_speedadj * 100; cpu_load = loadadjfreq / pcpu->target_freq; - boosted = boost_val || now < boostpulse_endtime; + boosted = tunables->boost_val || now < tunables->boostpulse_endtime;

- if (cpu_load >= go_hispeed_load || boosted) { - if (pcpu->target_freq < hispeed_freq) { - new_freq = hispeed_freq; + if (cpu_load >= tunables->go_hispeed_load || boosted) { + if (pcpu->target_freq < tunables->hispeed_freq) { + new_freq = tunables->hispeed_freq; } else { new_freq = choose_freq(pcpu, loadadjfreq);

- if (new_freq < hispeed_freq) - new_freq = hispeed_freq; + if (new_freq < tunables->hispeed_freq) + new_freq = tunables->hispeed_freq; } } else { new_freq = choose_freq(pcpu, loadadjfreq); }

- if (pcpu->target_freq >= hispeed_freq && + if (pcpu->target_freq >= tunables->hispeed_freq && new_freq > pcpu->target_freq && now - pcpu->hispeed_validate_time < - freq_to_above_hispeed_delay(pcpu->target_freq)) { + freq_to_above_hispeed_delay(tunables, pcpu->target_freq)) { trace_cpufreq_interactive_notyet( data, cpu_load, pcpu->target_freq, pcpu->policy->cur, new_freq); @@ -350,7 +373,8 @@ static void cpufreq_interactive_timer(unsigned long data) * floor frequency for the minimum sample time since last validated. */ if (new_freq < pcpu->floor_freq) { - if (now - pcpu->floor_validate_time < min_sample_time) { + if (now - pcpu->floor_validate_time < + tunables->min_sample_time) { trace_cpufreq_interactive_notyet( data, cpu_load, pcpu->target_freq, pcpu->policy->cur, new_freq); @@ -366,7 +390,7 @@ static void cpufreq_interactive_timer(unsigned long data) * (or the indefinite boost is turned off). */

- if (!boosted || new_freq > hispeed_freq) { + if (!boosted || new_freq > tunables->hispeed_freq) { pcpu->floor_freq = new_freq; pcpu->floor_validate_time = now; } @@ -527,14 +551,16 @@ static void cpufreq_interactive_boost(void) int anyboost = 0; unsigned long flags; struct cpufreq_interactive_cpuinfo *pcpu; + struct cpufreq_interactive_tunables *tunables;

spin_lock_irqsave(&speedchange_cpumask_lock, flags);

for_each_online_cpu(i) { pcpu = &per_cpu(cpuinfo, i); + tunables = pcpu->policy->governor_data;

- if (pcpu->target_freq < hispeed_freq) { - pcpu->target_freq = hispeed_freq; + if (pcpu->target_freq < tunables->hispeed_freq) { + pcpu->target_freq = tunables->hispeed_freq; cpumask_set_cpu(i, &speedchange_cpumask); pcpu->hispeed_validate_time = ktime_to_us(ktime_get()); @@ -546,7 +572,7 @@ static void cpufreq_interactive_boost(void) * validated. */

- pcpu->floor_freq = hispeed_freq; + pcpu->floor_freq = tunables->hispeed_freq; pcpu->floor_validate_time = ktime_to_us(ktime_get()); }

@@ -635,27 +661,26 @@ err: return ERR_PTR(err); }

-static ssize_t show_target_loads( - struct kobject *kobj, struct attribute *attr, char *buf) +static ssize_t show_target_loads(struct cpufreq_interactive_tunables *tunables, + char *buf) { int i; ssize_t ret = 0; unsigned long flags;

- spin_lock_irqsave(&target_loads_lock, flags); + spin_lock_irqsave(&tunables->target_loads_lock, flags);

- for (i = 0; i < ntarget_loads; i++) - ret += sprintf(buf + ret, "%u%s", target_loads[i], + for (i = 0; i < tunables->ntarget_loads; i++) + ret += sprintf(buf + ret, "%u%s", tunables->target_loads[i], i & 0x1 ? ":" : " ");

ret += sprintf(buf + ret, "\n"); - spin_unlock_irqrestore(&target_loads_lock, flags); + spin_unlock_irqrestore(&tunables->target_loads_lock, flags); return ret; }

-static ssize_t store_target_loads( - struct kobject *kobj, struct attribute *attr, const char *buf, - size_t count) +static ssize_t store_target_loads(struct cpufreq_interactive_tunables *tunables, + const char *buf, size_t count) { int ntokens; unsigned int *new_target_loads = NULL; @@ -665,40 +690,36 @@ static ssize_t store_target_loads( if (IS_ERR(new_target_loads)) return PTR_RET(new_target_loads);

- spin_lock_irqsave(&target_loads_lock, flags); - if (target_loads != default_target_loads) - kfree(target_loads); - target_loads = new_target_loads; - ntarget_loads = ntokens; - spin_unlock_irqrestore(&target_loads_lock, flags); + spin_lock_irqsave(&tunables->target_loads_lock, flags); + if (tunables->target_loads != default_target_loads) + kfree(tunables->target_loads); + tunables->target_loads = new_target_loads; + tunables->ntarget_loads = ntokens; + spin_unlock_irqrestore(&tunables->target_loads_lock, flags); return count; }

-static struct global_attr target_loads_attr = - __ATTR(target_loads, S_IRUGO | S_IWUSR, - show_target_loads, store_target_loads); - -static ssize_t show_above_hispeed_delay( - struct kobject *kobj, struct attribute *attr, char *buf) +static ssize_t show_above_hispeed_delay(struct cpufreq_interactive_tunables + *tunables, char *buf) { int i; ssize_t ret = 0; unsigned long flags;

- spin_lock_irqsave(&above_hispeed_delay_lock, flags); + spin_lock_irqsave(&tunables->above_hispeed_delay_lock, flags);

- for (i = 0; i < nabove_hispeed_delay; i++) - ret += sprintf(buf + ret, "%u%s", above_hispeed_delay[i], + for (i = 0; i < tunables->nabove_hispeed_delay; i++) + ret += sprintf(buf + ret, "%u%s", + tunables->above_hispeed_delay[i], i & 0x1 ? ":" : " ");

ret += sprintf(buf + ret, "\n"); - spin_unlock_irqrestore(&above_hispeed_delay_lock, flags); + spin_unlock_irqrestore(&tunables->above_hispeed_delay_lock, flags); return ret; }

-static ssize_t store_above_hispeed_delay( - struct kobject *kobj, struct attribute *attr, const char *buf, - size_t count) +static ssize_t store_above_hispeed_delay(struct cpufreq_interactive_tunables + *tunables, const char *buf, size_t count) { int ntokens; unsigned int *new_above_hispeed_delay = NULL; @@ -708,29 +729,24 @@ static ssize_t store_above_hispeed_delay( if (IS_ERR(new_above_hispeed_delay)) return PTR_RET(new_above_hispeed_delay);

- spin_lock_irqsave(&above_hispeed_delay_lock, flags); - if (above_hispeed_delay != default_above_hispeed_delay) - kfree(above_hispeed_delay); - above_hispeed_delay = new_above_hispeed_delay; - nabove_hispeed_delay = ntokens; - spin_unlock_irqrestore(&above_hispeed_delay_lock, flags); + spin_lock_irqsave(&tunables->above_hispeed_delay_lock, flags); + if (tunables->above_hispeed_delay != default_above_hispeed_delay) + kfree(tunables->above_hispeed_delay); + tunables->above_hispeed_delay = new_above_hispeed_delay; + tunables->nabove_hispeed_delay = ntokens; + spin_unlock_irqrestore(&tunables->above_hispeed_delay_lock, flags); return count;

}

-static struct global_attr above_hispeed_delay_attr = - __ATTR(above_hispeed_delay, S_IRUGO | S_IWUSR, - show_above_hispeed_delay, store_above_hispeed_delay); - -static ssize_t show_hispeed_freq(struct kobject *kobj, - struct attribute *attr, char *buf) +static ssize_t show_hispeed_freq(struct cpufreq_interactive_tunables *tunables, + char *buf) { - return sprintf(buf, "%u\n", hispeed_freq); + return sprintf(buf, "%u\n", tunables->hispeed_freq); }

-static ssize_t store_hispeed_freq(struct kobject *kobj, - struct attribute *attr, const char *buf, - size_t count) +static ssize_t store_hispeed_freq(struct cpufreq_interactive_tunables *tunables, + const char *buf, size_t count) { int ret; long unsigned int val; @@ -738,22 +754,18 @@ static ssize_t store_hispeed_freq(struct kobject *kobj, ret = strict_strtoul(buf, 0, &val); if (ret < 0) return ret; - hispeed_freq = val; + tunables->hispeed_freq = val; return count; }

-static struct global_attr hispeed_freq_attr = __ATTR(hispeed_freq, 0644, - show_hispeed_freq, store_hispeed_freq); - - -static ssize_t show_go_hispeed_load(struct kobject *kobj, - struct attribute *attr, char *buf) +static ssize_t show_go_hispeed_load(struct cpufreq_interactive_tunables + *tunables, char *buf) { - return sprintf(buf, "%lu\n", go_hispeed_load); + return sprintf(buf, "%lu\n", tunables->go_hispeed_load); }

-static ssize_t store_go_hispeed_load(struct kobject *kobj, - struct attribute *attr, const char *buf, size_t count) +static ssize_t store_go_hispeed_load(struct cpufreq_interactive_tunables + *tunables, const char *buf, size_t count) { int ret; unsigned long val; @@ -761,21 +773,18 @@ static ssize_t store_go_hispeed_load(struct kobject *kobj, ret = strict_strtoul(buf, 0, &val); if (ret < 0) return ret; - go_hispeed_load = val; + tunables->go_hispeed_load = val; return count; }

-static struct global_attr go_hispeed_load_attr = __ATTR(go_hispeed_load, 0644, - show_go_hispeed_load, store_go_hispeed_load); - -static ssize_t show_min_sample_time(struct kobject *kobj, - struct attribute *attr, char *buf) +static ssize_t show_min_sample_time(struct cpufreq_interactive_tunables + *tunables, char *buf) { - return sprintf(buf, "%lu\n", min_sample_time); + return sprintf(buf, "%lu\n", tunables->min_sample_time); }

-static ssize_t store_min_sample_time(struct kobject *kobj, - struct attribute *attr, const char *buf, size_t count) +static ssize_t store_min_sample_time(struct cpufreq_interactive_tunables + *tunables, const char *buf, size_t count) { int ret; unsigned long val; @@ -783,21 +792,18 @@ static ssize_t store_min_sample_time(struct kobject *kobj, ret = strict_strtoul(buf, 0, &val); if (ret < 0) return ret; - min_sample_time = val; + tunables->min_sample_time = val; return count; }

-static struct global_attr min_sample_time_attr = __ATTR(min_sample_time, 0644, - show_min_sample_time, store_min_sample_time); - -static ssize_t show_timer_rate(struct kobject *kobj, - struct attribute *attr, char *buf) +static ssize_t show_timer_rate(struct cpufreq_interactive_tunables *tunables, + char *buf) { - return sprintf(buf, "%lu\n", timer_rate); + return sprintf(buf, "%lu\n", tunables->timer_rate); }

-static ssize_t store_timer_rate(struct kobject *kobj, - struct attribute *attr, const char *buf, size_t count) +static ssize_t store_timer_rate(struct cpufreq_interactive_tunables *tunables, + const char *buf, size_t count) { int ret; unsigned long val; @@ -805,22 +811,18 @@ static ssize_t store_timer_rate(struct kobject *kobj, ret = strict_strtoul(buf, 0, &val); if (ret < 0) return ret; - timer_rate = val; + tunables->timer_rate = val; return count; }

-static struct global_attr timer_rate_attr = __ATTR(timer_rate, 0644, - show_timer_rate, store_timer_rate); - -static ssize_t show_timer_slack( - struct kobject *kobj, struct attribute *attr, char *buf) +static ssize_t show_timer_slack(struct cpufreq_interactive_tunables *tunables, + char *buf) { - return sprintf(buf, "%d\n", timer_slack_val); + return sprintf(buf, "%d\n", tunables->timer_slack_val); }

-static ssize_t store_timer_slack( - struct kobject *kobj, struct attribute *attr, const char *buf, - size_t count) +static ssize_t store_timer_slack(struct cpufreq_interactive_tunables *tunables, + const char *buf, size_t count) { int ret; unsigned long val; @@ -829,19 +831,17 @@ static ssize_t store_timer_slack( if (ret < 0) return ret;

- timer_slack_val = val; + tunables->timer_slack_val = val; return count; }

-define_one_global_rw(timer_slack); - -static ssize_t show_boost(struct kobject *kobj, struct attribute *attr, +static ssize_t show_boost(struct cpufreq_interactive_tunables *tunables, char *buf) { - return sprintf(buf, "%d\n", boost_val); + return sprintf(buf, "%d\n", tunables->boost_val); }

-static ssize_t store_boost(struct kobject *kobj, struct attribute *attr, +static ssize_t store_boost(struct cpufreq_interactive_tunables *tunables, const char *buf, size_t count) { int ret; @@ -851,9 +851,9 @@ static ssize_t store_boost(struct kobject *kobj, struct attribute *attr, if (ret < 0) return ret;

- boost_val = val; + tunables->boost_val = val;

- if (boost_val) { + if (tunables->boost_val) { trace_cpufreq_interactive_boost("on"); cpufreq_interactive_boost(); } else { @@ -863,9 +863,7 @@ static ssize_t store_boost(struct kobject *kobj, struct attribute *attr, return count; }

-define_one_global_rw(boost); - -static ssize_t store_boostpulse(struct kobject *kobj, struct attribute *attr, +static ssize_t store_boostpulse(struct cpufreq_interactive_tunables *tunables, const char *buf, size_t count) { int ret; @@ -875,24 +873,21 @@ static ssize_t store_boostpulse(struct kobject *kobj, struct attribute *attr, if (ret < 0) return ret;

- boostpulse_endtime = ktime_to_us(ktime_get()) + boostpulse_duration_val; + tunables->boostpulse_endtime = ktime_to_us(ktime_get()) + + tunables->boostpulse_duration_val; trace_cpufreq_interactive_boost("pulse"); cpufreq_interactive_boost(); return count; }

-static struct global_attr boostpulse = - __ATTR(boostpulse, 0200, NULL, store_boostpulse); - -static ssize_t show_boostpulse_duration( - struct kobject *kobj, struct attribute *attr, char *buf) +static ssize_t show_boostpulse_duration(struct cpufreq_interactive_tunables + *tunables, char *buf) { - return sprintf(buf, "%d\n", boostpulse_duration_val); + return sprintf(buf, "%d\n", tunables->boostpulse_duration_val); }

-static ssize_t store_boostpulse_duration( - struct kobject *kobj, struct attribute *attr, const char *buf, - size_t count) +static ssize_t store_boostpulse_duration(struct cpufreq_interactive_tunables + *tunables, const char *buf, size_t count) { int ret; unsigned long val; @@ -901,20 +896,18 @@ static ssize_t store_boostpulse_duration( if (ret < 0) return ret;

- boostpulse_duration_val = val; + tunables->boostpulse_duration_val = val; return count; }

-define_one_global_rw(boostpulse_duration); - -static ssize_t show_io_is_busy(struct kobject *kobj, - struct attribute *attr, char *buf) +static ssize_t show_io_is_busy(struct cpufreq_interactive_tunables *tunables, + char *buf) { - return sprintf(buf, "%u\n", io_is_busy); + return sprintf(buf, "%u\n", tunables->io_is_busy); }

-static ssize_t store_io_is_busy(struct kobject *kobj, - struct attribute *attr, const char *buf, size_t count) +static ssize_t store_io_is_busy(struct cpufreq_interactive_tunables *tunables, + const char *buf, size_t count) { int ret; unsigned long val; @@ -922,33 +915,125 @@ static ssize_t store_io_is_busy(struct kobject *kobj, ret = kstrtoul(buf, 0, &val); if (ret < 0) return ret; - io_is_busy = val; + tunables->io_is_busy = val; return count; }

-static struct global_attr io_is_busy_attr = __ATTR(io_is_busy, 0644, - show_io_is_busy, store_io_is_busy); - -static struct attribute *interactive_attributes[] = { - &target_loads_attr.attr, - &above_hispeed_delay_attr.attr, - &hispeed_freq_attr.attr, - &go_hispeed_load_attr.attr, - &min_sample_time_attr.attr, - &timer_rate_attr.attr, - &timer_slack.attr, - &boost.attr, - &boostpulse.attr, - &boostpulse_duration.attr, - &io_is_busy_attr.attr, +/* + * Create show/store routines + * - sys: One governor instance for complete SYSTEM + * - pol: One governor instance per struct cpufreq_policy + */ +#define show_gov_pol_sys(file_name) \ +static ssize_t show_##file_name##_gov_sys \ +(struct kobject *kobj, struct attribute *attr, char *buf) \ +{ \ + return show_##file_name(common_tunables, buf); \ +} \ + \ +static ssize_t show_##file_name##_gov_pol \ +(struct cpufreq_policy *policy, char *buf) \ +{ \ + return show_##file_name(policy->governor_data, buf); \ +} + +#define store_gov_pol_sys(file_name) \ +static ssize_t store_##file_name##_gov_sys \ +(struct kobject *kobj, struct attribute *attr, const char *buf, \ + size_t count) \ +{ \ + return store_##file_name(common_tunables, buf, count); \ +} \ + \ +static ssize_t store_##file_name##_gov_pol \ +(struct cpufreq_policy *policy, const char *buf, size_t count) \ +{ \ + return store_##file_name(policy->governor_data, buf, count); \ +} + +#define show_store_gov_pol_sys(file_name) \ +show_gov_pol_sys(file_name); \ +store_gov_pol_sys(file_name) + +show_store_gov_pol_sys(target_loads); +show_store_gov_pol_sys(above_hispeed_delay); +show_store_gov_pol_sys(hispeed_freq); +show_store_gov_pol_sys(go_hispeed_load); +show_store_gov_pol_sys(min_sample_time); +show_store_gov_pol_sys(timer_rate); +show_store_gov_pol_sys(timer_slack); +show_store_gov_pol_sys(boost); +store_gov_pol_sys(boostpulse); +show_store_gov_pol_sys(boostpulse_duration); +show_store_gov_pol_sys(io_is_busy); + +gov_sys_pol_attr_rw(target_loads); +gov_sys_pol_attr_rw(above_hispeed_delay); +gov_sys_pol_attr_rw(hispeed_freq); +gov_sys_pol_attr_rw(go_hispeed_load); +gov_sys_pol_attr_rw(min_sample_time); +gov_sys_pol_attr_rw(timer_rate); +gov_sys_pol_attr_rw(timer_slack); +gov_sys_pol_attr_rw(boost); +gov_sys_pol_attr_rw(boostpulse_duration); +gov_sys_pol_attr_rw(io_is_busy); + +static struct global_attr boostpulse_gov_sys = + __ATTR(boostpulse, 0200, NULL, store_boostpulse_gov_sys); + +static struct freq_attr boostpulse_gov_pol = + __ATTR(boostpulse, 0200, NULL, store_boostpulse_gov_pol); + +/* One Governor instance for entire system */ +static struct attribute *interactive_attributes_gov_sys[] = { + &target_loads_gov_sys.attr, + &above_hispeed_delay_gov_sys.attr, + &hispeed_freq_gov_sys.attr, + &go_hispeed_load_gov_sys.attr, + &min_sample_time_gov_sys.attr, + &timer_rate_gov_sys.attr, + &timer_slack_gov_sys.attr, + &boost_gov_sys.attr, + &boostpulse_gov_sys.attr, + &boostpulse_duration_gov_sys.attr, + &io_is_busy_gov_sys.attr, + NULL, +}; + +static struct attribute_group interactive_attr_group_gov_sys = { + .attrs = interactive_attributes_gov_sys, + .name = "interactive", +}; + +/* Per policy governor instance */ +static struct attribute *interactive_attributes_gov_pol[] = { + &target_loads_gov_pol.attr, + &above_hispeed_delay_gov_pol.attr, + &hispeed_freq_gov_pol.attr, + &go_hispeed_load_gov_pol.attr, + &min_sample_time_gov_pol.attr, + &timer_rate_gov_pol.attr, + &timer_slack_gov_pol.attr, + &boost_gov_pol.attr, + &boostpulse_gov_pol.attr, + &boostpulse_duration_gov_pol.attr, + &io_is_busy_gov_pol.attr, NULL, };

-static struct attribute_group interactive_attr_group = { - .attrs = interactive_attributes, +static struct attribute_group interactive_attr_group_gov_pol = { + .attrs = interactive_attributes_gov_pol, .name = "interactive", };

+static struct attribute_group *get_sysfs_attr(void) +{ + if (have_governor_per_policy()) + return &interactive_attr_group_gov_pol; + else + return &interactive_attr_group_gov_sys; +} + static int cpufreq_interactive_idle_notifier(struct notifier_block *nb, unsigned long val, void *data) @@ -976,15 +1061,88 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy, unsigned int j; struct cpufreq_interactive_cpuinfo *pcpu; struct cpufreq_frequency_table *freq_table; + struct cpufreq_interactive_tunables *tunables; + + if (have_governor_per_policy()) + tunables = policy->governor_data; + else + tunables = common_tunables; + + WARN_ON(!tunables && (event != CPUFREQ_GOV_POLICY_INIT));

switch (event) { + case CPUFREQ_GOV_POLICY_INIT: + if (have_governor_per_policy()) { + WARN_ON(tunables); + } else if (tunables) { + tunables->usage_count++; + policy->governor_data = tunables; + return 0; + } + + tunables = kzalloc(sizeof(*tunables), GFP_KERNEL); + if (!tunables) { + pr_err("%s: POLICY_INIT: kzalloc failed\n", __func__); + return -ENOMEM; + } + + rc = sysfs_create_group(get_governor_parent_kobj(policy), + get_sysfs_attr()); + if (rc) { + kfree(tunables); + return rc; + } + + tunables->usage_count = 1; + tunables->above_hispeed_delay = default_above_hispeed_delay; + tunables->nabove_hispeed_delay = + ARRAY_SIZE(default_above_hispeed_delay); + tunables->go_hispeed_load = DEFAULT_GO_HISPEED_LOAD; + tunables->target_loads = default_target_loads; + tunables->ntarget_loads = ARRAY_SIZE(default_target_loads); + tunables->min_sample_time = DEFAULT_MIN_SAMPLE_TIME; + tunables->timer_rate = DEFAULT_TIMER_RATE; + tunables->boostpulse_duration_val = DEFAULT_MIN_SAMPLE_TIME; + tunables->timer_slack_val = DEFAULT_TIMER_SLACK; + + spin_lock_init(&tunables->target_loads_lock); + spin_lock_init(&tunables->above_hispeed_delay_lock); + + if (!policy->governor->initialized) { + idle_notifier_register(&cpufreq_interactive_idle_nb); + cpufreq_register_notifier(&cpufreq_notifier_block, + CPUFREQ_TRANSITION_NOTIFIER); + } + + policy->governor_data = tunables; + if (!have_governor_per_policy()) + common_tunables = tunables; + + break; + + case CPUFREQ_GOV_POLICY_EXIT: + if (!--tunables->usage_count) { + if (policy->governor->initialized == 1) { + cpufreq_unregister_notifier(&cpufreq_notifier_block, + CPUFREQ_TRANSITION_NOTIFIER); + idle_notifier_unregister(&cpufreq_interactive_idle_nb); + } + + sysfs_remove_group(get_governor_parent_kobj(policy), + get_sysfs_attr()); + kfree(tunables); + common_tunables = NULL; + } + + policy->governor_data = NULL; + break; + case CPUFREQ_GOV_START: mutex_lock(&gov_lock);

- freq_table = - cpufreq_frequency_get_table(policy->cpu); - if (!hispeed_freq) - hispeed_freq = policy->max; + freq_table = cpufreq_frequency_get_table(policy->cpu); + if (!tunables->hispeed_freq) + tunables->hispeed_freq = policy->max;

for_each_cpu(j, policy->cpus) { unsigned long expires; @@ -999,11 +1157,12 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy, pcpu->hispeed_validate_time = pcpu->floor_validate_time; down_write(&pcpu->enable_sem); - expires = jiffies + usecs_to_jiffies(timer_rate); + expires = jiffies + + usecs_to_jiffies(tunables->timer_rate); pcpu->cpu_timer.expires = expires; add_timer_on(&pcpu->cpu_timer, j); - if (timer_slack_val >= 0) { - expires += usecs_to_jiffies(timer_slack_val); + if (tunables->timer_slack_val >= 0) { + expires += usecs_to_jiffies(tunables->timer_slack_val); pcpu->cpu_slack_timer.expires = expires; add_timer_on(&pcpu->cpu_slack_timer, j); } @@ -1011,25 +1170,6 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy, up_write(&pcpu->enable_sem); }

- /* - * Do not register the idle hook and create sysfs - * entries if we have already done so. - */ - if (++active_count > 1) { - mutex_unlock(&gov_lock); - return 0; - } - - rc = sysfs_create_group(cpufreq_global_kobject, - &interactive_attr_group); - if (rc) { - mutex_unlock(&gov_lock); - return rc; - } - - idle_notifier_register(&cpufreq_interactive_idle_nb); - cpufreq_register_notifier( - &cpufreq_notifier_block, CPUFREQ_TRANSITION_NOTIFIER); mutex_unlock(&gov_lock); break;

@@ -1044,18 +1184,7 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy, up_write(&pcpu->enable_sem); }

- if (--active_count > 0) { - mutex_unlock(&gov_lock); - return 0; - } - - cpufreq_unregister_notifier( - &cpufreq_notifier_block, CPUFREQ_TRANSITION_NOTIFIER); - idle_notifier_unregister(&cpufreq_interactive_idle_nb); - sysfs_remove_group(cpufreq_global_kobject, - &interactive_attr_group); mutex_unlock(&gov_lock); - break;

case CPUFREQ_GOV_LIMITS: @@ -1102,9 +1231,7 @@ static int __init cpufreq_interactive_init(void) init_rwsem(&pcpu->enable_sem); }

- spin_lock_init(&target_loads_lock); spin_lock_init(&speedchange_cpumask_lock); - spin_lock_init(&above_hispeed_delay_lock); mutex_init(&gov_lock); speedchange_task = kthread_create(cpufreq_interactive_speedchange_task, NULL,

- android-kernel@googlegroups.com, a subscription-only list

On Thu, May 16, 2013 at 2:28 AM, Viresh Kumar viresh.kumar@linaro.org wrote:

Generic version of get_cpu_idle_time() is present in cpufreq.c file. We must not replicate that for Interactive governor, rather use the same one. Lets do it.

Thanks, will touch this up to simply say "Use generic get_cpu_idle_time().", which didn't exist in the 3.4 code.

Signed-off-by: Viresh Kumar viresh.kumar@linaro.org

drivers/cpufreq/cpufreq_interactive.c | 42 ++--------------------------------- 1 file changed, 2 insertions(+), 40 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c index baf8740..4ed2fed 100644 --- a/drivers/cpufreq/cpufreq_interactive.c +++ b/drivers/cpufreq/cpufreq_interactive.c @@ -30,8 +30,6 @@ #include <linux/workqueue.h> #include <linux/kthread.h> #include <linux/slab.h> -#include <linux/kernel_stat.h> -#include <asm/cputime.h>

#define CREATE_TRACE_POINTS #include <trace/events/cpufreq_interactive.h> @@ -130,42 +128,6 @@ struct cpufreq_governor cpufreq_gov_interactive = { .owner = THIS_MODULE, };

-static inline cputime64_t get_cpu_idle_time_jiffy(unsigned int cpu,

•                                             cputime64_t *wall)
  

-{

•   u64 idle_time;
  

•   u64 cur_wall_time;
  

•   u64 busy_time;
  

•   cur_wall_time = jiffies64_to_cputime64(get_jiffies_64());
  

•   busy_time  = kcpustat_cpu(cpu).cpustat[CPUTIME_USER];
  

•   busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_SYSTEM];
  

•   busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_IRQ];
  

•   busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_SOFTIRQ];
  

•   busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_STEAL];
  

•   busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_NICE];
  

•   idle_time = cur_wall_time - busy_time;
  

•   if (wall)
  

•           *wall = jiffies_to_usecs(cur_wall_time);
  

•   return jiffies_to_usecs(idle_time);
  

-}

-static inline cputime64_t get_cpu_idle_time(unsigned int cpu,

•                                       cputime64_t *wall)
  

-{

•   u64 idle_time = get_cpu_idle_time_us(cpu, wall);
  

•   if (idle_time == -1ULL)
  

•           idle_time = get_cpu_idle_time_jiffy(cpu, wall);
  

•   else if (!io_is_busy)
  

•           idle_time += get_cpu_iowait_time_us(cpu, wall);
  

•   return idle_time;
  

-}

static void cpufreq_interactive_timer_resched( struct cpufreq_interactive_cpuinfo *pcpu) { @@ -181,7 +143,7 @@ static void cpufreq_interactive_timer_resched( spin_lock_irqsave(&pcpu->load_lock, flags); pcpu->time_in_idle = get_cpu_idle_time(smp_processor_id(),

•                                &pcpu->time_in_idle_timestamp);
  

•                                &pcpu->time_in_idle_timestamp, io_is_busy);
    pcpu->cputime_speedadj = 0;
    pcpu->cputime_speedadj_timestamp = pcpu->time_in_idle_timestamp;
    spin_unlock_irqrestore(&pcpu->load_lock, flags);
  

@@ -317,7 +279,7 @@ static u64 update_load(int cpu) unsigned int delta_time; u64 active_time;

•   now_idle = get_cpu_idle_time(cpu, &now);
  

•   now_idle = get_cpu_idle_time(cpu, &now, io_is_busy);
    delta_idle = (unsigned int)(now_idle - pcpu->time_in_idle);
    delta_time = (unsigned int)(now - pcpu->time_in_idle_timestamp);
    active_time = delta_time - delta_idle;
  

-- 1.7.12.rc2.18.g61b472e

Will merge, thanks.

On Thu, May 16, 2013 at 2:28 AM, Viresh Kumar viresh.kumar@linaro.org wrote:

This moves definition of cpufreq_gov_interactive towards the bottom of file, so that we don't have to add prototype of cpufreq_governor_interactive() in the beginning of file.

Signed-off-by: Viresh Kumar viresh.kumar@linaro.org

drivers/cpufreq/cpufreq_interactive.c | 23 ++++++++++------------- 1 file changed, 10 insertions(+), 13 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c index baa5638..8821873 100644 --- a/drivers/cpufreq/cpufreq_interactive.c +++ b/drivers/cpufreq/cpufreq_interactive.c @@ -115,19 +115,6 @@ static int timer_slack_val = DEFAULT_TIMER_SLACK;

static bool io_is_busy;

-static int cpufreq_governor_interactive(struct cpufreq_policy *policy,

•           unsigned int event);
  

-#ifndef CONFIG_CPU_FREQ_DEFAULT_GOV_INTERACTIVE -static -#endif -struct cpufreq_governor cpufreq_gov_interactive = {

•   .name = "interactive",
  

•   .governor = cpufreq_governor_interactive,
  

•   .max_transition_latency = 10000000,
  

•   .owner = THIS_MODULE,
  

-};

static void cpufreq_interactive_timer_resched( struct cpufreq_interactive_cpuinfo *pcpu) { @@ -1083,6 +1070,16 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy, return 0; }

+#ifndef CONFIG_CPU_FREQ_DEFAULT_GOV_INTERACTIVE +static +#endif +struct cpufreq_governor cpufreq_gov_interactive = {

•   .name = "interactive",
  

•   .governor = cpufreq_governor_interactive,
  

•   .max_transition_latency = 10000000,
  

•   .owner = THIS_MODULE,
  

+};

static void cpufreq_interactive_nop_timer(unsigned long data) { } -- 1.7.12.rc2.18.g61b472e

Thanks Viresh, I'm currently embroiled in stuff based on earlier kernels but have this on my list to merge to android-3.10 when it's ready.

Todd

On Thu, May 16, 2013 at 2:28 AM, Viresh Kumar viresh.kumar@linaro.org wrote:

If we have a multi-package system, where we have multiple instances of struct policy (per package), currently we can't have multiple instances of same governor. i.e. We can't have multiple instances of Interactive governor for multiple packages.

This is a bottleneck for multicluster system, where we want different packages to use Interactive governor, but with different tunables.

This patch uses the infrastructure provided by earlier patches pushed in Mainline in v3.10-rc1/rc2 and implements per policy instances of Interactive governor.

Signed-off-by: Viresh Kumar viresh.kumar@linaro.org

drivers/cpufreq/cpufreq_interactive.c | 615 ++++++++++++++++++++-------------- 1 file changed, 371 insertions(+), 244 deletions(-)

diff --git a/drivers/cpufreq/cpufreq_interactive.c b/drivers/cpufreq/cpufreq_interactive.c index 8821873..05733ea 100644 --- a/drivers/cpufreq/cpufreq_interactive.c +++ b/drivers/cpufreq/cpufreq_interactive.c @@ -30,12 +30,11 @@ #include <linux/workqueue.h> #include <linux/kthread.h> #include <linux/slab.h> +#include "cpufreq_governor.h"

#define CREATE_TRACE_POINTS #include <trace/events/cpufreq_interactive.h>

-static int active_count;

struct cpufreq_interactive_cpuinfo { struct timer_list cpu_timer; struct timer_list cpu_slack_timer; @@ -62,110 +61,131 @@ static cpumask_t speedchange_cpumask; static spinlock_t speedchange_cpumask_lock; static struct mutex gov_lock;

-/* Hi speed to bump to from lo speed when load burst (default max) */ -static unsigned int hispeed_freq;

-/* Go to hi speed when CPU load at or above this value. */ -#define DEFAULT_GO_HISPEED_LOAD 99 -static unsigned long go_hispeed_load = DEFAULT_GO_HISPEED_LOAD;

-/* Target load. Lower values result in higher CPU speeds. */ #define DEFAULT_TARGET_LOAD 90 static unsigned int default_target_loads[] = {DEFAULT_TARGET_LOAD}; -static spinlock_t target_loads_lock; -static unsigned int *target_loads = default_target_loads; -static int ntarget_loads = ARRAY_SIZE(default_target_loads);

-/*

• • The minimum amount of time to spend at a frequency before we can ramp down.
• */

-#define DEFAULT_MIN_SAMPLE_TIME (80 * USEC_PER_MSEC) -static unsigned long min_sample_time = DEFAULT_MIN_SAMPLE_TIME;

-/*

• • The sample rate of the timer used to increase frequency
• */

#define DEFAULT_TIMER_RATE (20 * USEC_PER_MSEC) -static unsigned long timer_rate = DEFAULT_TIMER_RATE;

-/*

• • Wait this long before raising speed above hispeed, by default a single
• • timer interval.
• */

#define DEFAULT_ABOVE_HISPEED_DELAY DEFAULT_TIMER_RATE static unsigned int default_above_hispeed_delay[] = { DEFAULT_ABOVE_HISPEED_DELAY }; -static spinlock_t above_hispeed_delay_lock; -static unsigned int *above_hispeed_delay = default_above_hispeed_delay; -static int nabove_hispeed_delay = ARRAY_SIZE(default_above_hispeed_delay);

-/* Non-zero means indefinite speed boost active */ -static int boost_val; -/* Duration of a boot pulse in usecs */ -static int boostpulse_duration_val = DEFAULT_MIN_SAMPLE_TIME; -/* End time of boost pulse in ktime converted to usecs */ -static u64 boostpulse_endtime; +struct cpufreq_interactive_tunables {

•   int usage_count;
  
  

-/*

• • Max additional time to wait in idle, beyond timer_rate, at speeds above
• • minimum before wakeup to reduce speed, or -1 if unnecessary.
• */

•   /* Hi speed to bump to from lo speed when load burst (default max) */
  

•   unsigned int hispeed_freq;
  

•   /* Go to hi speed when CPU load at or above this value. */
  

+#define DEFAULT_GO_HISPEED_LOAD 99

•   unsigned long go_hispeed_load;
  

•   /* Target load. Lower values result in higher CPU speeds. */
  

•   spinlock_t target_loads_lock;
  

•   unsigned int *target_loads;
  

•   int ntarget_loads;
  

•   /*
  

•    * The minimum amount of time to spend at a frequency before we can ramp
  

•    * down.
  

•    */
  

+#define DEFAULT_MIN_SAMPLE_TIME (80 * USEC_PER_MSEC)

•   unsigned long min_sample_time;
  

•   /*
  

•    * The sample rate of the timer used to increase frequency
  

•    */
  

•   unsigned long timer_rate;
  

•   /*
  

•    * Wait this long before raising speed above hispeed, by default a
  

•    * single timer interval.
  

•    */
  

•   spinlock_t above_hispeed_delay_lock;
  

•   unsigned int *above_hispeed_delay;
  

•   int nabove_hispeed_delay;
  

•   /* Non-zero means indefinite speed boost active */
  

•   int boost_val;
  

•   /* Duration of a boot pulse in usecs */
  

•   int boostpulse_duration_val;
  

•   /* End time of boost pulse in ktime converted to usecs */
  

•   u64 boostpulse_endtime;
  

•   /*
  

•    * Max additional time to wait in idle, beyond timer_rate, at speeds
  

•    * above minimum before wakeup to reduce speed, or -1 if unnecessary.
  

•    */
  

#define DEFAULT_TIMER_SLACK (4 * DEFAULT_TIMER_RATE) -static int timer_slack_val = DEFAULT_TIMER_SLACK;

•   int timer_slack_val;
  

•   bool io_is_busy;
  

+};

-static bool io_is_busy; +/* For cases where we have single governor instance for system */ +struct cpufreq_interactive_tunables *common_tunables;

+static struct attribute_group *get_sysfs_attr(void);

static void cpufreq_interactive_timer_resched( struct cpufreq_interactive_cpuinfo *pcpu) {

•   unsigned long expires = jiffies + usecs_to_jiffies(timer_rate);
  

•   struct cpufreq_interactive_tunables *tunables =
  

•           pcpu->policy->governor_data;
  

•   unsigned long expires = jiffies +
  

•           usecs_to_jiffies(tunables->timer_rate);
    unsigned long flags;
  
    mod_timer_pinned(&pcpu->cpu_timer, expires);
  

•   if (timer_slack_val >= 0 && pcpu->target_freq > pcpu->policy->min) {
  

•           expires += usecs_to_jiffies(timer_slack_val);
  

•   if (tunables->timer_slack_val >= 0 &&
  

•                   pcpu->target_freq > pcpu->policy->min) {
  

•           expires += usecs_to_jiffies(tunables->timer_slack_val);
            mod_timer_pinned(&pcpu->cpu_slack_timer, expires);
    }
  
    spin_lock_irqsave(&pcpu->load_lock, flags);
    pcpu->time_in_idle =
            get_cpu_idle_time(smp_processor_id(),
  

•                                &pcpu->time_in_idle_timestamp, io_is_busy);
  

•                                &pcpu->time_in_idle_timestamp,
  

•                                tunables->io_is_busy);
    pcpu->cputime_speedadj = 0;
    pcpu->cputime_speedadj_timestamp = pcpu->time_in_idle_timestamp;
    spin_unlock_irqrestore(&pcpu->load_lock, flags);
  

}

-static unsigned int freq_to_above_hispeed_delay(unsigned int freq) +static unsigned int freq_to_above_hispeed_delay(

•           struct cpufreq_interactive_tunables *tunables,
  

•           unsigned int freq)
  

{ int i; unsigned int ret; unsigned long flags;

•   spin_lock_irqsave(&above_hispeed_delay_lock, flags);
  

•   spin_lock_irqsave(&tunables->above_hispeed_delay_lock, flags);
  
  

•   for (i = 0; i < nabove_hispeed_delay - 1 &&
  

•                   freq >= above_hispeed_delay[i+1]; i += 2)
  

•   for (i = 0; i < tunables->nabove_hispeed_delay - 1 &&
  

•                   freq >= tunables->above_hispeed_delay[i+1]; i += 2)
            ;
  
  

•   ret = above_hispeed_delay[i];
  

•   spin_unlock_irqrestore(&above_hispeed_delay_lock, flags);
  

•   ret = tunables->above_hispeed_delay[i];
  

•   spin_unlock_irqrestore(&tunables->above_hispeed_delay_lock, flags);
    return ret;
  

}

-static unsigned int freq_to_targetload(unsigned int freq) +static unsigned int freq_to_targetload(struct cpufreq_interactive_tunables

•           *tunables, unsigned int freq)
  

{ int i; unsigned int ret; unsigned long flags;

•   spin_lock_irqsave(&target_loads_lock, flags);
  

•   spin_lock_irqsave(&tunables->target_loads_lock, flags);
  
  

•   for (i = 0; i < ntarget_loads - 1 && freq >= target_loads[i+1]; i += 2)
  

•   for (i = 0; i < tunables->ntarget_loads - 1 &&
  

•                   freq >= tunables->target_loads[i+1]; i += 2)
            ;
  
  

•   ret = target_loads[i];
  

•   spin_unlock_irqrestore(&target_loads_lock, flags);
  

•   ret = tunables->target_loads[i];
  

•   spin_unlock_irqrestore(&tunables->target_loads_lock, flags);
    return ret;
  

}

@@ -174,9 +194,8 @@ static unsigned int freq_to_targetload(unsigned int freq)

• choose_freq() will find the minimum frequency that does not exceed its
• target load given the current load.

*/

-static unsigned int choose_freq(

•   struct cpufreq_interactive_cpuinfo *pcpu, unsigned int loadadjfreq)
  

+static unsigned int choose_freq(struct cpufreq_interactive_cpuinfo *pcpu,

•           unsigned int loadadjfreq)
  

{ unsigned int freq = pcpu->policy->cur; unsigned int prevfreq, freqmin, freqmax; @@ -188,7 +207,7 @@ static unsigned int choose_freq(

    do {
            prevfreq = freq;

•           tl = freq_to_targetload(freq);
  

•           tl = freq_to_targetload(pcpu->policy->governor_data, freq);
  
            /*
             * Find the lowest frequency where the computed load is less
  

@@ -260,13 +279,15 @@ static unsigned int choose_freq( static u64 update_load(int cpu) { struct cpufreq_interactive_cpuinfo *pcpu = &per_cpu(cpuinfo, cpu);

•   struct cpufreq_interactive_tunables *tunables =
  

•           pcpu->policy->governor_data;
    u64 now;
    u64 now_idle;
    unsigned int delta_idle;
    unsigned int delta_time;
    u64 active_time;
  
  

•   now_idle = get_cpu_idle_time(cpu, &now, io_is_busy);
  

•   now_idle = get_cpu_idle_time(cpu, &now, tunables->io_is_busy);
    delta_idle = (unsigned int)(now_idle - pcpu->time_in_idle);
    delta_time = (unsigned int)(now - pcpu->time_in_idle_timestamp);
    active_time = delta_time - delta_idle;
  

@@ -285,6 +306,8 @@ static void cpufreq_interactive_timer(unsigned long data) int cpu_load; struct cpufreq_interactive_cpuinfo *pcpu = &per_cpu(cpuinfo, data);

•   struct cpufreq_interactive_tunables *tunables =
  

•           pcpu->policy->governor_data;
    unsigned int new_freq;
    unsigned int loadadjfreq;
    unsigned int index;
  

@@ -308,25 +331,25 @@ static void cpufreq_interactive_timer(unsigned long data) do_div(cputime_speedadj, delta_time); loadadjfreq = (unsigned int)cputime_speedadj * 100; cpu_load = loadadjfreq / pcpu->target_freq;

•   boosted = boost_val || now < boostpulse_endtime;
  

•   boosted = tunables->boost_val || now < tunables->boostpulse_endtime;
  
  

•   if (cpu_load >= go_hispeed_load || boosted) {
  

•           if (pcpu->target_freq < hispeed_freq) {
  

•                   new_freq = hispeed_freq;
  

•   if (cpu_load >= tunables->go_hispeed_load || boosted) {
  

•           if (pcpu->target_freq < tunables->hispeed_freq) {
  

•                   new_freq = tunables->hispeed_freq;
            } else {
                    new_freq = choose_freq(pcpu, loadadjfreq);
  
  

•                   if (new_freq < hispeed_freq)
  

•                           new_freq = hispeed_freq;
  

•                   if (new_freq < tunables->hispeed_freq)
  

•                           new_freq = tunables->hispeed_freq;
            }
    } else {
            new_freq = choose_freq(pcpu, loadadjfreq);
    }
  
  

•   if (pcpu->target_freq >= hispeed_freq &&
  

•   if (pcpu->target_freq >= tunables->hispeed_freq &&
        new_freq > pcpu->target_freq &&
        now - pcpu->hispeed_validate_time <
  

•       freq_to_above_hispeed_delay(pcpu->target_freq)) {
  

•       freq_to_above_hispeed_delay(tunables, pcpu->target_freq)) {
            trace_cpufreq_interactive_notyet(
                    data, cpu_load, pcpu->target_freq,
                    pcpu->policy->cur, new_freq);
  

@@ -350,7 +373,8 @@ static void cpufreq_interactive_timer(unsigned long data) * floor frequency for the minimum sample time since last validated. */ if (new_freq < pcpu->floor_freq) {

•           if (now - pcpu->floor_validate_time < min_sample_time) {
  

•           if (now - pcpu->floor_validate_time <
  

•                           tunables->min_sample_time) {
                    trace_cpufreq_interactive_notyet(
                            data, cpu_load, pcpu->target_freq,
                            pcpu->policy->cur, new_freq);
  

@@ -366,7 +390,7 @@ static void cpufreq_interactive_timer(unsigned long data) * (or the indefinite boost is turned off). */

•   if (!boosted || new_freq > hispeed_freq) {
  

•   if (!boosted || new_freq > tunables->hispeed_freq) {
            pcpu->floor_freq = new_freq;
            pcpu->floor_validate_time = now;
    }
  

@@ -527,14 +551,16 @@ static void cpufreq_interactive_boost(void) int anyboost = 0; unsigned long flags; struct cpufreq_interactive_cpuinfo *pcpu;

•   struct cpufreq_interactive_tunables *tunables;
  
    spin_lock_irqsave(&speedchange_cpumask_lock, flags);
  
    for_each_online_cpu(i) {
            pcpu = &per_cpu(cpuinfo, i);
  

•           tunables = pcpu->policy->governor_data;
  
  

•           if (pcpu->target_freq < hispeed_freq) {
  

•                   pcpu->target_freq = hispeed_freq;
  

•           if (pcpu->target_freq < tunables->hispeed_freq) {
  

•                   pcpu->target_freq = tunables->hispeed_freq;
                    cpumask_set_cpu(i, &speedchange_cpumask);
                    pcpu->hispeed_validate_time =
                            ktime_to_us(ktime_get());
  

@@ -546,7 +572,7 @@ static void cpufreq_interactive_boost(void) * validated. */

•           pcpu->floor_freq = hispeed_freq;
  

•           pcpu->floor_freq = tunables->hispeed_freq;
            pcpu->floor_validate_time = ktime_to_us(ktime_get());
    }
  
  

@@ -635,27 +661,26 @@ err: return ERR_PTR(err); }

-static ssize_t show_target_loads(

•   struct kobject *kobj, struct attribute *attr, char *buf)
  

+static ssize_t show_target_loads(struct cpufreq_interactive_tunables *tunables,

•           char *buf)
  

{ int i; ssize_t ret = 0; unsigned long flags;

•   spin_lock_irqsave(&target_loads_lock, flags);
  

•   spin_lock_irqsave(&tunables->target_loads_lock, flags);
  
  

•   for (i = 0; i < ntarget_loads; i++)
  

•           ret += sprintf(buf + ret, "%u%s", target_loads[i],
  

•   for (i = 0; i < tunables->ntarget_loads; i++)
  

•           ret += sprintf(buf + ret, "%u%s", tunables->target_loads[i],
                           i & 0x1 ? ":" : " ");
  
    ret += sprintf(buf + ret, "\n");
  

•   spin_unlock_irqrestore(&target_loads_lock, flags);
  

•   spin_unlock_irqrestore(&tunables->target_loads_lock, flags);
    return ret;
  

}

-static ssize_t store_target_loads(

•   struct kobject *kobj, struct attribute *attr, const char *buf,
  

•   size_t count)
  

+static ssize_t store_target_loads(struct cpufreq_interactive_tunables *tunables,

•           const char *buf, size_t count)
  

{ int ntokens; unsigned int *new_target_loads = NULL; @@ -665,40 +690,36 @@ static ssize_t store_target_loads( if (IS_ERR(new_target_loads)) return PTR_RET(new_target_loads);

•   spin_lock_irqsave(&target_loads_lock, flags);
  

•   if (target_loads != default_target_loads)
  

•           kfree(target_loads);
  

•   target_loads = new_target_loads;
  

•   ntarget_loads = ntokens;
  

•   spin_unlock_irqrestore(&target_loads_lock, flags);
  

•   spin_lock_irqsave(&tunables->target_loads_lock, flags);
  

•   if (tunables->target_loads != default_target_loads)
  

•           kfree(tunables->target_loads);
  

•   tunables->target_loads = new_target_loads;
  

•   tunables->ntarget_loads = ntokens;
  

•   spin_unlock_irqrestore(&tunables->target_loads_lock, flags);
    return count;
  

}

-static struct global_attr target_loads_attr =

•   __ATTR(target_loads, S_IRUGO | S_IWUSR,
  

•           show_target_loads, store_target_loads);
  

-static ssize_t show_above_hispeed_delay(

•   struct kobject *kobj, struct attribute *attr, char *buf)
  

+static ssize_t show_above_hispeed_delay(struct cpufreq_interactive_tunables

•           *tunables, char *buf)
  

{ int i; ssize_t ret = 0; unsigned long flags;

•   spin_lock_irqsave(&above_hispeed_delay_lock, flags);
  

•   spin_lock_irqsave(&tunables->above_hispeed_delay_lock, flags);
  
  

•   for (i = 0; i < nabove_hispeed_delay; i++)
  

•           ret += sprintf(buf + ret, "%u%s", above_hispeed_delay[i],
  

•   for (i = 0; i < tunables->nabove_hispeed_delay; i++)
  

•           ret += sprintf(buf + ret, "%u%s",
  

•                          tunables->above_hispeed_delay[i],
                           i & 0x1 ? ":" : " ");
  
    ret += sprintf(buf + ret, "\n");
  

•   spin_unlock_irqrestore(&above_hispeed_delay_lock, flags);
  

•   spin_unlock_irqrestore(&tunables->above_hispeed_delay_lock, flags);
    return ret;
  

}

-static ssize_t store_above_hispeed_delay(

•   struct kobject *kobj, struct attribute *attr, const char *buf,
  

•   size_t count)
  

+static ssize_t store_above_hispeed_delay(struct cpufreq_interactive_tunables

•           *tunables, const char *buf, size_t count)
  

{ int ntokens; unsigned int *new_above_hispeed_delay = NULL; @@ -708,29 +729,24 @@ static ssize_t store_above_hispeed_delay( if (IS_ERR(new_above_hispeed_delay)) return PTR_RET(new_above_hispeed_delay);

•   spin_lock_irqsave(&above_hispeed_delay_lock, flags);
  

•   if (above_hispeed_delay != default_above_hispeed_delay)
  

•           kfree(above_hispeed_delay);
  

•   above_hispeed_delay = new_above_hispeed_delay;
  

•   nabove_hispeed_delay = ntokens;
  

•   spin_unlock_irqrestore(&above_hispeed_delay_lock, flags);
  

•   spin_lock_irqsave(&tunables->above_hispeed_delay_lock, flags);
  

•   if (tunables->above_hispeed_delay != default_above_hispeed_delay)
  

•           kfree(tunables->above_hispeed_delay);
  

•   tunables->above_hispeed_delay = new_above_hispeed_delay;
  

•   tunables->nabove_hispeed_delay = ntokens;
  

•   spin_unlock_irqrestore(&tunables->above_hispeed_delay_lock, flags);
    return count;
  
  

}

-static struct global_attr above_hispeed_delay_attr =

•   __ATTR(above_hispeed_delay, S_IRUGO | S_IWUSR,
  

•           show_above_hispeed_delay, store_above_hispeed_delay);
  

-static ssize_t show_hispeed_freq(struct kobject *kobj,

•                            struct attribute *attr, char *buf)
  

+static ssize_t show_hispeed_freq(struct cpufreq_interactive_tunables *tunables,

•           char *buf)
  

{

•   return sprintf(buf, "%u\n", hispeed_freq);
  

•   return sprintf(buf, "%u\n", tunables->hispeed_freq);
  

}

-static ssize_t store_hispeed_freq(struct kobject *kobj,

•                             struct attribute *attr, const char *buf,
  

•                             size_t count)
  

+static ssize_t store_hispeed_freq(struct cpufreq_interactive_tunables *tunables,

•           const char *buf, size_t count)
  

{ int ret; long unsigned int val; @@ -738,22 +754,18 @@ static ssize_t store_hispeed_freq(struct kobject *kobj, ret = strict_strtoul(buf, 0, &val); if (ret < 0) return ret;

•   hispeed_freq = val;
  

•   tunables->hispeed_freq = val;
    return count;
  

}

-static struct global_attr hispeed_freq_attr = __ATTR(hispeed_freq, 0644,

•           show_hispeed_freq, store_hispeed_freq);
  

-static ssize_t show_go_hispeed_load(struct kobject *kobj,

•                                struct attribute *attr, char *buf)
  

+static ssize_t show_go_hispeed_load(struct cpufreq_interactive_tunables

•           *tunables, char *buf)
  

{

•   return sprintf(buf, "%lu\n", go_hispeed_load);
  

•   return sprintf(buf, "%lu\n", tunables->go_hispeed_load);
  

}

-static ssize_t store_go_hispeed_load(struct kobject *kobj,

•                   struct attribute *attr, const char *buf, size_t count)
  

+static ssize_t store_go_hispeed_load(struct cpufreq_interactive_tunables

•           *tunables, const char *buf, size_t count)
  

{ int ret; unsigned long val; @@ -761,21 +773,18 @@ static ssize_t store_go_hispeed_load(struct kobject *kobj, ret = strict_strtoul(buf, 0, &val); if (ret < 0) return ret;

•   go_hispeed_load = val;
  

•   tunables->go_hispeed_load = val;
    return count;
  

}

-static struct global_attr go_hispeed_load_attr = __ATTR(go_hispeed_load, 0644,

•           show_go_hispeed_load, store_go_hispeed_load);
  

-static ssize_t show_min_sample_time(struct kobject *kobj,

•                           struct attribute *attr, char *buf)
  

+static ssize_t show_min_sample_time(struct cpufreq_interactive_tunables

•           *tunables, char *buf)
  

{

•   return sprintf(buf, "%lu\n", min_sample_time);
  

•   return sprintf(buf, "%lu\n", tunables->min_sample_time);
  

}

-static ssize_t store_min_sample_time(struct kobject *kobj,

•                   struct attribute *attr, const char *buf, size_t count)
  

+static ssize_t store_min_sample_time(struct cpufreq_interactive_tunables

•           *tunables, const char *buf, size_t count)
  

{ int ret; unsigned long val; @@ -783,21 +792,18 @@ static ssize_t store_min_sample_time(struct kobject *kobj, ret = strict_strtoul(buf, 0, &val); if (ret < 0) return ret;

•   min_sample_time = val;
  

•   tunables->min_sample_time = val;
    return count;
  

}

-static struct global_attr min_sample_time_attr = __ATTR(min_sample_time, 0644,

•           show_min_sample_time, store_min_sample_time);
  

-static ssize_t show_timer_rate(struct kobject *kobj,

•                   struct attribute *attr, char *buf)
  

+static ssize_t show_timer_rate(struct cpufreq_interactive_tunables *tunables,

•           char *buf)
  

{

•   return sprintf(buf, "%lu\n", timer_rate);
  

•   return sprintf(buf, "%lu\n", tunables->timer_rate);
  

}

-static ssize_t store_timer_rate(struct kobject *kobj,

•                   struct attribute *attr, const char *buf, size_t count)
  

+static ssize_t store_timer_rate(struct cpufreq_interactive_tunables *tunables,

•           const char *buf, size_t count)
  

{ int ret; unsigned long val; @@ -805,22 +811,18 @@ static ssize_t store_timer_rate(struct kobject *kobj, ret = strict_strtoul(buf, 0, &val); if (ret < 0) return ret;

•   timer_rate = val;
  

•   tunables->timer_rate = val;
    return count;
  

}

-static struct global_attr timer_rate_attr = __ATTR(timer_rate, 0644,

•           show_timer_rate, store_timer_rate);
  

-static ssize_t show_timer_slack(

•   struct kobject *kobj, struct attribute *attr, char *buf)
  

+static ssize_t show_timer_slack(struct cpufreq_interactive_tunables *tunables,

•           char *buf)
  

{

•   return sprintf(buf, "%d\n", timer_slack_val);
  

•   return sprintf(buf, "%d\n", tunables->timer_slack_val);
  

}

-static ssize_t store_timer_slack(

•   struct kobject *kobj, struct attribute *attr, const char *buf,
  

•   size_t count)
  

+static ssize_t store_timer_slack(struct cpufreq_interactive_tunables *tunables,

•           const char *buf, size_t count)
  

{ int ret; unsigned long val; @@ -829,19 +831,17 @@ static ssize_t store_timer_slack( if (ret < 0) return ret;

•   timer_slack_val = val;
  

•   tunables->timer_slack_val = val;
    return count;
  

}

-define_one_global_rw(timer_slack);

-static ssize_t show_boost(struct kobject *kobj, struct attribute *attr, +static ssize_t show_boost(struct cpufreq_interactive_tunables *tunables, char *buf) {

•   return sprintf(buf, "%d\n", boost_val);
  

•   return sprintf(buf, "%d\n", tunables->boost_val);
  

}

-static ssize_t store_boost(struct kobject *kobj, struct attribute *attr, +static ssize_t store_boost(struct cpufreq_interactive_tunables *tunables, const char *buf, size_t count) { int ret; @@ -851,9 +851,9 @@ static ssize_t store_boost(struct kobject *kobj, struct attribute *attr, if (ret < 0) return ret;

•   boost_val = val;
  

•   tunables->boost_val = val;
  
  

•   if (boost_val) {
  

•   if (tunables->boost_val) {
            trace_cpufreq_interactive_boost("on");
            cpufreq_interactive_boost();
    } else {
  

@@ -863,9 +863,7 @@ static ssize_t store_boost(struct kobject *kobj, struct attribute *attr, return count; }

-define_one_global_rw(boost);

-static ssize_t store_boostpulse(struct kobject *kobj, struct attribute *attr, +static ssize_t store_boostpulse(struct cpufreq_interactive_tunables *tunables, const char *buf, size_t count) { int ret; @@ -875,24 +873,21 @@ static ssize_t store_boostpulse(struct kobject *kobj, struct attribute *attr, if (ret < 0) return ret;

•   boostpulse_endtime = ktime_to_us(ktime_get()) + boostpulse_duration_val;
  

•   tunables->boostpulse_endtime = ktime_to_us(ktime_get()) +
  

•           tunables->boostpulse_duration_val;
    trace_cpufreq_interactive_boost("pulse");
    cpufreq_interactive_boost();
    return count;
  

}

-static struct global_attr boostpulse =

•   __ATTR(boostpulse, 0200, NULL, store_boostpulse);
  

-static ssize_t show_boostpulse_duration(

•   struct kobject *kobj, struct attribute *attr, char *buf)
  

+static ssize_t show_boostpulse_duration(struct cpufreq_interactive_tunables

•           *tunables, char *buf)
  

{

•   return sprintf(buf, "%d\n", boostpulse_duration_val);
  

•   return sprintf(buf, "%d\n", tunables->boostpulse_duration_val);
  

}

-static ssize_t store_boostpulse_duration(

•   struct kobject *kobj, struct attribute *attr, const char *buf,
  

•   size_t count)
  

+static ssize_t store_boostpulse_duration(struct cpufreq_interactive_tunables

•           *tunables, const char *buf, size_t count)
  

{ int ret; unsigned long val; @@ -901,20 +896,18 @@ static ssize_t store_boostpulse_duration( if (ret < 0) return ret;

•   boostpulse_duration_val = val;
  

•   tunables->boostpulse_duration_val = val;
    return count;
  

}

-define_one_global_rw(boostpulse_duration);

-static ssize_t show_io_is_busy(struct kobject *kobj,

•                   struct attribute *attr, char *buf)
  

+static ssize_t show_io_is_busy(struct cpufreq_interactive_tunables *tunables,

•           char *buf)
  

{

•   return sprintf(buf, "%u\n", io_is_busy);
  

•   return sprintf(buf, "%u\n", tunables->io_is_busy);
  

}

-static ssize_t store_io_is_busy(struct kobject *kobj,

•                   struct attribute *attr, const char *buf, size_t count)
  

+static ssize_t store_io_is_busy(struct cpufreq_interactive_tunables *tunables,

•           const char *buf, size_t count)
  

{ int ret; unsigned long val; @@ -922,33 +915,125 @@ static ssize_t store_io_is_busy(struct kobject *kobj, ret = kstrtoul(buf, 0, &val); if (ret < 0) return ret;

•   io_is_busy = val;
  

•   tunables->io_is_busy = val;
    return count;
  

}

-static struct global_attr io_is_busy_attr = __ATTR(io_is_busy, 0644,

•           show_io_is_busy, store_io_is_busy);
  

-static struct attribute *interactive_attributes[] = {

•   &target_loads_attr.attr,
  

•   &above_hispeed_delay_attr.attr,
  

•   &hispeed_freq_attr.attr,
  

•   &go_hispeed_load_attr.attr,
  

•   &min_sample_time_attr.attr,
  

•   &timer_rate_attr.attr,
  

•   &timer_slack.attr,
  

•   &boost.attr,
  

•   &boostpulse.attr,
  

•   &boostpulse_duration.attr,
  

•   &io_is_busy_attr.attr,
  

+/*

• • Create show/store routines
• • • sys: One governor instance for complete SYSTEM
• • • pol: One governor instance per struct cpufreq_policy
• */

+#define show_gov_pol_sys(file_name) \ +static ssize_t show_##file_name##_gov_sys \ +(struct kobject *kobj, struct attribute *attr, char *buf) \ +{ \

•   return show_##file_name(common_tunables, buf);                  \
  

+} \

•                                                                   \
  

+static ssize_t show_##file_name##_gov_pol \ +(struct cpufreq_policy *policy, char *buf) \ +{ \

•   return show_##file_name(policy->governor_data, buf);            \
  

+}

+#define store_gov_pol_sys(file_name) \ +static ssize_t store_##file_name##_gov_sys \ +(struct kobject *kobj, struct attribute *attr, const char *buf, \

•   size_t count)                                                   \
  

+{ \

•   return store_##file_name(common_tunables, buf, count);          \
  

+} \

•                                                                   \
  

+static ssize_t store_##file_name##_gov_pol \ +(struct cpufreq_policy *policy, const char *buf, size_t count) \ +{ \

•   return store_##file_name(policy->governor_data, buf, count);    \
  

+}

+#define show_store_gov_pol_sys(file_name) \ +show_gov_pol_sys(file_name); \ +store_gov_pol_sys(file_name)

+show_store_gov_pol_sys(target_loads); +show_store_gov_pol_sys(above_hispeed_delay); +show_store_gov_pol_sys(hispeed_freq); +show_store_gov_pol_sys(go_hispeed_load); +show_store_gov_pol_sys(min_sample_time); +show_store_gov_pol_sys(timer_rate); +show_store_gov_pol_sys(timer_slack); +show_store_gov_pol_sys(boost); +store_gov_pol_sys(boostpulse); +show_store_gov_pol_sys(boostpulse_duration); +show_store_gov_pol_sys(io_is_busy);

+gov_sys_pol_attr_rw(target_loads); +gov_sys_pol_attr_rw(above_hispeed_delay); +gov_sys_pol_attr_rw(hispeed_freq); +gov_sys_pol_attr_rw(go_hispeed_load); +gov_sys_pol_attr_rw(min_sample_time); +gov_sys_pol_attr_rw(timer_rate); +gov_sys_pol_attr_rw(timer_slack); +gov_sys_pol_attr_rw(boost); +gov_sys_pol_attr_rw(boostpulse_duration); +gov_sys_pol_attr_rw(io_is_busy);

+static struct global_attr boostpulse_gov_sys =

•   __ATTR(boostpulse, 0200, NULL, store_boostpulse_gov_sys);
  

+static struct freq_attr boostpulse_gov_pol =

•   __ATTR(boostpulse, 0200, NULL, store_boostpulse_gov_pol);
  

+/* One Governor instance for entire system */ +static struct attribute *interactive_attributes_gov_sys[] = {

•   &target_loads_gov_sys.attr,
  

•   &above_hispeed_delay_gov_sys.attr,
  

•   &hispeed_freq_gov_sys.attr,
  

•   &go_hispeed_load_gov_sys.attr,
  

•   &min_sample_time_gov_sys.attr,
  

•   &timer_rate_gov_sys.attr,
  

•   &timer_slack_gov_sys.attr,
  

•   &boost_gov_sys.attr,
  

•   &boostpulse_gov_sys.attr,
  

•   &boostpulse_duration_gov_sys.attr,
  

•   &io_is_busy_gov_sys.attr,
  

•   NULL,
  

+};

+static struct attribute_group interactive_attr_group_gov_sys = {

•   .attrs = interactive_attributes_gov_sys,
  

•   .name = "interactive",
  

+};

+/* Per policy governor instance */ +static struct attribute *interactive_attributes_gov_pol[] = {

•   &target_loads_gov_pol.attr,
  

•   &above_hispeed_delay_gov_pol.attr,
  

•   &hispeed_freq_gov_pol.attr,
  

•   &go_hispeed_load_gov_pol.attr,
  

•   &min_sample_time_gov_pol.attr,
  

•   &timer_rate_gov_pol.attr,
  

•   &timer_slack_gov_pol.attr,
  

•   &boost_gov_pol.attr,
  

•   &boostpulse_gov_pol.attr,
  

•   &boostpulse_duration_gov_pol.attr,
  

•   &io_is_busy_gov_pol.attr,
    NULL,
  

};

-static struct attribute_group interactive_attr_group = {

•   .attrs = interactive_attributes,
  

+static struct attribute_group interactive_attr_group_gov_pol = {

•   .attrs = interactive_attributes_gov_pol,
    .name = "interactive",
  

};

+static struct attribute_group *get_sysfs_attr(void) +{

•   if (have_governor_per_policy())
  

•           return &interactive_attr_group_gov_pol;
  

•   else
  

•           return &interactive_attr_group_gov_sys;
  

+}

static int cpufreq_interactive_idle_notifier(struct notifier_block *nb, unsigned long val, void *data) @@ -976,15 +1061,88 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy, unsigned int j; struct cpufreq_interactive_cpuinfo *pcpu; struct cpufreq_frequency_table *freq_table;

•   struct cpufreq_interactive_tunables *tunables;
  

•   if (have_governor_per_policy())
  

•           tunables = policy->governor_data;
  

•   else
  

•           tunables = common_tunables;
  

•   WARN_ON(!tunables && (event != CPUFREQ_GOV_POLICY_INIT));
  
    switch (event) {
  

•   case CPUFREQ_GOV_POLICY_INIT:
  

•           if (have_governor_per_policy()) {
  

•                   WARN_ON(tunables);
  

•           } else if (tunables) {
  

•                   tunables->usage_count++;
  

•                   policy->governor_data = tunables;
  

•                   return 0;
  

•           }
  

•           tunables = kzalloc(sizeof(*tunables), GFP_KERNEL);
  

•           if (!tunables) {
  

•                   pr_err("%s: POLICY_INIT: kzalloc failed\n", __func__);
  

•                   return -ENOMEM;
  

•           }
  

•           rc = sysfs_create_group(get_governor_parent_kobj(policy),
  

•                           get_sysfs_attr());
  

•           if (rc) {
  

•                   kfree(tunables);
  

•                   return rc;
  

•           }
  

•           tunables->usage_count = 1;
  

•           tunables->above_hispeed_delay = default_above_hispeed_delay;
  

•           tunables->nabove_hispeed_delay =
  

•                   ARRAY_SIZE(default_above_hispeed_delay);
  

•           tunables->go_hispeed_load = DEFAULT_GO_HISPEED_LOAD;
  

•           tunables->target_loads = default_target_loads;
  

•           tunables->ntarget_loads = ARRAY_SIZE(default_target_loads);
  

•           tunables->min_sample_time = DEFAULT_MIN_SAMPLE_TIME;
  

•           tunables->timer_rate = DEFAULT_TIMER_RATE;
  

•           tunables->boostpulse_duration_val = DEFAULT_MIN_SAMPLE_TIME;
  

•           tunables->timer_slack_val = DEFAULT_TIMER_SLACK;
  

•           spin_lock_init(&tunables->target_loads_lock);
  

•           spin_lock_init(&tunables->above_hispeed_delay_lock);
  

•           if (!policy->governor->initialized) {
  

•                   idle_notifier_register(&cpufreq_interactive_idle_nb);
  

•                   cpufreq_register_notifier(&cpufreq_notifier_block,
  

•                                   CPUFREQ_TRANSITION_NOTIFIER);
  

•           }
  

•           policy->governor_data = tunables;
  

•           if (!have_governor_per_policy())
  

•                   common_tunables = tunables;
  

•           break;
  

•   case CPUFREQ_GOV_POLICY_EXIT:
  

•           if (!--tunables->usage_count) {
  

•                   if (policy->governor->initialized == 1) {
  

•                           cpufreq_unregister_notifier(&cpufreq_notifier_block,
  

•                                           CPUFREQ_TRANSITION_NOTIFIER);
  

•                           idle_notifier_unregister(&cpufreq_interactive_idle_nb);
  

•                   }
  

•                   sysfs_remove_group(get_governor_parent_kobj(policy),
  

•                                   get_sysfs_attr());
  

•                   kfree(tunables);
  

•                   common_tunables = NULL;
  

•           }
  

•           policy->governor_data = NULL;
  

•           break;
  

•   case CPUFREQ_GOV_START:
            mutex_lock(&gov_lock);
  
  

•           freq_table =
  

•                   cpufreq_frequency_get_table(policy->cpu);
  

•           if (!hispeed_freq)
  

•                   hispeed_freq = policy->max;
  

•           freq_table = cpufreq_frequency_get_table(policy->cpu);
  

•           if (!tunables->hispeed_freq)
  

•                   tunables->hispeed_freq = policy->max;
  
            for_each_cpu(j, policy->cpus) {
                    unsigned long expires;
  

@@ -999,11 +1157,12 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy, pcpu->hispeed_validate_time = pcpu->floor_validate_time; down_write(&pcpu->enable_sem);

•                   expires = jiffies + usecs_to_jiffies(timer_rate);
  

•                   expires = jiffies +
  

•                           usecs_to_jiffies(tunables->timer_rate);
                    pcpu->cpu_timer.expires = expires;
                    add_timer_on(&pcpu->cpu_timer, j);
  

•                   if (timer_slack_val >= 0) {
  

•                           expires += usecs_to_jiffies(timer_slack_val);
  

•                   if (tunables->timer_slack_val >= 0) {
  

•                           expires += usecs_to_jiffies(tunables->timer_slack_val);
                            pcpu->cpu_slack_timer.expires = expires;
                            add_timer_on(&pcpu->cpu_slack_timer, j);
                    }
  

@@ -1011,25 +1170,6 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy, up_write(&pcpu->enable_sem); }

•           /*
  

•            * Do not register the idle hook and create sysfs
  

•            * entries if we have already done so.
  

•            */
  

•           if (++active_count > 1) {
  

•                   mutex_unlock(&gov_lock);
  

•                   return 0;
  

•           }
  

•           rc = sysfs_create_group(cpufreq_global_kobject,
  

•                           &interactive_attr_group);
  

•           if (rc) {
  

•                   mutex_unlock(&gov_lock);
  

•                   return rc;
  

•           }
  

•           idle_notifier_register(&cpufreq_interactive_idle_nb);
  

•           cpufreq_register_notifier(
  

•                   &cpufreq_notifier_block, CPUFREQ_TRANSITION_NOTIFIER);
            mutex_unlock(&gov_lock);
            break;
  
  

@@ -1044,18 +1184,7 @@ static int cpufreq_governor_interactive(struct cpufreq_policy *policy, up_write(&pcpu->enable_sem); }

•           if (--active_count > 0) {
  

•                   mutex_unlock(&gov_lock);
  

•                   return 0;
  

•           }
  

•           cpufreq_unregister_notifier(
  

•                   &cpufreq_notifier_block, CPUFREQ_TRANSITION_NOTIFIER);
  

•           idle_notifier_unregister(&cpufreq_interactive_idle_nb);
  

•           sysfs_remove_group(cpufreq_global_kobject,
  

•                           &interactive_attr_group);
            mutex_unlock(&gov_lock);
  

•           break;
  
    case CPUFREQ_GOV_LIMITS:
  

@@ -1102,9 +1231,7 @@ static int __init cpufreq_interactive_init(void) init_rwsem(&pcpu->enable_sem); }

•   spin_lock_init(&target_loads_lock);
    spin_lock_init(&speedchange_cpumask_lock);
  

•   spin_lock_init(&above_hispeed_delay_lock);
    mutex_init(&gov_lock);
    speedchange_task =
            kthread_create(cpufreq_interactive_speedchange_task, NULL,
  

-- 1.7.12.rc2.18.g61b472e