Douglas Anderson, recently pointed out an interesting problem due to which udelay() was expiring earlier than it should.
While transitioning between frequencies few platforms may temporarily switch to a stable frequency, waiting for the main PLL to stabilize.
For example: When we transition between very low frequencies on exynos, like between 200MHz and 300MHz, we may temporarily switch to a PLL running at 800MHz. No CPUFREQ notification is sent for that. That means there's a period of time when we're running at 800MHz but loops_per_jiffy is calibrated at between 200MHz and 300MHz. And so udelay behaves badly.
To get this fixed in a generic way, lets introduce another set of callbacks get_intermediate() and target_intermediate(), only for drivers with target_index() and CPUFREQ_ASYNC_NOTIFICATION unset.
get_intermediate() should return a stable intermediate frequency platform wants to switch to, and target_intermediate() should set CPU to to that frequency, before jumping to the frequency corresponding to 'index'. Core will take care of sending notifications and driver doesn't have to handle them in target_intermediate() or target_index().
This patchset also update Tegra to use this new infrastructure and is already tested by Stephen.
V4->V5: - Moved setting old frequency to __target_index() from __target_intermediate() - Replaced retval with 0 during call to cpufreq_freq_transition_end() for restoring to restored freq. - Fix important issues with Tegra driver as reported by Stephen. - Dropped patch 1, already applied: "cpufreq: handle calls to ->target_index() in separate routine"
V3->V4: - Allow get_intermediate() to return zero when we don't need to switch to intermediate first - Get rid of 'goto' and create another routine for handling intermediate freqs - Allow target_index() to fail, its not a crime :) - Fix tegra driver to return zero from get_intermediate() for few situations (refer to patch 3/4) - Fix issues with tegra's patch, like s/rate/rate * 1000 - Overall there are more modifications that what Doug requested as I felt we need better support from core. - Looks much better now, thanks Doug :)
V2-V3: - Fix spelling error: s/Uset/Used - Update tegra with the changes Stephen suggested - Include a dependency patch sent separately earlier (3/4)
V1-V2: Almost changed completely, V1 was here: https://lkml.org/lkml/2014/5/15/40
Viresh Kumar (2): cpufreq: add support for intermediate (stable) frequencies cpufreq: Tegra: implement intermediate frequency callbacks
Documentation/cpu-freq/cpu-drivers.txt | 29 +++++++++- drivers/cpufreq/cpufreq.c | 67 ++++++++++++++++++++--- drivers/cpufreq/tegra-cpufreq.c | 97 ++++++++++++++++++++++------------ include/linux/cpufreq.h | 25 +++++++++ 4 files changed, 174 insertions(+), 44 deletions(-)
Douglas Anderson, recently pointed out an interesting problem due to which udelay() was expiring earlier than it should.
While transitioning between frequencies few platforms may temporarily switch to a stable frequency, waiting for the main PLL to stabilize.
For example: When we transition between very low frequencies on exynos, like between 200MHz and 300MHz, we may temporarily switch to a PLL running at 800MHz. No CPUFREQ notification is sent for that. That means there's a period of time when we're running at 800MHz but loops_per_jiffy is calibrated at between 200MHz and 300MHz. And so udelay behaves badly.
To get this fixed in a generic way, lets introduce another set of callbacks get_intermediate() and target_intermediate(), only for drivers with target_index() and CPUFREQ_ASYNC_NOTIFICATION unset.
get_intermediate() should return a stable intermediate frequency platform wants to switch to, and target_intermediate() should set CPU to to that frequency, before jumping to the frequency corresponding to 'index'. Core will take care of sending notifications and driver doesn't have to handle them in target_intermediate() or target_index().
NOTE: ->target_index() should restore to policy->restore_freq in case of failures as core would send notifications for that.
Tested-by: Stephen Warren swarren@nvidia.com Signed-off-by: Viresh Kumar viresh.kumar@linaro.org --- Documentation/cpu-freq/cpu-drivers.txt | 29 ++++++++++++++- drivers/cpufreq/cpufreq.c | 67 ++++++++++++++++++++++++++++++---- include/linux/cpufreq.h | 25 +++++++++++++ 3 files changed, 112 insertions(+), 9 deletions(-)
diff --git a/Documentation/cpu-freq/cpu-drivers.txt b/Documentation/cpu-freq/cpu-drivers.txt index b045fe5..14f4e63 100644 --- a/Documentation/cpu-freq/cpu-drivers.txt +++ b/Documentation/cpu-freq/cpu-drivers.txt @@ -26,6 +26,7 @@ Contents: 1.4 target/target_index or setpolicy? 1.5 target/target_index 1.6 setpolicy +1.7 get_intermediate and target_intermediate 2. Frequency Table Helpers
@@ -79,6 +80,10 @@ cpufreq_driver.attr - A pointer to a NULL-terminated list of "struct freq_attr" which allow to export values to sysfs.
+cpufreq_driver.get_intermediate +and target_intermediate Used to switch to stable frequency while + changing CPU frequency. +
1.2 Per-CPU Initialization -------------------------- @@ -151,7 +156,7 @@ Some cpufreq-capable processors switch the frequency between certain limits on their own. These shall use the ->setpolicy call
-1.4. target/target_index +1.5. target/target_index -------------
The target_index call has two arguments: struct cpufreq_policy *policy, @@ -160,6 +165,9 @@ and unsigned int index (into the exposed frequency table). The CPUfreq driver must set the new frequency when called here. The actual frequency must be determined by freq_table[index].frequency.
+It should always restore to earlier frequency (i.e. policy->restore_freq) in +case of errors, even if we switched to intermediate frequency earlier. + Deprecated: ---------- The target call has three arguments: struct cpufreq_policy *policy, @@ -179,7 +187,7 @@ Here again the frequency table helper might assist you - see section 2 for details.
-1.5 setpolicy +1.6 setpolicy ---------------
The setpolicy call only takes a struct cpufreq_policy *policy as @@ -190,6 +198,23 @@ setting when policy->policy is CPUFREQ_POLICY_PERFORMANCE, and a powersaving-oriented setting when CPUFREQ_POLICY_POWERSAVE. Also check the reference implementation in drivers/cpufreq/longrun.c
+1.7 get_intermediate and target_intermediate +-------------------------------------------- + +Only for drivers with target_index() and CPUFREQ_ASYNC_NOTIFICATION unset. + +get_intermediate should return a stable intermediate frequency platform wants to +switch to, and target_intermediate() should set CPU to to that frequency, before +jumping to the frequency corresponding to 'index'. Core will take care of +sending notifications and driver doesn't have to handle them in +target_intermediate() or target_index(). + +Drivers can return '0' from get_intermediate() in case they don't wish to switch +to intermediate frequency for some target frequency. In that case core will +directly call ->target_index(). + +NOTE: ->target_index() should restore to policy->restore_freq in case of +failures as core would send notifications for that.
2. Frequency Table Helpers diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index ae11dd5..aed2b0c 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -1816,20 +1816,55 @@ EXPORT_SYMBOL(cpufreq_unregister_notifier); * GOVERNORS * *********************************************************************/
+/* Must set freqs->new to intermediate frequency */ +static int __target_intermediate(struct cpufreq_policy *policy, + struct cpufreq_freqs *freqs, int index) +{ + int ret; + + freqs->new = cpufreq_driver->get_intermediate(policy, index); + + /* We don't need to switch to intermediate freq */ + if (!freqs->new) + return 0; + + pr_debug("%s: cpu: %d, switching to intermediate freq: oldfreq: %u, intermediate freq: %u\n", + __func__, policy->cpu, freqs->old, freqs->new); + + cpufreq_freq_transition_begin(policy, freqs); + ret = cpufreq_driver->target_intermediate(policy, index); + cpufreq_freq_transition_end(policy, freqs, ret); + + if (ret) + pr_err("%s: Failed to change to intermediate frequency: %d\n", + __func__, ret); + + return ret; +} + static int __target_index(struct cpufreq_policy *policy, struct cpufreq_frequency_table *freq_table, int index) { - struct cpufreq_freqs freqs; + struct cpufreq_freqs freqs = {.old = policy->cur, .flags = 0}; + unsigned int intermediate_freq = 0; int retval = -EINVAL; bool notify;
notify = !(cpufreq_driver->flags & CPUFREQ_ASYNC_NOTIFICATION); - if (notify) { - freqs.old = policy->cur; - freqs.new = freq_table[index].frequency; - freqs.flags = 0; + /* Handle switching to intermediate frequency */ + if (cpufreq_driver->get_intermediate) { + retval = __target_intermediate(policy, &freqs, index); + if (retval) + return retval; + + intermediate_freq = freqs.new; + /* Set old freq to intermediate */ + if (intermediate_freq) + freqs.old = freqs.new; + }
+ freqs.new = freq_table[index].frequency; pr_debug("%s: cpu: %d, oldfreq: %u, new freq: %u\n", __func__, policy->cpu, freqs.old, freqs.new);
@@ -1841,9 +1876,23 @@ static int __target_index(struct cpufreq_policy *policy, pr_err("%s: Failed to change cpu frequency: %d\n", __func__, retval);
- if (notify) + if (notify) { cpufreq_freq_transition_end(policy, &freqs, retval);
+ /* + * Failed after setting to intermediate freq? Driver should have + * reverted back to initial frequency and so should we. Check + * here for intermediate_freq instead of get_intermediate, in + * case we have't switched to intermediate freq at all. + */ + if (unlikely(retval && intermediate_freq)) { + freqs.old = intermediate_freq; + freqs.new = policy->restore_freq; + cpufreq_freq_transition_begin(policy, &freqs); + cpufreq_freq_transition_end(policy, &freqs, 0); + } + } + return retval; }
@@ -1875,6 +1924,9 @@ int __cpufreq_driver_target(struct cpufreq_policy *policy, if (target_freq == policy->cur) return 0;
+ /* Save last value to restore later on errors */ + policy->restore_freq = policy->cur; + if (cpufreq_driver->target) retval = cpufreq_driver->target(policy, target_freq, relation); else if (cpufreq_driver->target_index) { @@ -2361,7 +2413,8 @@ int cpufreq_register_driver(struct cpufreq_driver *driver_data) !(driver_data->setpolicy || driver_data->target_index || driver_data->target) || (driver_data->setpolicy && (driver_data->target_index || - driver_data->target))) + driver_data->target)) || + (!!driver_data->get_intermediate != !!driver_data->target_intermediate)) return -EINVAL;
pr_debug("trying to register driver %s\n", driver_data->name); diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h index 3f45889..ec4112d 100644 --- a/include/linux/cpufreq.h +++ b/include/linux/cpufreq.h @@ -75,6 +75,7 @@ struct cpufreq_policy { unsigned int max; /* in kHz */ unsigned int cur; /* in kHz, only needed if cpufreq * governors are used */ + unsigned int restore_freq; /* = policy->cur before transition */ unsigned int suspend_freq; /* freq to set during suspend */
unsigned int policy; /* see above */ @@ -221,11 +222,35 @@ struct cpufreq_driver {
/* define one out of two */ int (*setpolicy) (struct cpufreq_policy *policy); + + /* + * On failure, should always restore frequency to policy->restore_freq + * (i.e. old freq). + */ int (*target) (struct cpufreq_policy *policy, /* Deprecated */ unsigned int target_freq, unsigned int relation); int (*target_index) (struct cpufreq_policy *policy, unsigned int index); + /* + * Only for drivers with target_index() and CPUFREQ_ASYNC_NOTIFICATION + * unset. + * + * get_intermediate should return a stable intermediate frequency + * platform wants to switch to and target_intermediate() should set CPU + * to to that frequency, before jumping to the frequency corresponding + * to 'index'. Core will take care of sending notifications and driver + * doesn't have to handle them in target_intermediate() or + * target_index(). + * + * Drivers can return '0' from get_intermediate() in case they don't + * wish to switch to intermediate frequency for some target frequency. + * In that case core will directly call ->target_index(). + */ + unsigned int (*get_intermediate)(struct cpufreq_policy *policy, + unsigned int index); + int (*target_intermediate)(struct cpufreq_policy *policy, + unsigned int index);
/* should be defined, if possible */ unsigned int (*get) (unsigned int cpu);
Viresh,
On Mon, Jun 2, 2014 at 10:19 AM, Viresh Kumar viresh.kumar@linaro.org wrote:
Douglas Anderson, recently pointed out an interesting problem due to which udelay() was expiring earlier than it should.
While transitioning between frequencies few platforms may temporarily switch to a stable frequency, waiting for the main PLL to stabilize.
For example: When we transition between very low frequencies on exynos, like between 200MHz and 300MHz, we may temporarily switch to a PLL running at 800MHz. No CPUFREQ notification is sent for that. That means there's a period of time when we're running at 800MHz but loops_per_jiffy is calibrated at between 200MHz and 300MHz. And so udelay behaves badly.
To get this fixed in a generic way, lets introduce another set of callbacks get_intermediate() and target_intermediate(), only for drivers with target_index() and CPUFREQ_ASYNC_NOTIFICATION unset.
get_intermediate() should return a stable intermediate frequency platform wants to switch to, and target_intermediate() should set CPU to to that frequency, before jumping to the frequency corresponding to 'index'. Core will take care of sending notifications and driver doesn't have to handle them in target_intermediate() or target_index().
NOTE: ->target_index() should restore to policy->restore_freq in case of failures as core would send notifications for that.
Tested-by: Stephen Warren swarren@nvidia.com Signed-off-by: Viresh Kumar viresh.kumar@linaro.org
Documentation/cpu-freq/cpu-drivers.txt | 29 ++++++++++++++- drivers/cpufreq/cpufreq.c | 67 ++++++++++++++++++++++++++++++---- include/linux/cpufreq.h | 25 +++++++++++++ 3 files changed, 112 insertions(+), 9 deletions(-)
LGTM. Thanks for doing this! :)
Reviewed-by: Doug Anderson dianders@chromium.org
On 6 June 2014 01:21, Doug Anderson dianders@chromium.org wrote:
LGTM. Thanks for doing this! :)
Thanks and yours welcome :)
Reviewed-by: Doug Anderson dianders@chromium.org
Thanks a lot. Its pushed by Rafael now :)
Tegra had always been switching to intermediate frequency (pll_p_clk) since ever. CPUFreq core has better support for handling notifications for these frequencies and so we can adapt Tegra's driver to it.
Also do a WARN() if clk_set_parent() fails while moving back to pll_x as we should have atleast restored to earlier frequency on error.
Tested-by: Stephen Warren swarren@nvidia.com Signed-off-by: Viresh Kumar viresh.kumar@linaro.org --- drivers/cpufreq/tegra-cpufreq.c | 97 ++++++++++++++++++++++++++--------------- 1 file changed, 62 insertions(+), 35 deletions(-)
diff --git a/drivers/cpufreq/tegra-cpufreq.c b/drivers/cpufreq/tegra-cpufreq.c index 6e774c6..a5fbc0a 100644 --- a/drivers/cpufreq/tegra-cpufreq.c +++ b/drivers/cpufreq/tegra-cpufreq.c @@ -45,46 +45,51 @@ static struct clk *cpu_clk; static struct clk *pll_x_clk; static struct clk *pll_p_clk; static struct clk *emc_clk; +static bool pll_x_prepared;
-static int tegra_cpu_clk_set_rate(unsigned long rate) +static unsigned int tegra_get_intermediate(struct cpufreq_policy *policy, + unsigned int index) +{ + unsigned int ifreq = clk_get_rate(pll_p_clk) / 1000; + + /* + * Don't switch to intermediate freq if: + * - we are already at it, i.e. policy->cur == ifreq + * - index corresponds to ifreq + */ + if ((freq_table[index].frequency == ifreq) || (policy->cur == ifreq)) + return 0; + + return ifreq; +} + +static int tegra_target_intermediate(struct cpufreq_policy *policy, + unsigned int index) { int ret;
/* * Take an extra reference to the main pll so it doesn't turn - * off when we move the cpu off of it + * off when we move the cpu off of it as enabling it again while we + * switch to it from tegra_target() would take additional time. Though + * when target-freq is intermediate freq, we don't need to take this + * reference. */ clk_prepare_enable(pll_x_clk);
ret = clk_set_parent(cpu_clk, pll_p_clk); - if (ret) { - pr_err("Failed to switch cpu to clock pll_p\n"); - goto out; - } - - if (rate == clk_get_rate(pll_p_clk)) - goto out; - - ret = clk_set_rate(pll_x_clk, rate); - if (ret) { - pr_err("Failed to change pll_x to %lu\n", rate); - goto out; - } - - ret = clk_set_parent(cpu_clk, pll_x_clk); - if (ret) { - pr_err("Failed to switch cpu to clock pll_x\n"); - goto out; - } + if (ret) + clk_disable_unprepare(pll_x_clk); + else + pll_x_prepared = true;
-out: - clk_disable_unprepare(pll_x_clk); return ret; }
static int tegra_target(struct cpufreq_policy *policy, unsigned int index) { unsigned long rate = freq_table[index].frequency; + unsigned int ifreq = clk_get_rate(pll_p_clk) / 1000; int ret = 0;
/* @@ -98,10 +103,30 @@ static int tegra_target(struct cpufreq_policy *policy, unsigned int index) else clk_set_rate(emc_clk, 100000000); /* emc 50Mhz */
- ret = tegra_cpu_clk_set_rate(rate * 1000); + /* + * target freq == pll_p, don't need to take extra reference to pll_x_clk + * as it isn't used anymore. + */ + if (rate == ifreq) + return clk_set_parent(cpu_clk, pll_p_clk); + + ret = clk_set_rate(pll_x_clk, rate * 1000); + /* Restore to earlier frequency on error, i.e. pll_x */ if (ret) - pr_err("cpu-tegra: Failed to set cpu frequency to %lu kHz\n", - rate); + pr_err("Failed to change pll_x to %lu\n", rate); + + ret = clk_set_parent(cpu_clk, pll_x_clk); + /* This shouldn't fail while changing or restoring */ + WARN_ON(ret); + + /* + * Drop count to pll_x clock only if we switched to intermediate freq + * earlier while transitioning to a target frequency. + */ + if (pll_x_prepared) { + clk_disable_unprepare(pll_x_clk); + pll_x_prepared = false; + }
return ret; } @@ -137,16 +162,18 @@ static int tegra_cpu_exit(struct cpufreq_policy *policy) }
static struct cpufreq_driver tegra_cpufreq_driver = { - .flags = CPUFREQ_NEED_INITIAL_FREQ_CHECK, - .verify = cpufreq_generic_frequency_table_verify, - .target_index = tegra_target, - .get = cpufreq_generic_get, - .init = tegra_cpu_init, - .exit = tegra_cpu_exit, - .name = "tegra", - .attr = cpufreq_generic_attr, + .flags = CPUFREQ_NEED_INITIAL_FREQ_CHECK, + .verify = cpufreq_generic_frequency_table_verify, + .get_intermediate = tegra_get_intermediate, + .target_intermediate = tegra_target_intermediate, + .target_index = tegra_target, + .get = cpufreq_generic_get, + .init = tegra_cpu_init, + .exit = tegra_cpu_exit, + .name = "tegra", + .attr = cpufreq_generic_attr, #ifdef CONFIG_PM - .suspend = cpufreq_generic_suspend, + .suspend = cpufreq_generic_suspend, #endif };
Viresh,
On Mon, Jun 2, 2014 at 10:19 AM, Viresh Kumar viresh.kumar@linaro.org wrote:
Tegra had always been switching to intermediate frequency (pll_p_clk) since ever. CPUFreq core has better support for handling notifications for these frequencies and so we can adapt Tegra's driver to it.
Also do a WARN() if clk_set_parent() fails while moving back to pll_x as we should have atleast restored to earlier frequency on error.
Tested-by: Stephen Warren swarren@nvidia.com Signed-off-by: Viresh Kumar viresh.kumar@linaro.org
drivers/cpufreq/tegra-cpufreq.c | 97 ++++++++++++++++++++++++++--------------- 1 file changed, 62 insertions(+), 35 deletions(-)
diff --git a/drivers/cpufreq/tegra-cpufreq.c b/drivers/cpufreq/tegra-cpufreq.c index 6e774c6..a5fbc0a 100644 --- a/drivers/cpufreq/tegra-cpufreq.c +++ b/drivers/cpufreq/tegra-cpufreq.c @@ -45,46 +45,51 @@ static struct clk *cpu_clk; static struct clk *pll_x_clk; static struct clk *pll_p_clk; static struct clk *emc_clk; +static bool pll_x_prepared;
-static int tegra_cpu_clk_set_rate(unsigned long rate) +static unsigned int tegra_get_intermediate(struct cpufreq_policy *policy,
unsigned int index)+{
unsigned int ifreq = clk_get_rate(pll_p_clk) / 1000;/** Don't switch to intermediate freq if:* - we are already at it, i.e. policy->cur == ifreq* - index corresponds to ifreq*/if ((freq_table[index].frequency == ifreq) || (policy->cur == ifreq))return 0;return ifreq;+}
+static int tegra_target_intermediate(struct cpufreq_policy *policy,
unsigned int index){ int ret;
/* * Take an extra reference to the main pll so it doesn't turn
* off when we move the cpu off of it
* off when we move the cpu off of it as enabling it again while we* switch to it from tegra_target() would take additional time. Though* when target-freq is intermediate freq, we don't need to take this* reference.
The "Though when target-freq is intermediate freq, we don't need to take this reference." makes me think that this function is actually called when target-freq is intermediate freq. I don't think it is, right?
I don't think that's a huge deal, though and code wise this looks good to me.
Reviewed-by: Doug Anderson dianders@chromium.org
On 6 June 2014 01:21, Doug Anderson dianders@chromium.org wrote:
The "Though when target-freq is intermediate freq, we don't need to take this reference." makes me think that this function is actually called when target-freq is intermediate freq. I don't think it is, right?
Yes, it isn't called for that combination.
I don't think that's a huge deal, though and code wise this looks good to me.
I don't know either, let me send a patch for that and see what people think :)
Reviewed-by: Doug Anderson dianders@chromium.org
Thanks.
linaro-kernel@lists.linaro.org
-
Doug Anderson -
Viresh Kumar