Hi all We found a warning from objtool: arch/x86/entry/entry_64.o: warning: objtool: .entry.text+0x1d1: unsupported intra-function call
and if we enable retpoline in config: arch/x86/entry/entry_64.o: warning: objtool: .entry.text+0x1c1: unsupported intra-function call arch/x86/entry/entry_64.o: warning: objtool: If this is a retpoline, please patch it in with alternatives and annotate it with ANNOTATE_NOSPEC_ALTERNATIVE.
I found this issue has been introduced since “x86/speculation: Change FILL_RETURN_BUFFER to work with objtool( commit 8afd1c7da2)”backported in v5.4.217. Comparing with the upstream version(commit 089dd8e53): There is no “ANNOTATE_INTRA_FUNCTION_CALL” in v5.4 for missing dependency patch. When the “ANNOTATE_NOSPEC_ALTERNATIVE” is removed, this issue just occurs.
I tried to backport “ANNOTATE_INTRA_FUNCTION_CALL”and its dependency patchs in v5.4, but I met the CFA miss match issue from objtool. So, please help check this issue in v5.4 LTS version.
On Mon, Feb 06, 2023 at 11:09:48AM +0800, Xinghui Li wrote:
Hi all We found a warning from objtool: arch/x86/entry/entry_64.o: warning: objtool: .entry.text+0x1d1: unsupported intra-function call
and if we enable retpoline in config: arch/x86/entry/entry_64.o: warning: objtool: .entry.text+0x1c1: unsupported intra-function call arch/x86/entry/entry_64.o: warning: objtool: If this is a retpoline, please patch it in with alternatives and annotate it with ANNOTATE_NOSPEC_ALTERNATIVE.
I found this issue has been introduced since “x86/speculation: Change FILL_RETURN_BUFFER to work with objtool( commit 8afd1c7da2)”backported in v5.4.217. Comparing with the upstream version(commit 089dd8e53): There is no “ANNOTATE_INTRA_FUNCTION_CALL” in v5.4 for missing dependency patch. When the “ANNOTATE_NOSPEC_ALTERNATIVE” is removed, this issue just occurs.
I tried to backport “ANNOTATE_INTRA_FUNCTION_CALL”and its dependency patchs in v5.4, but I met the CFA miss match issue from objtool. So, please help check this issue in v5.4 LTS version.
If you rely on the 5.4.y kernel tree, and you need this speculation fixes and feel this is a real problem, please provide some backported patches to resolve the problem.
It's been reported many times in the past, but no one seems to actually want to fix this bad enough to send in a patch :(
Usually people just move to a newer kernel, what is preventing you from doing that right now?
thanks,
greg k-h
Greg KH gregkh@linuxfoundation.org 于2023年2月6日周一 13:40写道：
If you rely on the 5.4.y kernel tree, and you need this speculation fixes and feel this is a real problem, please provide some backported patches to resolve the problem.
It's been reported many times in the past, but no one seems to actually want to fix this bad enough to send in a patch :(
Usually people just move to a newer kernel, what is preventing you from doing that right now?
thanks,
greg k-h
Thanks for your reply~ I am sorry about reporting the known Issue. I am trying to fix this issue right now. And I met the CFA issue, so I just want to discuss it with the community.
We keep staying in v5.4 because we developed the product base on v5.4 3 years ago. So it is unstable and difficult to update the kernel version.
I will try to find out a way to fix this issue.
Thanks a lot.
On Mon, Feb 06, 2023 at 05:22:54PM +0800, Xinghui Li wrote:
Greg KH gregkh@linuxfoundation.org 于2023年2月6日周一 13:40写道：
If you rely on the 5.4.y kernel tree, and you need this speculation fixes and feel this is a real problem, please provide some backported patches to resolve the problem.
It's been reported many times in the past, but no one seems to actually want to fix this bad enough to send in a patch :(
Usually people just move to a newer kernel, what is preventing you from doing that right now?
thanks,
greg k-h
Thanks for your reply~ I am sorry about reporting the known Issue. I am trying to fix this issue right now. And I met the CFA issue, so I just want to discuss it with the community.
Is this an actual real problem that you can detect with testing? Or is it just a warning message in the build?
We keep staying in v5.4 because we developed the product base on v5.4 3 years ago. So it is unstable and difficult to update the kernel version.
That is very odd, why is it hard to update to a new kernel? What happens if 5.4 stops being maintained tomorrow, what are your plans to move to a more modern kernel? Being stuck with an old kernel version with no plans to move does not seem like a wise business decision:(
I will try to find out a way to fix this issue.
Great, and we will be glad to review patches.
thanks,
greg k-h
Hi,
From: Greg KH gregkh@linuxfoundation.org Date: Mon, 6 Feb 2023 10:39:45 +0100
On Mon, Feb 06, 2023 at 05:22:54PM +0800, Xinghui Li wrote:
Greg KH gregkh@linuxfoundation.org 于2023年2月6日周一 13:40写道：
If you rely on the 5.4.y kernel tree, and you need this speculation fixes and feel this is a real problem, please provide some backported patches to resolve the problem.
It's been reported many times in the past, but no one seems to actually want to fix this bad enough to send in a patch :(
Usually people just move to a newer kernel, what is preventing you from doing that right now?
thanks,
greg k-h
Thanks for your reply~ I am sorry about reporting the known Issue. I am trying to fix this issue right now. And I met the CFA issue, so I just want to discuss it with the community.
Is this an actual real problem that you can detect with testing? Or is it just a warning message in the build?
I've just received a related report from a customer and I found the same commit was the first bad one while investigating. I've attached a simple repro below.
After 8afd1c7da2b0 ("x86/speculation: Change FILL_RETURN_BUFFER to work with objtool"), calling stack_trace_save_tsk_reliable() from the kernel module below fails with -EINVAL if CONFIG_UNWINDER_ORC=y and CONFIG_RETPOLINE=y.
I confirmed that this issue does not happen on 5.10 and 5.15 at least (and told the customer to use either).
Interestingly, after the commit, stack_trace_save_tsk_reliable() fails but seems to fill the buffer with the correct stack trace.
---8<--- #include <linux/module.h> #include <linux/kallsyms.h>
typedef int (*func_t)(struct task_struct *tsk, unsigned long *store, unsigned int size);
static __init int buggy_orc_init(void) { unsigned long store[32] = {0}; func_t func; int ret, i;
func = (func_t)kallsyms_lookup_name("stack_trace_save_tsk_reliable"); if (!func) return -ENOENT;
ret = func(current, store, ARRAY_SIZE(store));
for (i = 0; i < ARRAY_SIZE(store); i++) printk(KERN_ERR "%px: %pS\n", (void *)store[i], (void *)store[i]);
return ret > 0 ? 0 : ret; }
module_init(buggy_orc_init);
MODULE_LICENSE("GPL"); ---8<---
---8<--- # insmod buggy_orc.ko [ 8.576683] buggy_orc: loading out-of-tree module taints kernel. [ 8.578414] ffffffff810d1538: stack_trace_save_tsk_reliable+0x78/0xc0 [ 8.578414] ffffffffc000405c: buggy_orc_init+0x5c/0x1000 [buggy_orc] [ 8.579066] ffffffff81000b56: do_one_initcall+0x46/0x1f0 [ 8.579066] ffffffff810f005c: do_init_module+0x4c/0x240 [ 8.579066] ffffffff810f2cbf: __do_sys_finit_module+0xbf/0xe0 [ 8.580379] ffffffff81002198: do_syscall_64+0x48/0x110 [ 8.580379] ffffffff81c00094: entry_SYSCALL_64_after_hwframe+0x5c/0xc1 [ 8.581328] 0000000000000000: 0x0 [ 8.581328] 0000000000000000: 0x0 [ 8.581328] 0000000000000000: 0x0 [ 8.582145] 0000000000000000: 0x0 [ 8.582145] 0000000000000000: 0x0 [ 8.582145] 0000000000000000: 0x0 [ 8.582145] 0000000000000000: 0x0 [ 8.583229] 0000000000000000: 0x0 [ 8.583229] 0000000000000000: 0x0 [ 8.583229] 0000000000000000: 0x0 [ 8.584046] 0000000000000000: 0x0 [ 8.584046] 0000000000000000: 0x0 [ 8.584046] 0000000000000000: 0x0 [ 8.584046] 0000000000000000: 0x0 [ 8.585130] 0000000000000000: 0x0 [ 8.585130] 0000000000000000: 0x0 [ 8.585130] 0000000000000000: 0x0 [ 8.585130] 0000000000000000: 0x0 [ 8.586221] 0000000000000000: 0x0 [ 8.586221] 0000000000000000: 0x0 [ 8.586221] 0000000000000000: 0x0 [ 8.587038] 0000000000000000: 0x0 [ 8.587038] 0000000000000000: 0x0 [ 8.587038] 0000000000000000: 0x0 [ 8.587038] 0000000000000000: 0x0 insmod: ERROR: could not insert module buggy_orc.ko: Invalid parameters ---8<---
Thank you, Kuniyuki
We keep staying in v5.4 because we developed the product base on v5.4 3 years ago. So it is unstable and difficult to update the kernel version.
That is very odd, why is it hard to update to a new kernel? What happens if 5.4 stops being maintained tomorrow, what are your plans to move to a more modern kernel? Being stuck with an old kernel version with no plans to move does not seem like a wise business decision:(
I will try to find out a way to fix this issue.
Great, and we will be glad to review patches.
thanks,
greg k-h
Greg KH gregkh@linuxfoundation.org 于2023年2月6日周一 17:39写道：
Is this an actual real problem that you can detect with testing? Or is it just a warning message in the build?
So far, I have not met the actual error in running time. But according to Kuniyuki's report, it seems like this commit will cause an actual running time error. I will continue to analyze these error reports.
That is very odd, why is it hard to update to a new kernel? What happens if 5.4 stops being maintained tomorrow, what are your plans to move to a more modern kernel? Being stuck with an old kernel version with no plans to move does not seem like a wise business decision:(
The product base on v5.4 is the LTS version just like the stable-kernel in the upstream community.
Great, and we will be glad to review patches.
thanks,
greg k-h
Anyway, I will try to figure it out :-)
Thanks
On Tue, Feb 07, 2023 at 02:43:17PM +0800, Xinghui Li wrote:
Greg KH gregkh@linuxfoundation.org 于2023年2月6日周一 17:39写道：
Is this an actual real problem that you can detect with testing? Or is it just a warning message in the build?
So far, I have not met the actual error in running time. But according to Kuniyuki's report, it seems like this commit will cause an actual running time error.
I do not understand the actual runtime error, sorry. Stack traces don't matter at runtime, do they?
That is very odd, why is it hard to update to a new kernel? What happens if 5.4 stops being maintained tomorrow, what are your plans to move to a more modern kernel? Being stuck with an old kernel version with no plans to move does not seem like a wise business decision:(
The product base on v5.4 is the LTS version just like the stable-kernel in the upstream community.
That was not the question I asked :(
thanks,
greg k-h
Greg KH gregkh@linuxfoundation.org 于2023年2月7日周二 14:52写道：
I do not understand the actual runtime error, sorry. Stack traces don't matter at runtime, do they?
Sorry for the inaccurate description~ I mean it might not just a compiling warning.
That is very odd, why is it hard to update to a new kernel? What
Some features we developed based on v5.4's API. Update the kernel verison could cause the KABI issue.
happens if 5.4 stops being maintained tomorrow, what are your plans to
We will align with the LTS's lifecycle on our product lifecycle(actually shorter). If v5.4 do stop being maintained(I know it is not real), It looks like we can only maintain it all by ourselves :-(.
move to a more modern kernel? Being stuck with an old kernel version with no plans to move does not seem like a wise business decision:(
The product base on v5.4 is the LTS version just like the stable-kernel in the upstream community.
That was not the question I asked :(
Will the above reply answer your question? Besides, we do base the new kernel to develop the new version product such as v5.18 and 6.x.
Thanks~
On Tue, Feb 07, 2023 at 03:19:06PM +0800, Xinghui Li wrote:
Greg KH gregkh@linuxfoundation.org 于2023年2月7日周二 14:52写道：
I do not understand the actual runtime error, sorry. Stack traces don't matter at runtime, do they?
Sorry for the inaccurate description~ I mean it might not just a compiling warning.
That is very odd, why is it hard to update to a new kernel? What
Some features we developed based on v5.4's API. Update the kernel verison could cause the KABI issue.
What abi issue are you referring to? The user/kernel api? Or out-of-tree-code that is in your kernel tree?
If out-of-tree stuff, just forward port it, like you are going to have to do anyway. You have to always have a plan to move forward, otherwise you are guaranteed to have an insecure kernel someday.
Moving to a new kernel version should always be planned, and possible. If not, something is very wrong with your management process :(
happens if 5.4 stops being maintained tomorrow, what are your plans to
We will align with the LTS's lifecycle on our product lifecycle(actually shorter). If v5.4 do stop being maintained(I know it is not real), It looks like we can only maintain it all by ourselves :-(.
Again, why? What is so magical about this release?
thanks,
greg k-h
Greg KH gregkh@linuxfoundation.org 于2023年2月7日周二 15:30写道：
On Tue, Feb 07, 2023 at 03:19:06PM +0800, Xinghui Li wrote:
Greg KH gregkh@linuxfoundation.org 于2023年2月7日周二 14:52写道：
I do not understand the actual runtime error, sorry. Stack traces don't matter at runtime, do they?
Sorry for the inaccurate description~ I mean it might not just a compiling warning.
That is very odd, why is it hard to update to a new kernel? What
Some features we developed based on v5.4's API. Update the kernel verison could cause the KABI issue.
What abi issue are you referring to? The user/kernel api? Or out-of-tree-code that is in your kernel tree?
If out-of-tree stuff, just forward port it, like you are going to have to do anyway. You have to always have a plan to move forward, otherwise you are guaranteed to have an insecure kernel someday.
Moving to a new kernel version should always be planned, and possible. If not, something is very wrong with your management process :(
happens if 5.4 stops being maintained tomorrow, what are your plans to
We will align with the LTS's lifecycle on our product lifecycle(actually shorter). If v5.4 do stop being maintained(I know it is not real), It looks like we can only maintain it all by ourselves :-(.
Again, why? What is so magical about this release?
thanks,
greg k-h
Hi Greg, first of all, I have to say I totally agree with your opinion about keeping updating the product's kernel. But maybe because I'm not a native English speaker, it seems I didn't convey my meaning clearly or didn't get you well.
We do have plans to update the kernel of the product and we keep doing that.
However, for conservative reasons, parts of our customers are opposed to making significant upgrades to the products. Meanwhile, for commercial purposes, we are committed to maintaining the products we sell over a period of time. For the above reasons, we also maintain older versions of the product, so we will fix/report the issues in older kernels just like this time.
As for your question about why we chose v5.4, most likely because it was the latest LTS release at the time.
Thanks~
On Mon, Feb 13, 2023 at 08:20:03PM +0800, Xinghui Li wrote:
However, for conservative reasons, parts of our customers are opposed to making significant upgrades to the products.
Your customers are not concervative at all if they are consuming the RETBLEED and other Spectre-like fixes into your kernels. Those are major changes as you can see (affecting every single file you build), so the "do not change anything" argument does not make any sense here, right?
Meanwhile, for commercial purposes, we are committed to maintaining the products we sell over a period of time. For the above reasons, we also maintain older versions of the product, so we will fix/report the issues in older kernels just like this time.
That's great, but some help in fixing this reported issue would also be most appreciated, as you are the best one to test and verify the fix.
thanks,
greg k-h
linux-stable-mirror@lists.linaro.org
-
Greg KH -
Kuniyuki Iwashima -
Xinghui Li