stable/linux-4.14.y boot: 108 boots: 0 failed, 107 passed with 1 conflict (v4.14.11)
Full Boot Summary: https://kernelci.org/boot/all/job/stable/branch/linux-4.14.y/kernel/v4.14.11... Full Build Summary: https://kernelci.org/build/stable/branch/linux-4.14.y/kernel/v4.14.11/
Tree: stable Branch: linux-4.14.y Git Describe: v4.14.11 Git Commit: 0d59679df5b53755c00ea0292df696f97bfc950d Git URL: http://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git Tested: 59 unique boards, 23 SoC families, 16 builds out of 185
Boot Regressions Detected:
x86:
x86_64_defconfig: qemu: lab-mhart: new failure (last pass: v4.14.10)
Conflicting Boot Failure Detected: (These likely are not failures as other labs are reporting PASS. Needs review.)
x86:
x86_64_defconfig: qemu: lab-mhart: FAIL lab-collabora: PASS
--- For more info write to info@kernelci.org
On 03/01/18 03:11, kernelci.org bot wrote:
stable/linux-4.14.y boot: 108 boots: 0 failed, 107 passed with 1 conflict (v4.14.11)
Full Boot Summary: https://kernelci.org/boot/all/job/stable/branch/linux-4.14.y/kernel/v4.14.11... Full Build Summary: https://kernelci.org/build/stable/branch/linux-4.14.y/kernel/v4.14.11/
Tree: stable Branch: linux-4.14.y Git Describe: v4.14.11 Git Commit: 0d59679df5b53755c00ea0292df696f97bfc950d Git URL: http://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git Tested: 59 unique boards, 23 SoC families, 16 builds out of 185
Boot Regressions Detected:
x86:
x86_64_defconfig: qemu: lab-mhart: new failure (last pass: v4.14.10)
Conflicting Boot Failure Detected: (These likely are not failures as other labs are reporting PASS. Needs review.)
x86:
x86_64_defconfig: qemu: lab-mhart: FAIL lab-collabora: PASS
Well, it turns out this is not exactly a conflict as there's a subtle difference between the qemu devices in lab-mhart and in lab-collabora. The ones in lab-collabora are configured to use KVM, and it looks like the ones in lab-mhart aren't.
So this job with KVM enabled passes in lab-collabora:
https://lava.collabora.co.uk/scheduler/job/1032358
but it fails if I tell LAVA (qemu) to disable KVM:
https://lava.collabora.co.uk/scheduler/job/1032359
with the same panic as in lab-mhart. It seems like it's failing to return from an interrupt:
http://lava.streamtester.net/scheduler/job/87308
[ 2.678828] ? native_iret+0x7/0x7 [ 2.679208] WARNING: can't dereference iret registers at 00000000ffc66068 for ip page_fault+0x11/0x60
This triggered an automated bisection on kernelci.org, please see the results below.
I may run another bisection with this config enabled earlier in the history to track down the actual change in the code that introduced the issue, let me know if it's worth doing.
Hope this helps!
Best wishes, Guillaume
--8<-----------------------------------------------------------------------8<--
Bisection result for stable/linux-4.14.y (v4.14.11) on qemu
Good known revision:
b8ce823 Linux 4.14.10
Bad known revision:
0d59679 Linux 4.14.11
Extra parameters:
Tree: stable URL: http://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git Branch: linux-4.14.y Target: qemu Lab: lab-mhart Defconfig: x86_64_defconfig Plan: boot
Breaking commit found:
------------------------------------------------------------------------------- commit 3dfd9fd8d897214b1a880c7fd8ed36b88faa1c02 Author: Dave Hansen dave.hansen@linux.intel.com Date: Mon Dec 4 15:08:03 2017 +0100
x86/mm/pti: Add Kconfig
commit 385ce0ea4c078517fa51c261882c4e72fba53005 upstream.
Finally allow CONFIG_PAGE_TABLE_ISOLATION to be enabled.
PARAVIRT generally requires that the kernel not manage its own page tables. It also means that the hypervisor and kernel must agree wholeheartedly about what format the page tables are in and what they contain. PAGE_TABLE_ISOLATION, unfortunately, changes the rules and they can not be used together.
I've seen conflicting feedback from maintainers lately about whether they want the Kconfig magic to go first or last in a patch series. It's going last here because the partially-applied series leads to kernels that can not boot in a bunch of cases. I did a run through the entire series with CONFIG_PAGE_TABLE_ISOLATION=y to look for build errors, though.
[ tglx: Removed SMP and !PARAVIRT dependencies as they not longer exist ]
Signed-off-by: Dave Hansen dave.hansen@linux.intel.com Signed-off-by: Thomas Gleixner tglx@linutronix.de Cc: Andy Lutomirski luto@kernel.org Cc: Boris Ostrovsky boris.ostrovsky@oracle.com Cc: Borislav Petkov bp@alien8.de Cc: Brian Gerst brgerst@gmail.com Cc: David Laight David.Laight@aculab.com Cc: Denys Vlasenko dvlasenk@redhat.com Cc: Eduardo Valentin eduval@amazon.com Cc: Greg KH gregkh@linuxfoundation.org Cc: H. Peter Anvin hpa@zytor.com Cc: Josh Poimboeuf jpoimboe@redhat.com Cc: Juergen Gross jgross@suse.com Cc: Linus Torvalds torvalds@linux-foundation.org Cc: Peter Zijlstra peterz@infradead.org Cc: Will Deacon will.deacon@arm.com Cc: aliguori@amazon.com Cc: daniel.gruss@iaik.tugraz.at Cc: hughd@google.com Cc: keescook@google.com Cc: linux-mm@kvack.org Signed-off-by: Ingo Molnar mingo@kernel.org Signed-off-by: Greg Kroah-Hartman gregkh@linuxfoundation.org
diff --git a/security/Kconfig b/security/Kconfig index e8e4494..6614b93 100644 --- a/security/Kconfig +++ b/security/Kconfig @@ -54,6 +54,17 @@ config SECURITY_NETWORK implement socket and networking access controls. If you are unsure how to answer this question, answer N.
+config PAGE_TABLE_ISOLATION + bool "Remove the kernel mapping in user mode" + depends on X86_64 && !UML + default y + help + This feature reduces the number of hardware side channels by + ensuring that the majority of kernel addresses are not mapped + into userspace. + + See Documentation/x86/pagetable-isolation.txt for more details. + config SECURITY_INFINIBAND bool "Infiniband Security Hooks" depends on SECURITY && INFINIBAND -------------------------------------------------------------------------------
Git bisection log:
------------------------------------------------------------------------------- git bisect start # good: [b8ce8232fcc37fe7a97db79ea0a5f32098c25e72] Linux 4.14.10 git bisect good b8ce8232fcc37fe7a97db79ea0a5f32098c25e72 # bad: [0d59679df5b53755c00ea0292df696f97bfc950d] Linux 4.14.11 git bisect bad 0d59679df5b53755c00ea0292df696f97bfc950d # bad: [621b5ae0f9f4f9ef91bf441afc086ecf5e752d51] s390/qeth: apply takeover changes when mode is toggled git bisect bad 621b5ae0f9f4f9ef91bf441afc086ecf5e752d51 # bad: [66f833dbed02d39c44440b6b35ac088655c32edb] ring-buffer: Mask out the info bits when returning buffer page length git bisect bad 66f833dbed02d39c44440b6b35ac088655c32edb # good: [d230c1917f57c3beee2e0204a4c8c58999758b95] x86/mm/pti: Map ESPFIX into user space git bisect good d230c1917f57c3beee2e0204a4c8c58999758b95 # good: [36a72ab52c8d969a7a302082f52731c1be0e9ada] x86/mm: Optimize RESTORE_CR3 git bisect good 36a72ab52c8d969a7a302082f52731c1be0e9ada # bad: [3dfd9fd8d897214b1a880c7fd8ed36b88faa1c02] x86/mm/pti: Add Kconfig git bisect bad 3dfd9fd8d897214b1a880c7fd8ed36b88faa1c02 # good: [ef4b38472d6b1bf587554dfc7d5ab7abc835c1a5] x86/mm: Clarify the whole ASID/kernel PCID/user PCID naming git bisect good ef4b38472d6b1bf587554dfc7d5ab7abc835c1a5 # good: [33d9d7836f0fa02777667d72bc815c12fbe61cac] x86/dumpstack: Indicate in Oops whether PTI is configured and enabled git bisect good 33d9d7836f0fa02777667d72bc815c12fbe61cac # first bad commit: [3dfd9fd8d897214b1a880c7fd8ed36b88faa1c02] x86/mm/pti: Add Kconfig -------------------------------------------------------------------------------
For more info write to info@kernelci.org
Kernel-build-reports mailing list Kernel-build-reports@lists.linaro.org https://lists.linaro.org/mailman/listinfo/kernel-build-reports
On Wed, 3 Jan 2018, Guillaume Tucker wrote:
On 03/01/18 03:11, kernelci.org bot wrote:
stable/linux-4.14.y boot: 108 boots: 0 failed, 107 passed with 1 conflict (v4.14.11)
Full Boot Summary: https://kernelci.org/boot/all/job/stable/branch/linux-4.14.y/kernel/v4.14.11... Full Build Summary: https://kernelci.org/build/stable/branch/linux-4.14.y/kernel/v4.14.11/
Tree: stable Branch: linux-4.14.y Git Describe: v4.14.11 Git Commit: 0d59679df5b53755c00ea0292df696f97bfc950d Git URL: http://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git Tested: 59 unique boards, 23 SoC families, 16 builds out of 185
Boot Regressions Detected:
x86:
x86_64_defconfig: qemu: lab-mhart: new failure (last pass: v4.14.10)
Conflicting Boot Failure Detected: (These likely are not failures as other labs are reporting PASS. Needs review.)
x86:
x86_64_defconfig: qemu: lab-mhart: FAIL lab-collabora: PASS
Well, it turns out this is not exactly a conflict as there's a subtle difference between the qemu devices in lab-mhart and in lab-collabora. The ones in lab-collabora are configured to use KVM, and it looks like the ones in lab-mhart aren't.
So this job with KVM enabled passes in lab-collabora:
https://lava.collabora.co.uk/scheduler/job/1032358
but it fails if I tell LAVA (qemu) to disable KVM:
https://lava.collabora.co.uk/scheduler/job/1032359
with the same panic as in lab-mhart. It seems like it's failing to return from an interrupt:
http://lava.streamtester.net/scheduler/job/87308
[ 2.678828] ? native_iret+0x7/0x7 [ 2.679208] WARNING: can't dereference iret registers at 00000000ffc66068 for ip page_fault+0x11/0x60
This triggered an automated bisection on kernelci.org, please see the results below.
I may run another bisection with this config enabled earlier in the history to track down the actual change in the code that introduced the issue, let me know if it's worth doing.
No, because before that commit not all pieces are in place.
Can you please try the failing kernel with pti=off on the command line?
I'll start a test instance here as well.
Thanks,
tglx
On 03/01/18 09:48, Thomas Gleixner wrote:
On Wed, 3 Jan 2018, Guillaume Tucker wrote:
On 03/01/18 03:11, kernelci.org bot wrote:
stable/linux-4.14.y boot: 108 boots: 0 failed, 107 passed with 1 conflict (v4.14.11)
Full Boot Summary: https://kernelci.org/boot/all/job/stable/branch/linux-4.14.y/kernel/v4.14.11... Full Build Summary: https://kernelci.org/build/stable/branch/linux-4.14.y/kernel/v4.14.11/
Tree: stable Branch: linux-4.14.y Git Describe: v4.14.11 Git Commit: 0d59679df5b53755c00ea0292df696f97bfc950d Git URL: http://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git Tested: 59 unique boards, 23 SoC families, 16 builds out of 185
Boot Regressions Detected:
x86:
x86_64_defconfig: qemu: lab-mhart: new failure (last pass: v4.14.10)
Conflicting Boot Failure Detected: (These likely are not failures as other labs are reporting PASS. Needs review.)
x86:
x86_64_defconfig: qemu: lab-mhart: FAIL lab-collabora: PASS
Well, it turns out this is not exactly a conflict as there's a subtle difference between the qemu devices in lab-mhart and in lab-collabora. The ones in lab-collabora are configured to use KVM, and it looks like the ones in lab-mhart aren't.
So this job with KVM enabled passes in lab-collabora:
https://lava.collabora.co.uk/scheduler/job/1032358
but it fails if I tell LAVA (qemu) to disable KVM:
https://lava.collabora.co.uk/scheduler/job/1032359
with the same panic as in lab-mhart. It seems like it's failing to return from an interrupt:
http://lava.streamtester.net/scheduler/job/87308
[ 2.678828] ? native_iret+0x7/0x7 [ 2.679208] WARNING: can't dereference iret registers at 00000000ffc66068 for ip page_fault+0x11/0x60
This triggered an automated bisection on kernelci.org, please see the results below.
I may run another bisection with this config enabled earlier in the history to track down the actual change in the code that introduced the issue, let me know if it's worth doing.
No, because before that commit not all pieces are in place.
Can you please try the failing kernel with pti=off on the command line?
It does boot with pti=off (and KVM disabled):
https://lava.collabora.co.uk/scheduler/job/1032387
I'll start a test instance here as well.
Thanks,
tglx
On Wed, 3 Jan 2018, Guillaume Tucker wrote:
On 03/01/18 09:48, Thomas Gleixner wrote:
Well, it turns out this is not exactly a conflict as there's a subtle difference between the qemu devices in lab-mhart and in lab-collabora. The ones in lab-collabora are configured to use KVM, and it looks like the ones in lab-mhart aren't.
So this job with KVM enabled passes in lab-collabora:
https://lava.collabora.co.uk/scheduler/job/1032358
but it fails if I tell LAVA (qemu) to disable KVM:
https://lava.collabora.co.uk/scheduler/job/1032359
with the same panic as in lab-mhart. It seems like it's failing to return from an interrupt:
http://lava.streamtester.net/scheduler/job/87308
[ 2.678828] ? native_iret+0x7/0x7 [ 2.679208] WARNING: can't dereference iret registers at 00000000ffc66068 for ip page_fault+0x11/0x60
This triggered an automated bisection on kernelci.org, please see the results below.
I may run another bisection with this config enabled earlier in the history to track down the actual change in the code that introduced the issue, let me know if it's worth doing.
No, because before that commit not all pieces are in place.
Can you please try the failing kernel with pti=off on the command line?
It does boot with pti=off (and KVM disabled):
So it's a qemu issue. Added qemu folks on Cc.
I'll start a test instance here as well.
Thanks,
tglx
On 03/01/2018 11:36, Thomas Gleixner wrote:
On Wed, 3 Jan 2018, Guillaume Tucker wrote:
On 03/01/18 09:48, Thomas Gleixner wrote:
Well, it turns out this is not exactly a conflict as there's a subtle difference between the qemu devices in lab-mhart and in lab-collabora. The ones in lab-collabora are configured to use KVM, and it looks like the ones in lab-mhart aren't.
So this job with KVM enabled passes in lab-collabora:
https://lava.collabora.co.uk/scheduler/job/1032358
but it fails if I tell LAVA (qemu) to disable KVM:
https://lava.collabora.co.uk/scheduler/job/1032359
with the same panic as in lab-mhart. It seems like it's failing to return from an interrupt:
http://lava.streamtester.net/scheduler/job/87308
[ 2.678828] ? native_iret+0x7/0x7 [ 2.679208] WARNING: can't dereference iret registers at 00000000ffc66068 for ip page_fault+0x11/0x60
This triggered an automated bisection on kernelci.org, please see the results below.
I may run another bisection with this config enabled earlier in the history to track down the actual change in the code that introduced the issue, let me know if it's worth doing.
No, because before that commit not all pieces are in place.
Can you please try the failing kernel with pti=off on the command line?
It does boot with pti=off (and KVM disabled):
So it's a qemu issue. Added qemu folks on Cc.
Reproduced, thanks. I will look into it.
Paolo
On Wed, 3 Jan 2018, Paolo Bonzini wrote:
On 03/01/2018 11:36, Thomas Gleixner wrote:
On Wed, 3 Jan 2018, Guillaume Tucker wrote:
On 03/01/18 09:48, Thomas Gleixner wrote:
Well, it turns out this is not exactly a conflict as there's a subtle difference between the qemu devices in lab-mhart and in lab-collabora. The ones in lab-collabora are configured to use KVM, and it looks like the ones in lab-mhart aren't.
So this job with KVM enabled passes in lab-collabora:
https://lava.collabora.co.uk/scheduler/job/1032358
but it fails if I tell LAVA (qemu) to disable KVM:
https://lava.collabora.co.uk/scheduler/job/1032359
with the same panic as in lab-mhart. It seems like it's failing to return from an interrupt:
http://lava.streamtester.net/scheduler/job/87308
[ 2.678828] ? native_iret+0x7/0x7 [ 2.679208] WARNING: can't dereference iret registers at 00000000ffc66068 for ip page_fault+0x11/0x60
This triggered an automated bisection on kernelci.org, please see the results below.
I may run another bisection with this config enabled earlier in the history to track down the actual change in the code that introduced the issue, let me know if it's worth doing.
No, because before that commit not all pieces are in place.
Can you please try the failing kernel with pti=off on the command line?
It does boot with pti=off (and KVM disabled):
So it's a qemu issue. Added qemu folks on Cc.
Reproduced, thanks. I will look into it.
I just noticed that the qemu instance emulates an AMD CPU.
We discovered an AMD related issue which fits in the problem you are seing today.
Can you try the patch below please?
Thanks,
tglx
8<------------------
--- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -190,8 +190,13 @@ ENTRY(entry_SYSCALL_compat) /* Interrupts are off on entry. */ swapgs
- /* Stash user ESP and switch to the kernel stack. */ + /* Stash user ESP */ movl %esp, %r8d + + /* Use %rsp as scratch reg. User ESP is stashed in r8 */ + SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp + + /* Switch to the kernel stack */ movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp
/* Construct struct pt_regs on stack */ @@ -220,12 +225,6 @@ GLOBAL(entry_SYSCALL_compat_after_hwfram pushq $0 /* pt_regs->r15 = 0 */
/* - * We just saved %rdi so it is safe to clobber. It is not - * preserved during the C calls inside TRACE_IRQS_OFF anyway. - */ - SWITCH_TO_KERNEL_CR3 scratch_reg=%rdi - - /* * User mode is traced as though IRQs are on, and SYSENTER * turned them off. */
kernel-build-reports@lists.linaro.org