On Fri, Jan 05, 2018 at 07:02:35PM +0100, Greg KH wrote:

On Fri, Jan 05, 2018 at 04:55:07PM +0100, Willy Tarreau wrote:

...

On Fri, Jan 05, 2018 at 03:54:33PM +0100, Greg KH wrote:

...

I'm announcing the release of the 4.4.110 kernel.

All users of the 4.4 kernel series must upgrade.

But be careful, there have been some reports of problems with this release during the -rc review cycle. Hopefully all of those issues are now resolved.

So please test, as of right now, it should be "bug compatible" with the "enterprise" kernel releases with regards to the Meltdown bug and proper support on all virtual platforms (meaning there is still a vdso issue that might trip up some old binaries, again, please test!)

If anyone has any problems, please let me know.

FWIW I've just booted one of our LBs on it and am hammering it at full load with pti enabled and will let it run for the week-end. It takes 860k irq/s and about 1.7M syscalls/s. For now it works well (but slowly). Hopefully if there are any rare race conditions left it has a chance to trigger them.

Thanks for the testing, let me know if you see anything.

Definitely! For now zero error after almost one billion connections and around 15 billion syscalls and 7B irqs.

And "slowly", does that mean it is noticable?

It depends by whom :-) We benchmarked this machine a while ago at 93k connections per second on 4.9 on a single process and now I'm seeing about 60k for a single process. I don't want to digress too much about numbers now as the test conditions certainly differ a bit, I'll have to rerun more detailed ones later. For 99.9% of the users it will not be noticeable. Those having to fight DDoS will certainly notice it. I'm pretty sure we'll run with pti=off at least at the beginning.

I have some querys from the virtual networking people that are getting worried about all of this. I told them to go test, but they were having a hard time finding a kernel to test with. Hopefully we hear back from them now that these are out...

I've tested and found a 40% perf drop on networking under KVM between pti=off and pti=on :-(

Fortunately in our case, people running in VMs are not those interested in performance (that's commonly the case) but I expect it willy impact some high-performance users who tune their VMs very precisely.

I'm currently testing a completely different approach for systems like these running basically a single task. The idea is to limit rdtsc to privileged processes only. I just discovered that my libc happily uses it in the ld.so so that limits my capabilities for now :-) But implementing an emulator could solve this for non-privileged processes, masking the lower bits and losing precision. It would not be a fix but an acceptable mitigation solution for some environments where pti=off is too expensive and where untrusted users are extremely rare (ie: just the remote cron job check disk space and collecting network stats). I already tested the variant of the spectre poc without rdtsc (using a thread and a counter) and it definitely is not something reasonably usable to steal reliable information anymore, I managed to get around 1/10 byte OK, but you never know which one.

For this reason, people considering pti=off as the only solution might sometimes prefer this one as a small improvement (and it could also stop other classes of future attacks, maybe something for KSPP later).

I'll continue to investigate and share my observations.

Have a nice week-end! Willy

On Fri, Jan 05, 2018 at 07:58:04PM +0000, Alan Cox wrote:

...

It depends by whom :-) We benchmarked this machine a while ago at 93k connections per second on 4.9 on a single process and now I'm seeing about 60k for a single process. I don't want to digress too much about numbers now as the test conditions certainly differ a bit, I'll have to rerun more detailed ones later. For 99.9% of the users it will not be noticeable. Those having to fight DDoS will certainly notice it. I'm pretty sure we'll run with pti=off at least at the beginning.

Are you running pti on the vm kernels or the host kernel or both ?

First, just to avoid confusion, numbers above were on bare metal (i7-4790K, having PCID). But as I said, I didn't necessarily apply the exact same test protocol as my goal was to quickly fire a long test to run for the week-end. In previous tests (pti vs non-pti) I only measured 17% difference.

For the tests I ran in KVM, PTI was only in the VM, the host was my laptop that has not yet been rebooted, hence is not protected yet. And here I used the same test protocol with/without PTI. The numbers I have in mind were respectively 30.3k conn/s and 18.3k (with pti).

...

I'm currently testing a completely different approach for systems like these running basically a single task. The idea is to limit rdtsc to privileged processes only. I just discovered that my libc happily uses

The javascript attack in the paper does not use rdtsc,

from what I've seen it uses the precise time reported by the browser which itself uses clock_gettime() (which we can thus control as well). But I might have missed certain points, there has been a lot to read this week...

and the techniques to deal with rdtsc disabling are well known and used in other existing attacks.

Yes i've tested one of them for the spectre poc, but it really did not work well, leading to about 1 among 10 bytes only to be valid. In fact either you run the counter thread on the other sibling of the same core and it significantly perturbates the local activity, or you run it on another core, and the time it takes to retrieve the time requires some L1+L2 traversal. I'm not saying it doesn't work at all, I'm saying that the accuracy is highly degraded and that can turn something 100% reproducible into something requiring a long time to run, making the attack more noticeable (and possibly letting observed data degrade during the period).

...

For this reason, people considering pti=off as the only solution might sometimes prefer this one as a small improvement (and it could also stop other classes of future attacks, maybe something for KSPP later).

For a large class of environments where you are only running code that you trust (or at least if anyone evil changes you've got much bigger problems) that is probably a rational approach anyway.

For now yes unfortunately :-/

Willy

On Fri, Jan 05, 2018 at 07:02:35PM +0100, Greg KH wrote:

On Fri, Jan 05, 2018 at 04:55:07PM +0100, Willy Tarreau wrote:

...

On Fri, Jan 05, 2018 at 03:54:33PM +0100, Greg KH wrote:

...

I'm announcing the release of the 4.4.110 kernel.

All users of the 4.4 kernel series must upgrade.

But be careful, there have been some reports of problems with this release during the -rc review cycle. Hopefully all of those issues are now resolved.

So please test, as of right now, it should be "bug compatible" with the "enterprise" kernel releases with regards to the Meltdown bug and proper support on all virtual platforms (meaning there is still a vdso issue that might trip up some old binaries, again, please test!)

If anyone has any problems, please let me know.

FWIW I've just booted one of our LBs on it and am hammering it at full load with pti enabled and will let it run for the week-end. It takes 860k irq/s and about 1.7M syscalls/s. For now it works well (but slowly). Hopefully if there are any rare race conditions left it has a chance to trigger them.

Thanks for the testing, let me know if you see anything. And "slowly", does that mean it is noticable? I have some querys from the virtual networking people that are getting worried about all of this. I told them to go test, but they were having a hard time finding a kernel to test with. Hopefully we hear back from them now that these are out...

So at least the good news is that after 2.5 days, it has flawlessly forwarded 21 billion connections, 3 TB of TCP payload and processed ~180 billion interrupts. No single error in dmesg nor in the test. Thus I'm now quite confident with this kernel's stability.

Regards Willy