On Fri, Dec 11, 2020 at 10:59:59PM +0100, Paolo Bonzini wrote:
On 11/12/20 22:04, Thomas Gleixner wrote:
Its 100ms off with migration, and can be reduced further (customers complained about 5 seconds but seem happy with 0.1ms).
What is 100ms? Guaranteed maximum migration time?
I suppose it's the length between the time from KVM_GET_CLOCK and KVM_GET_MSR(IA32_TSC) to KVM_SET_CLOCK and KVM_SET_MSR(IA32_TSC). But the VM is paused for much longer, the sequence for the non-live part of the migration (aka brownout) is as follows:
pause finish sending RAM receive RAM ~1 sec send paused-VM state finish receiving RAM \ receive paused-VM state ) 0.1 sec restart /The nanosecond and TSC times are sent as part of the paused-VM state at the very end of the live migration process.
So it's still true that the time advances during live migration brownout; 0.1 seconds is just the final part of the live migration process. But for _live_ migration there is no need to design things according to "people are happy if their clock is off by 0.1 seconds only".
Agree. What would be a good way to fix this?
It seems to me using CLOCK_REALTIME as in the interface Maxim is proposing is prone to difference in CLOCK_REALTIME itself.
Perhaps there is another way to measure that 0.1 sec which is independent of the clock values of the source and destination hosts (say by sending a packet once the clock stops counting).
Then on destination measure delta = clock_restart_time - packet_receival and increase clock by that amount.
Again, save-to-disk, reverse debugging and the like are a different story, which is why KVM should delegate policy to userspace (while documenting how to do it right).
Paolo
CLOCK_REALTIME and CLOCK_TAI are off by the time the VM is paused and this state persists up to the point where NTP corrects it with a time jump.
So if migration takes 5 seconds then CLOCK_REALTIME is not off by 100ms it's off by 5 seconds.
CLOCK_MONOTONIC/BOOTTIME might be off by 100ms between pause and resume.