This patchset provides a file descriptor for every VM and VCPU to read KVM statistics data in binary format. It is meant to provide a lightweight, flexible, scalable and efficient lock-free solution for user space telemetry applications to pull the statistics data periodically for large scale systems. The pulling frequency could be as high as a few times per second. In this patchset, every statistics data are treated to have some attributes as below: * architecture dependent or generic * VM statistics data or VCPU statistics data * type: cumulative, instantaneous, peak * unit: none for simple counter, nanosecond, microsecond, millisecond, second, Byte, KiByte, MiByte, GiByte. Clock Cycles Since no lock/synchronization is used, the consistency between all the statistics data is not guaranteed. That means not all statistics data are read out at the exact same time, since the statistics date are still being updated by KVM subsystems while they are read out.
---
* v8 -> v9 - Rebase to commit 8331a2bc0898 (KVM: X86: Introduce KVM_HC_MAP_GPA_RANGE hypercall) - Reduce code duplication between binary and debugfs interface - Add field "offset" in stats descriptor to let us define stats descriptors in any order (not necessary in the order of stats defined in vm/vcpu stats structures) - Add static check to make sure the number of stats descriptors is the same as the number of stats defined in vm/vcpu stats structures - Fix missing/mismatched stats descriptor definition caused by rebase
* v7 -> v8 - Rebase to kvm/queue, commit c1dc20e254b4 ("KVM: switch per-VM stats to u64") - Revise code to reflect the per-VM stats type from ulong to u64 - Addressed some other nits
* v6 -> v7 - Improve file descriptor allocation function by Krish suggestion - Use "generic stats" instead of "common stats" as Krish suggested - Addressed some other nits from Krish and David Matlack
* v5 -> v6 - Use designated initializers for STATS_DESC - Change KVM_STATS_SCALE... to KVM_STATS_BASE... - Use a common function for kvm_[vm|vcpu]_stats_read - Fix some documentation errors/missings - Use TEST_ASSERT in selftest - Use a common function for [vm|vcpu]_stats_test in selftest
* v4 -> v5 - Rebase to kvm/queue, commit a4345a7cecfb ("Merge tag 'kvmarm-fixes-5.13-1'") - Change maximum stats name length to 48 - Replace VM_STATS_COMMON/VCPU_STATS_COMMON macros with stats descriptor definition macros. - Fixed some errors/warnings reported by checkpatch.pl
* v3 -> v4 - Rebase to kvm/queue, commit 9f242010c3b4 ("KVM: avoid "deadlock" between install_new_memslots and MMU notifier") - Use C-stype comments in the whole patch - Fix wrong count for x86 VCPU stats descriptors - Fix KVM stats data size counting and validity check in selftest
* v2 -> v3 - Rebase to kvm/queue, commit edf408f5257b ("KVM: avoid "deadlock" between install_new_memslots and MMU notifier") - Resolve some nitpicks about format
* v1 -> v2 - Use ARRAY_SIZE to count the number of stats descriptors - Fix missing `size` field initialization in macro STATS_DESC
[1] https://lore.kernel.org/kvm/20210402224359.2297157-1-jingzhangos@google.com [2] https://lore.kernel.org/kvm/20210415151741.1607806-1-jingzhangos@google.com [3] https://lore.kernel.org/kvm/20210423181727.596466-1-jingzhangos@google.com [4] https://lore.kernel.org/kvm/20210429203740.1935629-1-jingzhangos@google.com [5] https://lore.kernel.org/kvm/20210517145314.157626-1-jingzhangos@google.com [6] https://lore.kernel.org/kvm/20210524151828.4113777-1-jingzhangos@google.com [7] https://lore.kernel.org/kvm/20210603211426.790093-1-jingzhangos@google.com [8] https://lore.kernel.org/kvm/20210611124624.1404010-1-jingzhangos@google.com
---
Jing Zhang (5): KVM: stats: Separate generic stats from architecture specific ones KVM: stats: Add fd-based API to read binary stats data KVM: stats: Add documentation for statistics data binary interface KVM: selftests: Add selftest for KVM statistics data binary interface KVM: stats: Remove code duplication for binary and debugfs stats
Documentation/virt/kvm/api.rst | 177 +++++++++++- arch/arm64/include/asm/kvm_host.h | 9 +- arch/arm64/kvm/guest.c | 50 +++- arch/mips/include/asm/kvm_host.h | 9 +- arch/mips/kvm/mips.c | 92 +++--- arch/powerpc/include/asm/kvm_host.h | 9 +- arch/powerpc/kvm/book3s.c | 93 ++++--- arch/powerpc/kvm/book3s_hv.c | 12 +- arch/powerpc/kvm/book3s_pr.c | 2 +- arch/powerpc/kvm/book3s_pr_papr.c | 2 +- arch/powerpc/kvm/booke.c | 78 ++++-- arch/s390/include/asm/kvm_host.h | 9 +- arch/s390/kvm/kvm-s390.c | 234 +++++++++------- arch/x86/include/asm/kvm_host.h | 9 +- arch/x86/kvm/x86.c | 111 +++++--- include/linux/kvm_host.h | 178 +++++++++++- include/linux/kvm_types.h | 12 + include/uapi/linux/kvm.h | 48 ++++ tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 3 + .../testing/selftests/kvm/include/kvm_util.h | 3 + .../selftests/kvm/kvm_binary_stats_test.c | 225 +++++++++++++++ tools/testing/selftests/kvm/lib/kvm_util.c | 12 + virt/kvm/kvm_main.c | 261 +++++++++++++++--- 24 files changed, 1293 insertions(+), 346 deletions(-) create mode 100644 tools/testing/selftests/kvm/kvm_binary_stats_test.c
base-commit: 8331a2bc089881d7fd2fc9a6658f39780817e4e0
Put all generic statistics in a separate structure to ease statistics handling for the incoming new statistics API.
No functional change intended.
Reviewed-by: David Matlack dmatlack@google.com Reviewed-by: Ricardo Koller ricarkol@google.com Reviewed-by: Krish Sadhukhan krish.sadhukhan@oracle.com Signed-off-by: Jing Zhang jingzhangos@google.com --- arch/arm64/include/asm/kvm_host.h | 9 ++------- arch/arm64/kvm/guest.c | 12 ++++++------ arch/mips/include/asm/kvm_host.h | 9 ++------- arch/mips/kvm/mips.c | 12 ++++++------ arch/powerpc/include/asm/kvm_host.h | 9 ++------- arch/powerpc/kvm/book3s.c | 12 ++++++------ arch/powerpc/kvm/book3s_hv.c | 12 ++++++------ arch/powerpc/kvm/book3s_pr.c | 2 +- arch/powerpc/kvm/book3s_pr_papr.c | 2 +- arch/powerpc/kvm/booke.c | 14 +++++++------- arch/s390/include/asm/kvm_host.h | 9 ++------- arch/s390/kvm/kvm-s390.c | 12 ++++++------ arch/x86/include/asm/kvm_host.h | 9 ++------- arch/x86/kvm/x86.c | 14 +++++++------- include/linux/kvm_host.h | 9 +++++++-- include/linux/kvm_types.h | 12 ++++++++++++ virt/kvm/kvm_main.c | 14 +++++++------- 17 files changed, 82 insertions(+), 90 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index d56f365b38a8..5a2c82f63baa 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -556,16 +556,11 @@ static inline bool __vcpu_write_sys_reg_to_cpu(u64 val, int reg) }
struct kvm_vm_stat { - u64 remote_tlb_flush; + struct kvm_vm_stat_generic generic; };
struct kvm_vcpu_stat { - u64 halt_successful_poll; - u64 halt_attempted_poll; - u64 halt_poll_success_ns; - u64 halt_poll_fail_ns; - u64 halt_poll_invalid; - u64 halt_wakeup; + struct kvm_vcpu_stat_generic generic; u64 hvc_exit_stat; u64 wfe_exit_stat; u64 wfi_exit_stat; diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 5cb4a1cd5603..4962331d01e6 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -29,18 +29,18 @@ #include "trace.h"
struct kvm_stats_debugfs_item debugfs_entries[] = { - VCPU_STAT("halt_successful_poll", halt_successful_poll), - VCPU_STAT("halt_attempted_poll", halt_attempted_poll), - VCPU_STAT("halt_poll_invalid", halt_poll_invalid), - VCPU_STAT("halt_wakeup", halt_wakeup), + VCPU_STAT_GENERIC("halt_successful_poll", halt_successful_poll), + VCPU_STAT_GENERIC("halt_attempted_poll", halt_attempted_poll), + VCPU_STAT_GENERIC("halt_poll_invalid", halt_poll_invalid), + VCPU_STAT_GENERIC("halt_wakeup", halt_wakeup), VCPU_STAT("hvc_exit_stat", hvc_exit_stat), VCPU_STAT("wfe_exit_stat", wfe_exit_stat), VCPU_STAT("wfi_exit_stat", wfi_exit_stat), VCPU_STAT("mmio_exit_user", mmio_exit_user), VCPU_STAT("mmio_exit_kernel", mmio_exit_kernel), VCPU_STAT("exits", exits), - VCPU_STAT("halt_poll_success_ns", halt_poll_success_ns), - VCPU_STAT("halt_poll_fail_ns", halt_poll_fail_ns), + VCPU_STAT_GENERIC("halt_poll_success_ns", halt_poll_success_ns), + VCPU_STAT_GENERIC("halt_poll_fail_ns", halt_poll_fail_ns), { NULL } };
diff --git a/arch/mips/include/asm/kvm_host.h b/arch/mips/include/asm/kvm_host.h index 4245c082095f..696f6b009377 100644 --- a/arch/mips/include/asm/kvm_host.h +++ b/arch/mips/include/asm/kvm_host.h @@ -109,10 +109,11 @@ static inline bool kvm_is_error_hva(unsigned long addr) }
struct kvm_vm_stat { - u64 remote_tlb_flush; + struct kvm_vm_stat_generic generic; };
struct kvm_vcpu_stat { + struct kvm_vcpu_stat_generic generic; u64 wait_exits; u64 cache_exits; u64 signal_exits; @@ -142,12 +143,6 @@ struct kvm_vcpu_stat { #ifdef CONFIG_CPU_LOONGSON64 u64 vz_cpucfg_exits; #endif - u64 halt_successful_poll; - u64 halt_attempted_poll; - u64 halt_poll_success_ns; - u64 halt_poll_fail_ns; - u64 halt_poll_invalid; - u64 halt_wakeup; };
struct kvm_arch_memory_slot { diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c index 4d4af97dcc88..ff205b35719b 100644 --- a/arch/mips/kvm/mips.c +++ b/arch/mips/kvm/mips.c @@ -68,12 +68,12 @@ struct kvm_stats_debugfs_item debugfs_entries[] = { #ifdef CONFIG_CPU_LOONGSON64 VCPU_STAT("vz_cpucfg", vz_cpucfg_exits), #endif - VCPU_STAT("halt_successful_poll", halt_successful_poll), - VCPU_STAT("halt_attempted_poll", halt_attempted_poll), - VCPU_STAT("halt_poll_invalid", halt_poll_invalid), - VCPU_STAT("halt_wakeup", halt_wakeup), - VCPU_STAT("halt_poll_success_ns", halt_poll_success_ns), - VCPU_STAT("halt_poll_fail_ns", halt_poll_fail_ns), + VCPU_STAT_GENERIC("halt_successful_poll", halt_successful_poll), + VCPU_STAT_GENERIC("halt_attempted_poll", halt_attempted_poll), + VCPU_STAT_GENERIC("halt_poll_invalid", halt_poll_invalid), + VCPU_STAT_GENERIC("halt_wakeup", halt_wakeup), + VCPU_STAT_GENERIC("halt_poll_success_ns", halt_poll_success_ns), + VCPU_STAT_GENERIC("halt_poll_fail_ns", halt_poll_fail_ns), {NULL} };
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h index ae3d4af61b66..80cfe4f42894 100644 --- a/arch/powerpc/include/asm/kvm_host.h +++ b/arch/powerpc/include/asm/kvm_host.h @@ -81,12 +81,13 @@ struct kvmppc_book3s_shadow_vcpu; struct kvm_nested_guest;
struct kvm_vm_stat { - u64 remote_tlb_flush; + struct kvm_vm_stat_generic generic; u64 num_2M_pages; u64 num_1G_pages; };
struct kvm_vcpu_stat { + struct kvm_vcpu_stat_generic generic; u64 sum_exits; u64 mmio_exits; u64 signal_exits; @@ -102,14 +103,8 @@ struct kvm_vcpu_stat { u64 emulated_inst_exits; u64 dec_exits; u64 ext_intr_exits; - u64 halt_poll_success_ns; - u64 halt_poll_fail_ns; u64 halt_wait_ns; - u64 halt_successful_poll; - u64 halt_attempted_poll; u64 halt_successful_wait; - u64 halt_poll_invalid; - u64 halt_wakeup; u64 dbell_exits; u64 gdbell_exits; u64 ld; diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c index 2b691f4d1f26..92cdb4175945 100644 --- a/arch/powerpc/kvm/book3s.c +++ b/arch/powerpc/kvm/book3s.c @@ -47,14 +47,14 @@ struct kvm_stats_debugfs_item debugfs_entries[] = { VCPU_STAT("dec", dec_exits), VCPU_STAT("ext_intr", ext_intr_exits), VCPU_STAT("queue_intr", queue_intr), - VCPU_STAT("halt_poll_success_ns", halt_poll_success_ns), - VCPU_STAT("halt_poll_fail_ns", halt_poll_fail_ns), + VCPU_STAT_GENERIC("halt_poll_success_ns", halt_poll_success_ns), + VCPU_STAT_GENERIC("halt_poll_fail_ns", halt_poll_fail_ns), VCPU_STAT("halt_wait_ns", halt_wait_ns), - VCPU_STAT("halt_successful_poll", halt_successful_poll), - VCPU_STAT("halt_attempted_poll", halt_attempted_poll), + VCPU_STAT_GENERIC("halt_successful_poll", halt_successful_poll), + VCPU_STAT_GENERIC("halt_attempted_poll", halt_attempted_poll), VCPU_STAT("halt_successful_wait", halt_successful_wait), - VCPU_STAT("halt_poll_invalid", halt_poll_invalid), - VCPU_STAT("halt_wakeup", halt_wakeup), + VCPU_STAT_GENERIC("halt_poll_invalid", halt_poll_invalid), + VCPU_STAT_GENERIC("halt_wakeup", halt_wakeup), VCPU_STAT("pf_storage", pf_storage), VCPU_STAT("sp_storage", sp_storage), VCPU_STAT("pf_instruc", pf_instruc), diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c index 7360350e66ff..b3506c35a46b 100644 --- a/arch/powerpc/kvm/book3s_hv.c +++ b/arch/powerpc/kvm/book3s_hv.c @@ -236,7 +236,7 @@ static void kvmppc_fast_vcpu_kick_hv(struct kvm_vcpu *vcpu)
waitp = kvm_arch_vcpu_get_wait(vcpu); if (rcuwait_wake_up(waitp)) - ++vcpu->stat.halt_wakeup; + ++vcpu->stat.generic.halt_wakeup;
cpu = READ_ONCE(vcpu->arch.thread_cpu); if (cpu >= 0 && kvmppc_ipi_thread(cpu)) @@ -3925,7 +3925,7 @@ static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc) cur = start_poll = ktime_get(); if (vc->halt_poll_ns) { ktime_t stop = ktime_add_ns(start_poll, vc->halt_poll_ns); - ++vc->runner->stat.halt_attempted_poll; + ++vc->runner->stat.generic.halt_attempted_poll;
vc->vcore_state = VCORE_POLLING; spin_unlock(&vc->lock); @@ -3942,7 +3942,7 @@ static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc) vc->vcore_state = VCORE_INACTIVE;
if (!do_sleep) { - ++vc->runner->stat.halt_successful_poll; + ++vc->runner->stat.generic.halt_successful_poll; goto out; } } @@ -3954,7 +3954,7 @@ static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc) do_sleep = 0; /* If we polled, count this as a successful poll */ if (vc->halt_poll_ns) - ++vc->runner->stat.halt_successful_poll; + ++vc->runner->stat.generic.halt_successful_poll; goto out; }
@@ -3981,13 +3981,13 @@ static void kvmppc_vcore_blocked(struct kvmppc_vcore *vc) ktime_to_ns(cur) - ktime_to_ns(start_wait); /* Attribute failed poll time */ if (vc->halt_poll_ns) - vc->runner->stat.halt_poll_fail_ns += + vc->runner->stat.generic.halt_poll_fail_ns += ktime_to_ns(start_wait) - ktime_to_ns(start_poll); } else { /* Attribute successful poll time */ if (vc->halt_poll_ns) - vc->runner->stat.halt_poll_success_ns += + vc->runner->stat.generic.halt_poll_success_ns += ktime_to_ns(cur) - ktime_to_ns(start_poll); } diff --git a/arch/powerpc/kvm/book3s_pr.c b/arch/powerpc/kvm/book3s_pr.c index d7733b07f489..71bcb0140461 100644 --- a/arch/powerpc/kvm/book3s_pr.c +++ b/arch/powerpc/kvm/book3s_pr.c @@ -493,7 +493,7 @@ static void kvmppc_set_msr_pr(struct kvm_vcpu *vcpu, u64 msr) if (!vcpu->arch.pending_exceptions) { kvm_vcpu_block(vcpu); kvm_clear_request(KVM_REQ_UNHALT, vcpu); - vcpu->stat.halt_wakeup++; + vcpu->stat.generic.halt_wakeup++;
/* Unset POW bit after we woke up */ msr &= ~MSR_POW; diff --git a/arch/powerpc/kvm/book3s_pr_papr.c b/arch/powerpc/kvm/book3s_pr_papr.c index 031c8015864a..ac14239f3424 100644 --- a/arch/powerpc/kvm/book3s_pr_papr.c +++ b/arch/powerpc/kvm/book3s_pr_papr.c @@ -378,7 +378,7 @@ int kvmppc_h_pr(struct kvm_vcpu *vcpu, unsigned long cmd) kvmppc_set_msr_fast(vcpu, kvmppc_get_msr(vcpu) | MSR_EE); kvm_vcpu_block(vcpu); kvm_clear_request(KVM_REQ_UNHALT, vcpu); - vcpu->stat.halt_wakeup++; + vcpu->stat.generic.halt_wakeup++; return EMULATE_DONE; case H_LOGICAL_CI_LOAD: return kvmppc_h_pr_logical_ci_load(vcpu); diff --git a/arch/powerpc/kvm/booke.c b/arch/powerpc/kvm/booke.c index 7d5fe43f85c4..80d3b39aa7ac 100644 --- a/arch/powerpc/kvm/booke.c +++ b/arch/powerpc/kvm/booke.c @@ -49,15 +49,15 @@ struct kvm_stats_debugfs_item debugfs_entries[] = { VCPU_STAT("inst_emu", emulated_inst_exits), VCPU_STAT("dec", dec_exits), VCPU_STAT("ext_intr", ext_intr_exits), - VCPU_STAT("halt_successful_poll", halt_successful_poll), - VCPU_STAT("halt_attempted_poll", halt_attempted_poll), - VCPU_STAT("halt_poll_invalid", halt_poll_invalid), - VCPU_STAT("halt_wakeup", halt_wakeup), + VCPU_STAT_GENERIC("halt_successful_poll", halt_successful_poll), + VCPU_STAT_GENERIC("halt_attempted_poll", halt_attempted_poll), + VCPU_STAT_GENERIC("halt_poll_invalid", halt_poll_invalid), + VCPU_STAT_GENERIC("halt_wakeup", halt_wakeup), VCPU_STAT("doorbell", dbell_exits), VCPU_STAT("guest doorbell", gdbell_exits), - VCPU_STAT("halt_poll_success_ns", halt_poll_success_ns), - VCPU_STAT("halt_poll_fail_ns", halt_poll_fail_ns), - VM_STAT("remote_tlb_flush", remote_tlb_flush), + VCPU_STAT_GENERIC("halt_poll_success_ns", halt_poll_success_ns), + VCPU_STAT_GENERIC("halt_poll_fail_ns", halt_poll_fail_ns), + VM_STAT_GENERIC("remote_tlb_flush", remote_tlb_flush), { NULL } };
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h index 8925f3969478..9b4473f76e56 100644 --- a/arch/s390/include/asm/kvm_host.h +++ b/arch/s390/include/asm/kvm_host.h @@ -361,6 +361,7 @@ struct sie_page { };
struct kvm_vcpu_stat { + struct kvm_vcpu_stat_generic generic; u64 exit_userspace; u64 exit_null; u64 exit_external_request; @@ -370,13 +371,7 @@ struct kvm_vcpu_stat { u64 exit_validity; u64 exit_instruction; u64 exit_pei; - u64 halt_successful_poll; - u64 halt_attempted_poll; - u64 halt_poll_invalid; u64 halt_no_poll_steal; - u64 halt_wakeup; - u64 halt_poll_success_ns; - u64 halt_poll_fail_ns; u64 instruction_lctl; u64 instruction_lctlg; u64 instruction_stctl; @@ -755,12 +750,12 @@ struct kvm_vcpu_arch { };
struct kvm_vm_stat { + struct kvm_vm_stat_generic generic; u64 inject_io; u64 inject_float_mchk; u64 inject_pfault_done; u64 inject_service_signal; u64 inject_virtio; - u64 remote_tlb_flush; };
struct kvm_arch_memory_slot { diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index 1296fc10f80c..e8bc7cd06794 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -72,13 +72,13 @@ struct kvm_stats_debugfs_item debugfs_entries[] = { VCPU_STAT("exit_program_interruption", exit_program_interruption), VCPU_STAT("exit_instr_and_program_int", exit_instr_and_program), VCPU_STAT("exit_operation_exception", exit_operation_exception), - VCPU_STAT("halt_successful_poll", halt_successful_poll), - VCPU_STAT("halt_attempted_poll", halt_attempted_poll), - VCPU_STAT("halt_poll_invalid", halt_poll_invalid), + VCPU_STAT_GENERIC("halt_successful_poll", halt_successful_poll), + VCPU_STAT_GENERIC("halt_attempted_poll", halt_attempted_poll), + VCPU_STAT_GENERIC("halt_poll_invalid", halt_poll_invalid), VCPU_STAT("halt_no_poll_steal", halt_no_poll_steal), - VCPU_STAT("halt_wakeup", halt_wakeup), - VCPU_STAT("halt_poll_success_ns", halt_poll_success_ns), - VCPU_STAT("halt_poll_fail_ns", halt_poll_fail_ns), + VCPU_STAT_GENERIC("halt_wakeup", halt_wakeup), + VCPU_STAT_GENERIC("halt_poll_success_ns", halt_poll_success_ns), + VCPU_STAT_GENERIC("halt_poll_fail_ns", halt_poll_fail_ns), VCPU_STAT("instruction_lctlg", instruction_lctlg), VCPU_STAT("instruction_lctl", instruction_lctl), VCPU_STAT("instruction_stctl", instruction_stctl), diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 5d7e4b99993d..f524fceaffaa 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1167,6 +1167,7 @@ struct kvm_arch { };
struct kvm_vm_stat { + struct kvm_vm_stat_generic generic; u64 mmu_shadow_zapped; u64 mmu_pte_write; u64 mmu_pde_zapped; @@ -1174,13 +1175,13 @@ struct kvm_vm_stat { u64 mmu_recycled; u64 mmu_cache_miss; u64 mmu_unsync; - u64 remote_tlb_flush; u64 lpages; u64 nx_lpage_splits; u64 max_mmu_page_hash_collisions; };
struct kvm_vcpu_stat { + struct kvm_vcpu_stat_generic generic; u64 pf_fixed; u64 pf_guest; u64 tlb_flush; @@ -1194,10 +1195,6 @@ struct kvm_vcpu_stat { u64 nmi_window_exits; u64 l1d_flush; u64 halt_exits; - u64 halt_successful_poll; - u64 halt_attempted_poll; - u64 halt_poll_invalid; - u64 halt_wakeup; u64 request_irq_exits; u64 irq_exits; u64 host_state_reload; @@ -1208,8 +1205,6 @@ struct kvm_vcpu_stat { u64 irq_injections; u64 nmi_injections; u64 req_event; - u64 halt_poll_success_ns; - u64 halt_poll_fail_ns; u64 nested_run; u64 directed_yield_attempted; u64 directed_yield_successful; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 8b898ec8d349..157212157aee 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -235,10 +235,10 @@ struct kvm_stats_debugfs_item debugfs_entries[] = { VCPU_STAT("irq_window", irq_window_exits), VCPU_STAT("nmi_window", nmi_window_exits), VCPU_STAT("halt_exits", halt_exits), - VCPU_STAT("halt_successful_poll", halt_successful_poll), - VCPU_STAT("halt_attempted_poll", halt_attempted_poll), - VCPU_STAT("halt_poll_invalid", halt_poll_invalid), - VCPU_STAT("halt_wakeup", halt_wakeup), + VCPU_STAT_GENERIC("halt_successful_poll", halt_successful_poll), + VCPU_STAT_GENERIC("halt_attempted_poll", halt_attempted_poll), + VCPU_STAT_GENERIC("halt_poll_invalid", halt_poll_invalid), + VCPU_STAT_GENERIC("halt_wakeup", halt_wakeup), VCPU_STAT("hypercalls", hypercalls), VCPU_STAT("request_irq", request_irq_exits), VCPU_STAT("irq_exits", irq_exits), @@ -250,8 +250,8 @@ struct kvm_stats_debugfs_item debugfs_entries[] = { VCPU_STAT("nmi_injections", nmi_injections), VCPU_STAT("req_event", req_event), VCPU_STAT("l1d_flush", l1d_flush), - VCPU_STAT("halt_poll_success_ns", halt_poll_success_ns), - VCPU_STAT("halt_poll_fail_ns", halt_poll_fail_ns), + VCPU_STAT_GENERIC("halt_poll_success_ns", halt_poll_success_ns), + VCPU_STAT_GENERIC("halt_poll_fail_ns", halt_poll_fail_ns), VCPU_STAT("nested_run", nested_run), VCPU_STAT("directed_yield_attempted", directed_yield_attempted), VCPU_STAT("directed_yield_successful", directed_yield_successful), @@ -263,7 +263,7 @@ struct kvm_stats_debugfs_item debugfs_entries[] = { VM_STAT("mmu_recycled", mmu_recycled), VM_STAT("mmu_cache_miss", mmu_cache_miss), VM_STAT("mmu_unsync", mmu_unsync), - VM_STAT("remote_tlb_flush", remote_tlb_flush), + VM_STAT_GENERIC("remote_tlb_flush", remote_tlb_flush), VM_STAT("largepages", lpages, .mode = 0444), VM_STAT("nx_largepages_splitted", nx_lpage_splits, .mode = 0444), VM_STAT("max_mmu_page_hash_collisions", max_mmu_page_hash_collisions), diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 37cbb56ccd09..5a31e0696360 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1275,10 +1275,15 @@ struct kvm_stats_debugfs_item { #define KVM_DBGFS_GET_MODE(dbgfs_item) \ ((dbgfs_item)->mode ? (dbgfs_item)->mode : 0644)
-#define VM_STAT(n, x, ...) \ +#define VM_STAT(n, x, ...) \ { n, offsetof(struct kvm, stat.x), KVM_STAT_VM, ## __VA_ARGS__ } -#define VCPU_STAT(n, x, ...) \ +#define VCPU_STAT(n, x, ...) \ { n, offsetof(struct kvm_vcpu, stat.x), KVM_STAT_VCPU, ## __VA_ARGS__ } +#define VM_STAT_GENERIC(n, x, ...) \ + { n, offsetof(struct kvm, stat.generic.x), KVM_STAT_VM, ## __VA_ARGS__ } +#define VCPU_STAT_GENERIC(n, x, ...) \ + { n, offsetof(struct kvm_vcpu, stat.generic.x), \ + KVM_STAT_VCPU, ## __VA_ARGS__ }
extern struct kvm_stats_debugfs_item debugfs_entries[]; extern struct dentry *kvm_debugfs_dir; diff --git a/include/linux/kvm_types.h b/include/linux/kvm_types.h index a7580f69dda0..48db778291b7 100644 --- a/include/linux/kvm_types.h +++ b/include/linux/kvm_types.h @@ -76,5 +76,17 @@ struct kvm_mmu_memory_cache { }; #endif
+struct kvm_vm_stat_generic { + u64 remote_tlb_flush; +}; + +struct kvm_vcpu_stat_generic { + u64 halt_successful_poll; + u64 halt_attempted_poll; + u64 halt_poll_invalid; + u64 halt_wakeup; + u64 halt_poll_success_ns; + u64 halt_poll_fail_ns; +};
#endif /* __KVM_TYPES_H__ */ diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index ed4d1581d502..cec986487b30 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -332,7 +332,7 @@ void kvm_flush_remote_tlbs(struct kvm *kvm) */ if (!kvm_arch_flush_remote_tlb(kvm) || kvm_make_all_cpus_request(kvm, KVM_REQ_TLB_FLUSH)) - ++kvm->stat.remote_tlb_flush; + ++kvm->stat.generic.remote_tlb_flush; cmpxchg(&kvm->tlbs_dirty, dirty_count, 0); } EXPORT_SYMBOL_GPL(kvm_flush_remote_tlbs); @@ -3029,9 +3029,9 @@ static inline void update_halt_poll_stats(struct kvm_vcpu *vcpu, u64 poll_ns, bool waited) { if (waited) - vcpu->stat.halt_poll_fail_ns += poll_ns; + vcpu->stat.generic.halt_poll_fail_ns += poll_ns; else - vcpu->stat.halt_poll_success_ns += poll_ns; + vcpu->stat.generic.halt_poll_success_ns += poll_ns; }
/* @@ -3049,16 +3049,16 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu) if (vcpu->halt_poll_ns && !kvm_arch_no_poll(vcpu)) { ktime_t stop = ktime_add_ns(ktime_get(), vcpu->halt_poll_ns);
- ++vcpu->stat.halt_attempted_poll; + ++vcpu->stat.generic.halt_attempted_poll; do { /* * This sets KVM_REQ_UNHALT if an interrupt * arrives. */ if (kvm_vcpu_check_block(vcpu) < 0) { - ++vcpu->stat.halt_successful_poll; + ++vcpu->stat.generic.halt_successful_poll; if (!vcpu_valid_wakeup(vcpu)) - ++vcpu->stat.halt_poll_invalid; + ++vcpu->stat.generic.halt_poll_invalid; goto out; } poll_end = cur = ktime_get(); @@ -3115,7 +3115,7 @@ bool kvm_vcpu_wake_up(struct kvm_vcpu *vcpu) waitp = kvm_arch_vcpu_get_wait(vcpu); if (rcuwait_wake_up(waitp)) { WRITE_ONCE(vcpu->ready, true); - ++vcpu->stat.halt_wakeup; + ++vcpu->stat.generic.halt_wakeup; return true; }
Provides a file descriptor per VM to read VM stats info/data. Provides a file descriptor per vCPU to read vCPU stats info/data.
Reviewed-by: David Matlack dmatlack@google.com Reviewed-by: Ricardo Koller ricarkol@google.com Reviewed-by: Krish Sadhukhan krish.sadhukhan@oracle.com Reviewed-by: Fuad Tabba tabba@google.com Tested-by: Fuad Tabba tabba@google.com #arm64 Signed-off-by: Jing Zhang jingzhangos@google.com --- arch/arm64/kvm/guest.c | 38 +++++++++ arch/mips/kvm/mips.c | 61 ++++++++++++++ arch/powerpc/kvm/book3s.c | 64 +++++++++++++++ arch/powerpc/kvm/booke.c | 57 ++++++++++++++ arch/s390/kvm/kvm-s390.c | 130 ++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 68 ++++++++++++++++ include/linux/kvm_host.h | 162 ++++++++++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 46 +++++++++++ virt/kvm/kvm_main.c | 143 +++++++++++++++++++++++++++++++++ 9 files changed, 769 insertions(+)
diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 4962331d01e6..99842e29c2de 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -28,6 +28,44 @@
#include "trace.h"
+struct _kvm_stats_desc kvm_vm_stats_desc[] = { + KVM_GENERIC_VM_STATS() +}; +static_assert(ARRAY_SIZE(kvm_vm_stats_desc) == + sizeof(struct kvm_vm_stat) / sizeof(u64)); + +struct _kvm_stats_header kvm_vm_stats_header = { + { + .name_size = KVM_STATS_NAME_LEN, + .count = ARRAY_SIZE(kvm_vm_stats_desc), + .desc_offset = sizeof(struct _kvm_stats_header), + .data_offset = sizeof(struct _kvm_stats_header) + + sizeof(kvm_vm_stats_desc), + } +}; + +struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { + KVM_GENERIC_VCPU_STATS(), + STATS_DESC_COUNTER(VCPU, hvc_exit_stat), + STATS_DESC_COUNTER(VCPU, wfe_exit_stat), + STATS_DESC_COUNTER(VCPU, wfi_exit_stat), + STATS_DESC_COUNTER(VCPU, mmio_exit_user), + STATS_DESC_COUNTER(VCPU, mmio_exit_kernel), + STATS_DESC_COUNTER(VCPU, exits) +}; +static_assert(ARRAY_SIZE(kvm_vcpu_stats_desc) == + sizeof(struct kvm_vcpu_stat) / sizeof(u64)); + +struct _kvm_stats_header kvm_vcpu_stats_header = { + { + .name_size = KVM_STATS_NAME_LEN, + .count = ARRAY_SIZE(kvm_vcpu_stats_desc), + .desc_offset = sizeof(struct _kvm_stats_header), + .data_offset = sizeof(struct _kvm_stats_header) + + sizeof(kvm_vcpu_stats_desc), + } +}; + struct kvm_stats_debugfs_item debugfs_entries[] = { VCPU_STAT_GENERIC("halt_successful_poll", halt_successful_poll), VCPU_STAT_GENERIC("halt_attempted_poll", halt_attempted_poll), diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c index ff205b35719b..67404f0947aa 100644 --- a/arch/mips/kvm/mips.c +++ b/arch/mips/kvm/mips.c @@ -38,6 +38,67 @@ #define VECTORSPACING 0x100 /* for EI/VI mode */ #endif
+struct _kvm_stats_desc kvm_vm_stats_desc[] = { + KVM_GENERIC_VM_STATS() +}; +static_assert(ARRAY_SIZE(kvm_vm_stats_desc) == + sizeof(struct kvm_vm_stat) / sizeof(u64)); + +struct _kvm_stats_header kvm_vm_stats_header = { + { + .name_size = KVM_STATS_NAME_LEN, + .count = ARRAY_SIZE(kvm_vm_stats_desc), + .desc_offset = sizeof(struct _kvm_stats_header), + .data_offset = sizeof(struct _kvm_stats_header) + + sizeof(kvm_vm_stats_desc), + } +}; + +struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { + KVM_GENERIC_VCPU_STATS(), + STATS_DESC_COUNTER(VCPU, wait_exits), + STATS_DESC_COUNTER(VCPU, cache_exits), + STATS_DESC_COUNTER(VCPU, signal_exits), + STATS_DESC_COUNTER(VCPU, int_exits), + STATS_DESC_COUNTER(VCPU, cop_unusable_exits), + STATS_DESC_COUNTER(VCPU, tlbmod_exits), + STATS_DESC_COUNTER(VCPU, tlbmiss_ld_exits), + STATS_DESC_COUNTER(VCPU, tlbmiss_st_exits), + STATS_DESC_COUNTER(VCPU, addrerr_st_exits), + STATS_DESC_COUNTER(VCPU, addrerr_ld_exits), + STATS_DESC_COUNTER(VCPU, syscall_exits), + STATS_DESC_COUNTER(VCPU, resvd_inst_exits), + STATS_DESC_COUNTER(VCPU, break_inst_exits), + STATS_DESC_COUNTER(VCPU, trap_inst_exits), + STATS_DESC_COUNTER(VCPU, msa_fpe_exits), + STATS_DESC_COUNTER(VCPU, fpe_exits), + STATS_DESC_COUNTER(VCPU, msa_disabled_exits), + STATS_DESC_COUNTER(VCPU, flush_dcache_exits), + STATS_DESC_COUNTER(VCPU, vz_gpsi_exits), + STATS_DESC_COUNTER(VCPU, vz_gsfc_exits), + STATS_DESC_COUNTER(VCPU, vz_hc_exits), + STATS_DESC_COUNTER(VCPU, vz_grr_exits), + STATS_DESC_COUNTER(VCPU, vz_gva_exits), + STATS_DESC_COUNTER(VCPU, vz_ghfc_exits), + STATS_DESC_COUNTER(VCPU, vz_gpa_exits), + STATS_DESC_COUNTER(VCPU, vz_resvd_exits), +#ifdef CONFIG_CPU_LOONGSON64 + STATS_DESC_COUNTER(VCPU, vz_cpucfg_exits), +#endif +}; +static_assert(ARRAY_SIZE(kvm_vcpu_stats_desc) == + sizeof(struct kvm_vcpu_stat) / sizeof(u64)); + +struct _kvm_stats_header kvm_vcpu_stats_header = { + { + .name_size = KVM_STATS_NAME_LEN, + .count = ARRAY_SIZE(kvm_vcpu_stats_desc), + .desc_offset = sizeof(struct _kvm_stats_header), + .data_offset = sizeof(struct _kvm_stats_header) + + sizeof(kvm_vcpu_stats_desc), + } +}; + struct kvm_stats_debugfs_item debugfs_entries[] = { VCPU_STAT("wait", wait_exits), VCPU_STAT("cache", cache_exits), diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c index 92cdb4175945..15436484e521 100644 --- a/arch/powerpc/kvm/book3s.c +++ b/arch/powerpc/kvm/book3s.c @@ -38,6 +38,70 @@
/* #define EXIT_DEBUG */
+struct _kvm_stats_desc kvm_vm_stats_desc[] = { + KVM_GENERIC_VM_STATS(), + STATS_DESC_ICOUNTER(VM, num_2M_pages), + STATS_DESC_ICOUNTER(VM, num_1G_pages) +}; +static_assert(ARRAY_SIZE(kvm_vm_stats_desc) == + sizeof(struct kvm_vm_stat) / sizeof(u64)); + +struct _kvm_stats_header kvm_vm_stats_header = { + { + .name_size = KVM_STATS_NAME_LEN, + .count = ARRAY_SIZE(kvm_vm_stats_desc), + .desc_offset = sizeof(struct _kvm_stats_header), + .data_offset = sizeof(struct _kvm_stats_header) + + sizeof(kvm_vm_stats_desc), + } +}; + +struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { + KVM_GENERIC_VCPU_STATS(), + STATS_DESC_COUNTER(VCPU, sum_exits), + STATS_DESC_COUNTER(VCPU, mmio_exits), + STATS_DESC_COUNTER(VCPU, signal_exits), + STATS_DESC_COUNTER(VCPU, light_exits), + STATS_DESC_COUNTER(VCPU, itlb_real_miss_exits), + STATS_DESC_COUNTER(VCPU, itlb_virt_miss_exits), + STATS_DESC_COUNTER(VCPU, dtlb_real_miss_exits), + STATS_DESC_COUNTER(VCPU, dtlb_virt_miss_exits), + STATS_DESC_COUNTER(VCPU, syscall_exits), + STATS_DESC_COUNTER(VCPU, isi_exits), + STATS_DESC_COUNTER(VCPU, dsi_exits), + STATS_DESC_COUNTER(VCPU, emulated_inst_exits), + STATS_DESC_COUNTER(VCPU, dec_exits), + STATS_DESC_COUNTER(VCPU, ext_intr_exits), + STATS_DESC_TIME_NSEC(VCPU, halt_wait_ns), + STATS_DESC_COUNTER(VCPU, halt_successful_wait), + STATS_DESC_COUNTER(VCPU, dbell_exits), + STATS_DESC_COUNTER(VCPU, gdbell_exits), + STATS_DESC_COUNTER(VCPU, ld), + STATS_DESC_COUNTER(VCPU, st), + STATS_DESC_COUNTER(VCPU, pf_storage), + STATS_DESC_COUNTER(VCPU, pf_instruc), + STATS_DESC_COUNTER(VCPU, sp_storage), + STATS_DESC_COUNTER(VCPU, sp_instruc), + STATS_DESC_COUNTER(VCPU, queue_intr), + STATS_DESC_COUNTER(VCPU, ld_slow), + STATS_DESC_COUNTER(VCPU, st_slow), + STATS_DESC_COUNTER(VCPU, pthru_all), + STATS_DESC_COUNTER(VCPU, pthru_host), + STATS_DESC_COUNTER(VCPU, pthru_bad_aff) +}; +static_assert(ARRAY_SIZE(kvm_vcpu_stats_desc) == + sizeof(struct kvm_vcpu_stat) / sizeof(u64)); + +struct _kvm_stats_header kvm_vcpu_stats_header = { + { + .name_size = KVM_STATS_NAME_LEN, + .count = ARRAY_SIZE(kvm_vcpu_stats_desc), + .desc_offset = sizeof(struct _kvm_stats_header), + .data_offset = sizeof(struct _kvm_stats_header) + + sizeof(kvm_vcpu_stats_desc), + } +}; + struct kvm_stats_debugfs_item debugfs_entries[] = { VCPU_STAT("exits", sum_exits), VCPU_STAT("mmio", mmio_exits), diff --git a/arch/powerpc/kvm/booke.c b/arch/powerpc/kvm/booke.c index 80d3b39aa7ac..fbc9b7b6af57 100644 --- a/arch/powerpc/kvm/booke.c +++ b/arch/powerpc/kvm/booke.c @@ -36,6 +36,63 @@
unsigned long kvmppc_booke_handlers;
+struct _kvm_stats_desc kvm_vm_stats_desc[] = { + KVM_GENERIC_VM_STATS(), + STATS_DESC_ICOUNTER(VM, num_2M_pages), + STATS_DESC_ICOUNTER(VM, num_1G_pages) +}; +static_assert(ARRAY_SIZE(kvm_vm_stats_desc) == + sizeof(struct kvm_vm_stat) / sizeof(u64)); + +struct _kvm_stats_header kvm_vm_stats_header = { + { + .name_size = KVM_STATS_NAME_LEN, + .count = ARRAY_SIZE(kvm_vm_stats_desc), + .desc_offset = sizeof(struct _kvm_stats_header), + .data_offset = sizeof(struct _kvm_stats_header) + + sizeof(kvm_vm_stats_desc), + } +}; + +struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { + KVM_GENERIC_VCPU_STATS(), + STATS_DESC_COUNTER(VCPU, sum_exits), + STATS_DESC_COUNTER(VCPU, mmio_exits), + STATS_DESC_COUNTER(VCPU, signal_exits), + STATS_DESC_COUNTER(VCPU, light_exits), + STATS_DESC_COUNTER(VCPU, itlb_real_miss_exits), + STATS_DESC_COUNTER(VCPU, itlb_virt_miss_exits), + STATS_DESC_COUNTER(VCPU, dtlb_real_miss_exits), + STATS_DESC_COUNTER(VCPU, dtlb_virt_miss_exits), + STATS_DESC_COUNTER(VCPU, syscall_exits), + STATS_DESC_COUNTER(VCPU, isi_exits), + STATS_DESC_COUNTER(VCPU, dsi_exits), + STATS_DESC_COUNTER(VCPU, emulated_inst_exits), + STATS_DESC_COUNTER(VCPU, dec_exits), + STATS_DESC_COUNTER(VCPU, ext_intr_exits), + STATS_DESC_TIME_NSEC(VCPU, halt_wait_ns), + STATS_DESC_COUNTER(VCPU, halt_successful_wait), + STATS_DESC_COUNTER(VCPU, dbell_exits), + STATS_DESC_COUNTER(VCPU, gdbell_exits), + STATS_DESC_COUNTER(VCPU, ld), + STATS_DESC_COUNTER(VCPU, st), + STATS_DESC_COUNTER(VCPU, pthru_all), + STATS_DESC_COUNTER(VCPU, pthru_host), + STATS_DESC_COUNTER(VCPU, pthru_bad_aff) +}; +static_assert(ARRAY_SIZE(kvm_vcpu_stats_desc) == + sizeof(struct kvm_vcpu_stat) / sizeof(u64)); + +struct _kvm_stats_header kvm_vcpu_stats_header = { + { + .name_size = KVM_STATS_NAME_LEN, + .count = ARRAY_SIZE(kvm_vcpu_stats_desc), + .desc_offset = sizeof(struct _kvm_stats_header), + .data_offset = sizeof(struct _kvm_stats_header) + + sizeof(kvm_vcpu_stats_desc), + } +}; + struct kvm_stats_debugfs_item debugfs_entries[] = { VCPU_STAT("mmio", mmio_exits), VCPU_STAT("sig", signal_exits), diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index e8bc7cd06794..ba50dd775fa8 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -58,6 +58,136 @@ #define VCPU_IRQS_MAX_BUF (sizeof(struct kvm_s390_irq) * \ (KVM_MAX_VCPUS + LOCAL_IRQS))
+struct _kvm_stats_desc kvm_vm_stats_desc[] = { + KVM_GENERIC_VM_STATS(), + STATS_DESC_COUNTER(VM, inject_io), + STATS_DESC_COUNTER(VM, inject_float_mchk), + STATS_DESC_COUNTER(VM, inject_pfault_done), + STATS_DESC_COUNTER(VM, inject_service_signal), + STATS_DESC_COUNTER(VM, inject_virtio) +}; +static_assert(ARRAY_SIZE(kvm_vm_stats_desc) == + sizeof(struct kvm_vm_stat) / sizeof(u64)); + +struct _kvm_stats_header kvm_vm_stats_header = { + { + .name_size = KVM_STATS_NAME_LEN, + .count = ARRAY_SIZE(kvm_vm_stats_desc), + .desc_offset = sizeof(struct _kvm_stats_header), + .data_offset = sizeof(struct _kvm_stats_header) + + sizeof(kvm_vm_stats_desc), + } +}; + +struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { + KVM_GENERIC_VCPU_STATS(), + STATS_DESC_COUNTER(VCPU, exit_userspace), + STATS_DESC_COUNTER(VCPU, exit_null), + STATS_DESC_COUNTER(VCPU, exit_external_request), + STATS_DESC_COUNTER(VCPU, exit_io_request), + STATS_DESC_COUNTER(VCPU, exit_external_interrupt), + STATS_DESC_COUNTER(VCPU, exit_stop_request), + STATS_DESC_COUNTER(VCPU, exit_validity), + STATS_DESC_COUNTER(VCPU, exit_instruction), + STATS_DESC_COUNTER(VCPU, exit_pei), + STATS_DESC_COUNTER(VCPU, halt_no_poll_steal), + STATS_DESC_COUNTER(VCPU, instruction_lctl), + STATS_DESC_COUNTER(VCPU, instruction_lctlg), + STATS_DESC_COUNTER(VCPU, instruction_stctl), + STATS_DESC_COUNTER(VCPU, instruction_stctg), + STATS_DESC_COUNTER(VCPU, exit_program_interruption), + STATS_DESC_COUNTER(VCPU, exit_instr_and_program), + STATS_DESC_COUNTER(VCPU, exit_operation_exception), + STATS_DESC_COUNTER(VCPU, deliver_ckc), + STATS_DESC_COUNTER(VCPU, deliver_cputm), + STATS_DESC_COUNTER(VCPU, deliver_external_call), + STATS_DESC_COUNTER(VCPU, deliver_emergency_signal), + STATS_DESC_COUNTER(VCPU, deliver_service_signal), + STATS_DESC_COUNTER(VCPU, deliver_virtio), + STATS_DESC_COUNTER(VCPU, deliver_stop_signal), + STATS_DESC_COUNTER(VCPU, deliver_prefix_signal), + STATS_DESC_COUNTER(VCPU, deliver_restart_signal), + STATS_DESC_COUNTER(VCPU, deliver_program), + STATS_DESC_COUNTER(VCPU, deliver_io), + STATS_DESC_COUNTER(VCPU, deliver_machine_check), + STATS_DESC_COUNTER(VCPU, exit_wait_state), + STATS_DESC_COUNTER(VCPU, inject_ckc), + STATS_DESC_COUNTER(VCPU, inject_cputm), + STATS_DESC_COUNTER(VCPU, inject_external_call), + STATS_DESC_COUNTER(VCPU, inject_emergency_signal), + STATS_DESC_COUNTER(VCPU, inject_mchk), + STATS_DESC_COUNTER(VCPU, inject_pfault_init), + STATS_DESC_COUNTER(VCPU, inject_program), + STATS_DESC_COUNTER(VCPU, inject_restart), + STATS_DESC_COUNTER(VCPU, inject_set_prefix), + STATS_DESC_COUNTER(VCPU, inject_stop_signal), + STATS_DESC_COUNTER(VCPU, instruction_epsw), + STATS_DESC_COUNTER(VCPU, instruction_gs), + STATS_DESC_COUNTER(VCPU, instruction_io_other), + STATS_DESC_COUNTER(VCPU, instruction_lpsw), + STATS_DESC_COUNTER(VCPU, instruction_lpswe), + STATS_DESC_COUNTER(VCPU, instruction_pfmf), + STATS_DESC_COUNTER(VCPU, instruction_ptff), + STATS_DESC_COUNTER(VCPU, instruction_sck), + STATS_DESC_COUNTER(VCPU, instruction_sckpf), + STATS_DESC_COUNTER(VCPU, instruction_stidp), + STATS_DESC_COUNTER(VCPU, instruction_spx), + STATS_DESC_COUNTER(VCPU, instruction_stpx), + STATS_DESC_COUNTER(VCPU, instruction_stap), + STATS_DESC_COUNTER(VCPU, instruction_iske), + STATS_DESC_COUNTER(VCPU, instruction_ri), + STATS_DESC_COUNTER(VCPU, instruction_rrbe), + STATS_DESC_COUNTER(VCPU, instruction_sske), + STATS_DESC_COUNTER(VCPU, instruction_ipte_interlock), + STATS_DESC_COUNTER(VCPU, instruction_stsi), + STATS_DESC_COUNTER(VCPU, instruction_stfl), + STATS_DESC_COUNTER(VCPU, instruction_tb), + STATS_DESC_COUNTER(VCPU, instruction_tpi), + STATS_DESC_COUNTER(VCPU, instruction_tprot), + STATS_DESC_COUNTER(VCPU, instruction_tsch), + STATS_DESC_COUNTER(VCPU, instruction_sie), + STATS_DESC_COUNTER(VCPU, instruction_essa), + STATS_DESC_COUNTER(VCPU, instruction_sthyi), + STATS_DESC_COUNTER(VCPU, instruction_sigp_sense), + STATS_DESC_COUNTER(VCPU, instruction_sigp_sense_running), + STATS_DESC_COUNTER(VCPU, instruction_sigp_external_call), + STATS_DESC_COUNTER(VCPU, instruction_sigp_emergency), + STATS_DESC_COUNTER(VCPU, instruction_sigp_cond_emergency), + STATS_DESC_COUNTER(VCPU, instruction_sigp_start), + STATS_DESC_COUNTER(VCPU, instruction_sigp_stop), + STATS_DESC_COUNTER(VCPU, instruction_sigp_stop_store_status), + STATS_DESC_COUNTER(VCPU, instruction_sigp_store_status), + STATS_DESC_COUNTER(VCPU, instruction_sigp_store_adtl_status), + STATS_DESC_COUNTER(VCPU, instruction_sigp_arch), + STATS_DESC_COUNTER(VCPU, instruction_sigp_prefix), + STATS_DESC_COUNTER(VCPU, instruction_sigp_restart), + STATS_DESC_COUNTER(VCPU, instruction_sigp_init_cpu_reset), + STATS_DESC_COUNTER(VCPU, instruction_sigp_cpu_reset), + STATS_DESC_COUNTER(VCPU, instruction_sigp_unknown), + STATS_DESC_COUNTER(VCPU, diagnose_10), + STATS_DESC_COUNTER(VCPU, diagnose_44), + STATS_DESC_COUNTER(VCPU, diagnose_9c), + STATS_DESC_COUNTER(VCPU, diagnose_9c_ignored), + STATS_DESC_COUNTER(VCPU, diagnose_9c_forward), + STATS_DESC_COUNTER(VCPU, diagnose_258), + STATS_DESC_COUNTER(VCPU, diagnose_308), + STATS_DESC_COUNTER(VCPU, diagnose_500), + STATS_DESC_COUNTER(VCPU, diagnose_other), + STATS_DESC_COUNTER(VCPU, pfault_sync) +}; +static_assert(ARRAY_SIZE(kvm_vcpu_stats_desc) == + sizeof(struct kvm_vcpu_stat) / sizeof(u64)); + +struct _kvm_stats_header kvm_vcpu_stats_header = { + { + .name_size = KVM_STATS_NAME_LEN, + .count = ARRAY_SIZE(kvm_vcpu_stats_desc), + .desc_offset = sizeof(struct _kvm_stats_header), + .data_offset = sizeof(struct _kvm_stats_header) + + sizeof(kvm_vcpu_stats_desc), + } +}; + struct kvm_stats_debugfs_item debugfs_entries[] = { VCPU_STAT("userspace_handled", exit_userspace), VCPU_STAT("exit_null", exit_null), diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 157212157aee..84438573b529 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -223,6 +223,74 @@ EXPORT_SYMBOL_GPL(host_xss); u64 __read_mostly supported_xss; EXPORT_SYMBOL_GPL(supported_xss);
+struct _kvm_stats_desc kvm_vm_stats_desc[] = { + KVM_GENERIC_VM_STATS(), + STATS_DESC_COUNTER(VM, mmu_shadow_zapped), + STATS_DESC_COUNTER(VM, mmu_pte_write), + STATS_DESC_COUNTER(VM, mmu_pde_zapped), + STATS_DESC_COUNTER(VM, mmu_flooded), + STATS_DESC_COUNTER(VM, mmu_recycled), + STATS_DESC_COUNTER(VM, mmu_cache_miss), + STATS_DESC_ICOUNTER(VM, mmu_unsync), + STATS_DESC_ICOUNTER(VM, lpages), + STATS_DESC_ICOUNTER(VM, nx_lpage_splits), + STATS_DESC_ICOUNTER(VM, max_mmu_page_hash_collisions) +}; +static_assert(ARRAY_SIZE(kvm_vm_stats_desc) == + sizeof(struct kvm_vm_stat) / sizeof(u64)); + +struct _kvm_stats_header kvm_vm_stats_header = { + { + .name_size = KVM_STATS_NAME_LEN, + .count = ARRAY_SIZE(kvm_vm_stats_desc), + .desc_offset = sizeof(struct _kvm_stats_header), + .data_offset = sizeof(struct _kvm_stats_header) + + sizeof(kvm_vm_stats_desc), + } +}; + +struct _kvm_stats_desc kvm_vcpu_stats_desc[] = { + KVM_GENERIC_VCPU_STATS(), + STATS_DESC_COUNTER(VCPU, pf_fixed), + STATS_DESC_COUNTER(VCPU, pf_guest), + STATS_DESC_COUNTER(VCPU, tlb_flush), + STATS_DESC_COUNTER(VCPU, invlpg), + STATS_DESC_COUNTER(VCPU, exits), + STATS_DESC_COUNTER(VCPU, io_exits), + STATS_DESC_COUNTER(VCPU, mmio_exits), + STATS_DESC_COUNTER(VCPU, signal_exits), + STATS_DESC_COUNTER(VCPU, irq_window_exits), + STATS_DESC_COUNTER(VCPU, nmi_window_exits), + STATS_DESC_COUNTER(VCPU, l1d_flush), + STATS_DESC_COUNTER(VCPU, halt_exits), + STATS_DESC_COUNTER(VCPU, request_irq_exits), + STATS_DESC_COUNTER(VCPU, irq_exits), + STATS_DESC_COUNTER(VCPU, host_state_reload), + STATS_DESC_COUNTER(VCPU, fpu_reload), + STATS_DESC_COUNTER(VCPU, insn_emulation), + STATS_DESC_COUNTER(VCPU, insn_emulation_fail), + STATS_DESC_COUNTER(VCPU, hypercalls), + STATS_DESC_COUNTER(VCPU, irq_injections), + STATS_DESC_COUNTER(VCPU, nmi_injections), + STATS_DESC_COUNTER(VCPU, req_event), + STATS_DESC_COUNTER(VCPU, nested_run), + STATS_DESC_COUNTER(VCPU, directed_yield_attempted), + STATS_DESC_COUNTER(VCPU, directed_yield_successful), + STATS_DESC_ICOUNTER(VCPU, guest_mode) +}; +static_assert(ARRAY_SIZE(kvm_vcpu_stats_desc) == + sizeof(struct kvm_vcpu_stat) / sizeof(u64)); + +struct _kvm_stats_header kvm_vcpu_stats_header = { + { + .name_size = KVM_STATS_NAME_LEN, + .count = ARRAY_SIZE(kvm_vcpu_stats_desc), + .desc_offset = sizeof(struct _kvm_stats_header), + .data_offset = sizeof(struct _kvm_stats_header) + + sizeof(kvm_vcpu_stats_desc), + } +}; + struct kvm_stats_debugfs_item debugfs_entries[] = { VCPU_STAT("pf_fixed", pf_fixed), VCPU_STAT("pf_guest", pf_guest), diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 5a31e0696360..692af9177c9f 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1272,6 +1272,17 @@ struct kvm_stats_debugfs_item { int mode; };
+struct _kvm_stats_header { + struct kvm_stats_header header; + char id[KVM_STATS_ID_MAXLEN]; +}; + +#define KVM_STATS_NAME_LEN 48 +struct _kvm_stats_desc { + struct kvm_stats_desc desc; + char name[KVM_STATS_NAME_LEN]; +}; + #define KVM_DBGFS_GET_MODE(dbgfs_item) \ ((dbgfs_item)->mode ? (dbgfs_item)->mode : 0644)
@@ -1285,8 +1296,159 @@ struct kvm_stats_debugfs_item { { n, offsetof(struct kvm_vcpu, stat.generic.x), \ KVM_STAT_VCPU, ## __VA_ARGS__ }
+#define STATS_DESC_COMMON(type, unit, base, exp) \ + .flags = type | unit | base | \ + BUILD_BUG_ON_ZERO(type & ~KVM_STATS_TYPE_MASK) | \ + BUILD_BUG_ON_ZERO(unit & ~KVM_STATS_UNIT_MASK) | \ + BUILD_BUG_ON_ZERO(base & ~KVM_STATS_BASE_MASK), \ + .exponent = exp, \ + .size = 1 + +#define VM_GENERIC_STATS_DESC(stat, type, unit, base, exp) \ + { \ + { \ + STATS_DESC_COMMON(type, unit, base, exp), \ + .offset = offsetof(struct kvm_vm_stat, generic.stat) \ + }, \ + .name = #stat, \ + } +#define VCPU_GENERIC_STATS_DESC(stat, type, unit, base, exp) \ + { \ + { \ + STATS_DESC_COMMON(type, unit, base, exp), \ + .offset = offsetof(struct kvm_vcpu_stat, generic.stat) \ + }, \ + .name = #stat, \ + } +#define VM_STATS_DESC(stat, type, unit, base, exp) \ + { \ + { \ + STATS_DESC_COMMON(type, unit, base, exp), \ + .offset = offsetof(struct kvm_vm_stat, stat) \ + }, \ + .name = #stat, \ + } +#define VCPU_STATS_DESC(stat, type, unit, base, exp) \ + { \ + { \ + STATS_DESC_COMMON(type, unit, base, exp), \ + .offset = offsetof(struct kvm_vcpu_stat, stat) \ + }, \ + .name = #stat, \ + } +/* SCOPE: VM, VM_GENERIC, VCPU, VCPU_GENERIC */ +#define STATS_DESC(SCOPE, stat, type, unit, base, exp) \ + SCOPE##_STATS_DESC(stat, type, unit, base, exp) + +#define STATS_DESC_CUMULATIVE(SCOPE, name, unit, base, exponent) \ + STATS_DESC(SCOPE, name, KVM_STATS_TYPE_CUMULATIVE, \ + unit, base, exponent) +#define STATS_DESC_INSTANT(SCOPE, name, unit, base, exponent) \ + STATS_DESC(SCOPE, name, KVM_STATS_TYPE_INSTANT, unit, base, exponent) \ + +/* Cumulative counter */ +#define STATS_DESC_COUNTER(SCOPE, name) \ + STATS_DESC_CUMULATIVE(SCOPE, name, KVM_STATS_UNIT_NONE, \ + KVM_STATS_BASE_POW10, 0) +/* Instantaneous counter */ +#define STATS_DESC_ICOUNTER(SCOPE, name) \ + STATS_DESC_INSTANT(SCOPE, name, KVM_STATS_UNIT_NONE, \ + KVM_STATS_BASE_POW10, 0) + +/* Cumulative clock cycles */ +#define STATS_DESC_CYCLE(SCOPE, name) \ + STATS_DESC_CUMULATIVE(SCOPE, name, KVM_STATS_UNIT_CYCLES, \ + KVM_STATS_BASE_POW10, 0) +/* Instantaneous clock cycles */ +#define STATS_DESC_ICYCLE(SCOPE, name) \ + STATS_DESC_INSTANT(SCOPE, name, KVM_STATS_UNIT_CYCLES, \ + KVM_STATS_BASE_POW10, 0) + +/* Cumulative memory size in Byte */ +#define STATS_DESC_SIZE_BYTE(SCOPE, name) \ + STATS_DESC_CUMULATIVE(SCOPE, name, KVM_STATS_UNIT_BYTES, \ + KVM_STATS_BASE_POW2, 0) +/* Cumulative memory size in KiByte */ +#define STATS_DESC_SIZE_KBYTE(SCOPE, name) \ + STATS_DESC_CUMULATIVE(SCOPE, name, KVM_STATS_UNIT_BYTES, \ + KVM_STATS_BASE_POW2, 10) +/* Cumulative memory size in MiByte */ +#define STATS_DESC_SIZE_MBYTE(SCOPE, name) \ + STATS_DESC_CUMULATIVE(SCOPE, name, KVM_STATS_UNIT_BYTES, \ + KVM_STATS_BASE_POW2, 20) +/* Cumulative memory size in GiByte */ +#define STATS_DESC_SIZE_GBYTE(SCOPE, name) \ + STATS_DESC_CUMULATIVE(SCOPE, name, KVM_STATS_UNIT_BYTES, \ + KVM_STATS_BASE_POW2, 30) + +/* Instantaneous memory size in Byte */ +#define STATS_DESC_ISIZE_BYTE(SCOPE, name) \ + STATS_DESC_INSTANT(SCOPE, name, KVM_STATS_UNIT_BYTES, \ + KVM_STATS_BASE_POW2, 0) +/* Instantaneous memory size in KiByte */ +#define STATS_DESC_ISIZE_KBYTE(SCOPE, name) \ + STATS_DESC_INSTANT(SCOPE, name, KVM_STATS_UNIT_BYTES, \ + KVM_STATS_BASE_POW2, 10) +/* Instantaneous memory size in MiByte */ +#define STATS_DESC_ISIZE_MBYTE(SCOPE, name) \ + STATS_DESC_INSTANT(SCOPE, name, KVM_STATS_UNIT_BYTES, \ + KVM_STATS_BASE_POW2, 20) +/* Instantaneous memory size in GiByte */ +#define STATS_DESC_ISIZE_GBYTE(SCOPE, name) \ + STATS_DESC_INSTANT(SCOPE, name, KVM_STATS_UNIT_BYTES, \ + KVM_STATS_BASE_POW2, 30) + +/* Cumulative time in second */ +#define STATS_DESC_TIME_SEC(SCOPE, name) \ + STATS_DESC_CUMULATIVE(SCOPE, name, KVM_STATS_UNIT_SECONDS, \ + KVM_STATS_BASE_POW10, 0) +/* Cumulative time in millisecond */ +#define STATS_DESC_TIME_MSEC(SCOPE, name) \ + STATS_DESC_CUMULATIVE(SCOPE, name, KVM_STATS_UNIT_SECONDS, \ + KVM_STATS_BASE_POW10, -3) +/* Cumulative time in microsecond */ +#define STATS_DESC_TIME_USEC(SCOPE, name) \ + STATS_DESC_CUMULATIVE(SCOPE, name, KVM_STATS_UNIT_SECONDS, \ + KVM_STATS_BASE_POW10, -6) +/* Cumulative time in nanosecond */ +#define STATS_DESC_TIME_NSEC(SCOPE, name) \ + STATS_DESC_CUMULATIVE(SCOPE, name, KVM_STATS_UNIT_SECONDS, \ + KVM_STATS_BASE_POW10, -9) + +/* Instantaneous time in second */ +#define STATS_DESC_ITIME_SEC(SCOPE, name) \ + STATS_DESC_INSTANT(SCOPE, name, KVM_STATS_UNIT_SECONDS, \ + KVM_STATS_BASE_POW10, 0) +/* Instantaneous time in millisecond */ +#define STATS_DESC_ITIME_MSEC(SCOPE, name) \ + STATS_DESC_INSTANT(SCOPE, name, KVM_STATS_UNIT_SECONDS, \ + KVM_STATS_BASE_POW10, -3) +/* Instantaneous time in microsecond */ +#define STATS_DESC_ITIME_USEC(SCOPE, name) \ + STATS_DESC_INSTANT(SCOPE, name, KVM_STATS_UNIT_SECONDS, \ + KVM_STATS_BASE_POW10, -6) +/* Instantaneous time in nanosecond */ +#define STATS_DESC_ITIME_NSEC(SCOPE, name) \ + STATS_DESC_INSTANT(SCOPE, name, KVM_STATS_UNIT_SECONDS, \ + KVM_STATS_BASE_POW10, -9) + +#define KVM_GENERIC_VM_STATS() \ + STATS_DESC_COUNTER(VM_GENERIC, remote_tlb_flush) + +#define KVM_GENERIC_VCPU_STATS() \ + STATS_DESC_COUNTER(VCPU_GENERIC, halt_successful_poll), \ + STATS_DESC_COUNTER(VCPU_GENERIC, halt_attempted_poll), \ + STATS_DESC_COUNTER(VCPU_GENERIC, halt_poll_invalid), \ + STATS_DESC_COUNTER(VCPU_GENERIC, halt_wakeup), \ + STATS_DESC_TIME_NSEC(VCPU_GENERIC, halt_poll_success_ns), \ + STATS_DESC_TIME_NSEC(VCPU_GENERIC, halt_poll_fail_ns) + extern struct kvm_stats_debugfs_item debugfs_entries[]; extern struct dentry *kvm_debugfs_dir; +extern struct _kvm_stats_header kvm_vm_stats_header; +extern struct _kvm_stats_header kvm_vcpu_stats_header; +extern struct _kvm_stats_desc kvm_vm_stats_desc[]; +extern struct _kvm_stats_desc kvm_vcpu_stats_desc[];
#if defined(CONFIG_MMU_NOTIFIER) && defined(KVM_ARCH_WANT_MMU_NOTIFIER) static inline int mmu_notifier_retry(struct kvm *kvm, unsigned long mmu_seq) diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 9febe1412f7a..d6e97b577d01 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -1086,6 +1086,7 @@ struct kvm_ppc_resize_hpt { #define KVM_CAP_HYPERV_ENFORCE_CPUID 199 #define KVM_CAP_SREGS2 200 #define KVM_CAP_EXIT_HYPERCALL 201 +#define KVM_CAP_BINARY_STATS_FD 202
#ifdef KVM_CAP_IRQ_ROUTING
@@ -1905,4 +1906,49 @@ struct kvm_dirty_gfn { #define KVM_BUS_LOCK_DETECTION_OFF (1 << 0) #define KVM_BUS_LOCK_DETECTION_EXIT (1 << 1)
+#define KVM_STATS_ID_MAXLEN 64 + +struct kvm_stats_header { + __u32 name_size; + __u32 count; + __u32 desc_offset; + __u32 data_offset; + char id[0]; +}; + +#define KVM_STATS_TYPE_SHIFT 0 +#define KVM_STATS_TYPE_MASK (0xF << KVM_STATS_TYPE_SHIFT) +#define KVM_STATS_TYPE_CUMULATIVE (0x0 << KVM_STATS_TYPE_SHIFT) +#define KVM_STATS_TYPE_INSTANT (0x1 << KVM_STATS_TYPE_SHIFT) +#define KVM_STATS_TYPE_MAX KVM_STATS_TYPE_INSTANT + +#define KVM_STATS_UNIT_SHIFT 4 +#define KVM_STATS_UNIT_MASK (0xF << KVM_STATS_UNIT_SHIFT) +#define KVM_STATS_UNIT_NONE (0x0 << KVM_STATS_UNIT_SHIFT) +#define KVM_STATS_UNIT_BYTES (0x1 << KVM_STATS_UNIT_SHIFT) +#define KVM_STATS_UNIT_SECONDS (0x2 << KVM_STATS_UNIT_SHIFT) +#define KVM_STATS_UNIT_CYCLES (0x3 << KVM_STATS_UNIT_SHIFT) +#define KVM_STATS_UNIT_MAX KVM_STATS_UNIT_CYCLES + +#define KVM_STATS_BASE_SHIFT 8 +#define KVM_STATS_BASE_MASK (0xF << KVM_STATS_BASE_SHIFT) +#define KVM_STATS_BASE_POW10 (0x0 << KVM_STATS_BASE_SHIFT) +#define KVM_STATS_BASE_POW2 (0x1 << KVM_STATS_BASE_SHIFT) +#define KVM_STATS_BASE_MAX KVM_STATS_BASE_POW2 + +struct kvm_stats_desc { + __u32 flags; + __s16 exponent; + __u16 size; + __u32 offset; + __u32 unused; + char name[0]; +}; + +struct kvm_stats_data { + __u64 value[0]; +}; + +#define KVM_GET_STATS_FD _IO(KVMIO, 0xce) + #endif /* __LINUX_KVM_H */ diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index cec986487b30..f5bea71ea61c 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -3497,6 +3497,103 @@ static int kvm_vcpu_ioctl_set_sigmask(struct kvm_vcpu *vcpu, sigset_t *sigset) return 0; }
+static ssize_t kvm_stats_read(struct _kvm_stats_header *header, + struct _kvm_stats_desc *desc, void *stats, size_t size_stats, + char __user *user_buffer, size_t size, loff_t *offset) +{ + ssize_t copylen, len, remain = size; + size_t size_header, size_desc; + loff_t pos = *offset; + char __user *dest = user_buffer; + void *src; + + size_header = sizeof(*header); + size_desc = header->header.count * sizeof(*desc); + + len = size_header + size_desc + size_stats - pos; + len = min(len, remain); + if (len <= 0) + return 0; + remain = len; + + /* Copy kvm stats header */ + copylen = size_header - pos; + copylen = min(copylen, remain); + if (copylen > 0) { + src = (void *)header + pos; + if (copy_to_user(dest, src, copylen)) + return -EFAULT; + remain -= copylen; + pos += copylen; + dest += copylen; + } + /* Copy kvm stats descriptors */ + copylen = header->header.desc_offset + size_desc - pos; + copylen = min(copylen, remain); + if (copylen > 0) { + src = (void *)desc + pos - header->header.desc_offset; + if (copy_to_user(dest, src, copylen)) + return -EFAULT; + remain -= copylen; + pos += copylen; + dest += copylen; + } + /* Copy kvm stats values */ + copylen = header->header.data_offset + size_stats - pos; + copylen = min(copylen, remain); + if (copylen > 0) { + src = stats + pos - header->header.data_offset; + if (copy_to_user(dest, src, copylen)) + return -EFAULT; + remain -= copylen; + pos += copylen; + dest += copylen; + } + + *offset = pos; + return len; +} + +static ssize_t kvm_vcpu_stats_read(struct file *file, char __user *user_buffer, + size_t size, loff_t *offset) +{ + struct kvm_vcpu *vcpu = file->private_data; + + snprintf(&kvm_vcpu_stats_header.id[0], sizeof(kvm_vcpu_stats_header.id), + "kvm-%d/vcpu-%d", task_pid_nr(current), vcpu->vcpu_id); + return kvm_stats_read(&kvm_vcpu_stats_header, + &kvm_vcpu_stats_desc[0], &vcpu->stat, + sizeof(vcpu->stat), user_buffer, size, offset); +} + +static const struct file_operations kvm_vcpu_stats_fops = { + .read = kvm_vcpu_stats_read, + .llseek = noop_llseek, +}; + +static int kvm_vcpu_ioctl_get_stats_fd(struct kvm_vcpu *vcpu) +{ + int fd; + struct file *file; + char name[15 + ITOA_MAX_LEN + 1]; + + snprintf(name, sizeof(name), "kvm-vcpu-stats:%d", vcpu->vcpu_id); + + fd = get_unused_fd_flags(O_CLOEXEC); + if (fd < 0) + return fd; + + file = anon_inode_getfile(name, &kvm_vcpu_stats_fops, vcpu, O_RDONLY); + if (IS_ERR(file)) { + put_unused_fd(fd); + return PTR_ERR(file); + } + file->f_mode |= FMODE_PREAD; + fd_install(fd, file); + + return fd; +} + static long kvm_vcpu_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { @@ -3694,6 +3791,10 @@ static long kvm_vcpu_ioctl(struct file *filp, r = kvm_arch_vcpu_ioctl_set_fpu(vcpu, fpu); break; } + case KVM_GET_STATS_FD: { + r = kvm_vcpu_ioctl_get_stats_fd(vcpu); + break; + } default: r = kvm_arch_vcpu_ioctl(filp, ioctl, arg); } @@ -3952,6 +4053,8 @@ static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg) #else return 0; #endif + case KVM_CAP_BINARY_STATS_FD: + return 1; default: break; } @@ -4055,6 +4158,43 @@ static int kvm_vm_ioctl_enable_cap_generic(struct kvm *kvm, } }
+static ssize_t kvm_vm_stats_read(struct file *file, char __user *user_buffer, + size_t size, loff_t *offset) +{ + struct kvm *kvm = file->private_data; + + snprintf(&kvm_vm_stats_header.id[0], sizeof(kvm_vm_stats_header.id), + "kvm-%d", task_pid_nr(current)); + return kvm_stats_read(&kvm_vm_stats_header, &kvm_vm_stats_desc[0], + &kvm->stat, sizeof(kvm->stat), user_buffer, size, offset); +} + +static const struct file_operations kvm_vm_stats_fops = { + .read = kvm_vm_stats_read, + .llseek = noop_llseek, +}; + +static int kvm_vm_ioctl_get_stats_fd(struct kvm *kvm) +{ + int fd; + struct file *file; + + fd = get_unused_fd_flags(O_CLOEXEC); + if (fd < 0) + return fd; + + file = anon_inode_getfile("kvm-vm-stats", + &kvm_vm_stats_fops, kvm, O_RDONLY); + if (IS_ERR(file)) { + put_unused_fd(fd); + return PTR_ERR(file); + } + file->f_mode |= FMODE_PREAD; + fd_install(fd, file); + + return fd; +} + static long kvm_vm_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { @@ -4237,6 +4377,9 @@ static long kvm_vm_ioctl(struct file *filp, case KVM_RESET_DIRTY_RINGS: r = kvm_vm_ioctl_reset_dirty_pages(kvm); break; + case KVM_GET_STATS_FD: + r = kvm_vm_ioctl_get_stats_fd(kvm); + break; default: r = kvm_arch_vm_ioctl(filp, ioctl, arg); }
Update KVM API documentation for binary statistics.
Reviewed-by: David Matlack dmatlack@google.com Reviewed-by: Ricardo Koller ricarkol@google.com Reviewed-by: Krish Sadhukhan krish.sadhukhan@oracle.com Reviewed-by: Fuad Tabba tabba@google.com Signed-off-by: Jing Zhang jingzhangos@google.com --- Documentation/virt/kvm/api.rst | 177 ++++++++++++++++++++++++++++++++- 1 file changed, 176 insertions(+), 1 deletion(-)
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index e328caa35d6c..35ee52dbec89 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -5034,7 +5034,6 @@ see KVM_XEN_VCPU_SET_ATTR above. The KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_ADJUST type may not be used with the KVM_XEN_VCPU_GET_ATTR ioctl.
- 4.131 KVM_GET_SREGS2 ------------------
@@ -5081,6 +5080,174 @@ Writes special registers into the vcpu. See KVM_GET_SREGS2 for the data structures. This ioctl (when supported) replaces the KVM_SET_SREGS.
+4.133 KVM_GET_STATS_FD +---------------------- + +:Capability: KVM_CAP_STATS_BINARY_FD +:Architectures: all +:Type: vm ioctl, vcpu ioctl +:Parameters: none +:Returns: statistics file descriptor on success, < 0 on error + +Errors: + + ====== ====================================================== + ENOMEM if the fd could not be created due to lack of memory + EMFILE if the number of opened files exceeds the limit + ====== ====================================================== + +The file descriptor can be used to read VM/vCPU statistics data in binary +format. The file data is organized into three blocks as below: ++-------------+ +| Header | ++-------------+ +| Descriptors | ++-------------+ +| Stats Data | ++-------------+ + +The Header block is always at the start of the file. It is only needed to be +read one time for the lifetime of the file descriptor. +It is in the form of ``struct kvm_stats_header`` as below:: + + #define KVM_STATS_ID_MAXLEN 64 + + struct kvm_stats_header { + __u32 name_size; + __u32 count; + __u32 desc_offset; + __u32 data_offset; + char id[0]; + }; + +The ``id`` field is identification for the corresponding KVM statistics. For +VM statistics, it is in the form of "kvm-{kvm pid}", like "kvm-12345". For +VCPU statistics, it is in the form of "kvm-{kvm pid}/vcpu-{vcpu id}", like +"kvm-12345/vcpu-12". + +The ``name_size`` field is the size (byte) of the statistics name string +(including trailing '\0') appended to the end of every statistics descriptor. + +The ``count`` field is the number of statistics. + +The ``desc_offset`` field is the offset of the Descriptors block from the start +of the file indicated by the file descriptor. + +The ``data_offset`` field is the offset of the Stats Data block from the start +of the file indicated by the file descriptor. + +The Descriptors block is only needed to be read once for the lifetime of the +file descriptor. It is an array of ``struct kvm_stats_desc`` as shown in +below code block:: + + #define KVM_STATS_TYPE_SHIFT 0 + #define KVM_STATS_TYPE_MASK (0xF << KVM_STATS_TYPE_SHIFT) + #define KVM_STATS_TYPE_CUMULATIVE (0x0 << KVM_STATS_TYPE_SHIFT) + #define KVM_STATS_TYPE_INSTANT (0x1 << KVM_STATS_TYPE_SHIFT) + #define KVM_STATS_TYPE_MAX KVM_STATS_TYPE_INSTANT + + #define KVM_STATS_UNIT_SHIFT 4 + #define KVM_STATS_UNIT_MASK (0xF << KVM_STATS_UNIT_SHIFT) + #define KVM_STATS_UNIT_NONE (0x0 << KVM_STATS_UNIT_SHIFT) + #define KVM_STATS_UNIT_BYTES (0x1 << KVM_STATS_UNIT_SHIFT) + #define KVM_STATS_UNIT_SECONDS (0x2 << KVM_STATS_UNIT_SHIFT) + #define KVM_STATS_UNIT_CYCLES (0x3 << KVM_STATS_UNIT_SHIFT) + #define KVM_STATS_UNIT_MAX KVM_STATS_UNIT_CYCLES + + #define KVM_STATS_BASE_SHIFT 8 + #define KVM_STATS_BASE_MASK (0xF << KVM_STATS_BASE_SHIFT) + #define KVM_STATS_BASE_POW10 (0x0 << KVM_STATS_BASE_SHIFT) + #define KVM_STATS_BASE_POW2 (0x1 << KVM_STATS_BASE_SHIFT) + #define KVM_STATS_BASE_MAX KVM_STATS_BASE_POW2 + + struct kvm_stats_desc { + __u32 flags; + __s16 exponent; + __u16 size; + __u32 offset; + __u32 unused; + char name[0]; + }; + +The ``flags`` field contains the type and unit of the statistics data described +by this descriptor. The following flags are supported: + +Bits 0-3 of ``flags`` encode the type: + * ``KVM_STATS_TYPE_CUMULATIVE`` + The statistics data is cumulative. The value of data can only be increased. + Most of the counters used in KVM are of this type. + The corresponding ``count`` field for this type is always 1. + * ``KVM_STATS_TYPE_INSTANT`` + The statistics data is instantaneous. Its value can be increased or + decreased. This type is usually used as a measurement of some resources, + like the number of dirty pages, the number of large pages, etc. + The corresponding ``count`` field for this type is always 1. + +Bits 4-7 of ``flags`` encode the unit: + * ``KVM_STATS_UNIT_NONE`` + There is no unit for the value of statistics data. This usually means that + the value is a simple counter of an event. + * ``KVM_STATS_UNIT_BYTES`` + It indicates that the statistics data is used to measure memory size, in the + unit of Byte, KiByte, MiByte, GiByte, etc. The unit of the data is + determined by the ``exponent`` field in the descriptor. The + ``KVM_STATS_BASE_POW2`` flag is valid in this case. The unit of the data is + determined by ``pow(2, exponent)``. For example, if value is 10, + ``exponent`` is 20, which means the unit of statistics data is MiByte, we + can get the statistics data in the unit of Byte by + ``value * pow(2, exponent) = 10 * pow(2, 20) = 10 MiByte`` which is + 10 * 1024 * 1024 Bytes. + * ``KVM_STATS_UNIT_SECONDS`` + It indicates that the statistics data is used to measure time/latency, in + the unit of nanosecond, microsecond, millisecond and second. The unit of the + data is determined by the ``exponent`` field in the descriptor. The + ``KVM_STATS_BASE_POW10`` flag is valid in this case. The unit of the data + is determined by ``pow(10, exponent)``. For example, if value is 2000000, + ``exponent`` is -6, which means the unit of statistics data is microsecond, + we can get the statistics data in the unit of second by + ``value * pow(10, exponent) = 2000000 * pow(10, -6) = 2 seconds``. + * ``KVM_STATS_UNIT_CYCLES`` + It indicates that the statistics data is used to measure CPU clock cycles. + The ``KVM_STATS_BASE_POW10`` flag is valid in this case. For example, if + value is 200, ``exponent`` is 4, we can get the number of CPU clock cycles + by ``value * pow(10, exponent) = 200 * pow(10, 4) = 2000000``. + +Bits 8-11 of ``flags`` encode the base: + * ``KVM_STATS_BASE_POW10`` + The scale is based on power of 10. It is used for measurement of time and + CPU clock cycles. + * ``KVM_STATS_BASE_POW2`` + The scale is based on power of 2. It is used for measurement of memory size. + +The ``exponent`` field is the scale of corresponding statistics data. For +example, if the unit is ``KVM_STATS_UNIT_BYTES``, the base is +``KVM_STATS_BASE_POW2``, the ``exponent`` is 10, then we know that the real +unit of the statistics data is KBytes a.k.a pow(2, 10) = 1024 bytes. + +The ``size`` field is the number of values (u64) of this statistics data. Its +value is usually 1 for most of simple statistics. + +The ``offset`` field is the offset from the start of Data Block to the start of +the corresponding statistics data. + +The ``unused`` fields are reserved for future support for other types of +statistics data, like log/linear histogram. + +The ``name`` field points to the name string of the statistics data. The name +string starts at the end of ``struct kvm_stats_desc``. +The maximum length (including trailing '\0') is indicated by ``name_size`` +in ``struct kvm_stats_header``. + +The Stats Data block contains an array of data values of type ``struct +kvm_vm_stats_data`` or ``struct kvm_vcpu_stats_data``. It would be read by +userspace periodically to pull statistics data. +The order of data value in Stats Data block is the same as the order of +descriptors in Descriptors block. + * Statistics data for VM/VCPU:: + + struct kvm_stats_data { + __u64 value[0]; + };
5. The kvm_run structure ======================== @@ -6969,3 +7136,11 @@ The argument to KVM_ENABLE_CAP is also a bitmask, and must be a subset of the result of KVM_CHECK_EXTENSION. KVM will forward to userspace the hypercalls whose corresponding bit is in the argument, and return ENOSYS for the others. + +8.35 KVM_CAP_STATS_BINARY_FD +---------------------------- + +:Architectures: all + +This capability indicates the feature that userspace can create get a file +descriptor for every VM and VCPU to read statistics data in binary format.
Add selftest to check KVM stats descriptors validity.
Reviewed-by: David Matlack dmatlack@google.com Reviewed-by: Ricardo Koller ricarkol@google.com Reviewed-by: Krish Sadhukhan krish.sadhukhan@oracle.com Signed-off-by: Jing Zhang jingzhangos@google.com --- tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile | 3 + .../testing/selftests/kvm/include/kvm_util.h | 3 + .../selftests/kvm/kvm_binary_stats_test.c | 225 ++++++++++++++++++ tools/testing/selftests/kvm/lib/kvm_util.c | 12 + 5 files changed, 244 insertions(+) create mode 100644 tools/testing/selftests/kvm/kvm_binary_stats_test.c
diff --git a/tools/testing/selftests/kvm/.gitignore b/tools/testing/selftests/kvm/.gitignore index e0e14150744e..ebe81d67c738 100644 --- a/tools/testing/selftests/kvm/.gitignore +++ b/tools/testing/selftests/kvm/.gitignore @@ -46,3 +46,4 @@ /memslot_perf_test /set_memory_region_test /steal_time +/kvm_binary_stats_test diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests/kvm/Makefile index 61e2accd080d..8e4eaf8cdc06 100644 --- a/tools/testing/selftests/kvm/Makefile +++ b/tools/testing/selftests/kvm/Makefile @@ -79,6 +79,7 @@ TEST_GEN_PROGS_x86_64 += memslot_modification_stress_test TEST_GEN_PROGS_x86_64 += memslot_perf_test TEST_GEN_PROGS_x86_64 += set_memory_region_test TEST_GEN_PROGS_x86_64 += steal_time +TEST_GEN_PROGS_x86_64 += kvm_binary_stats_test
TEST_GEN_PROGS_aarch64 += aarch64/get-reg-list TEST_GEN_PROGS_aarch64 += aarch64/get-reg-list-sve @@ -90,6 +91,7 @@ TEST_GEN_PROGS_aarch64 += kvm_create_max_vcpus TEST_GEN_PROGS_aarch64 += kvm_page_table_test TEST_GEN_PROGS_aarch64 += set_memory_region_test TEST_GEN_PROGS_aarch64 += steal_time +TEST_GEN_PROGS_aarch64 += kvm_binary_stats_test
TEST_GEN_PROGS_s390x = s390x/memop TEST_GEN_PROGS_s390x += s390x/resets @@ -99,6 +101,7 @@ TEST_GEN_PROGS_s390x += dirty_log_test TEST_GEN_PROGS_s390x += kvm_create_max_vcpus TEST_GEN_PROGS_s390x += kvm_page_table_test TEST_GEN_PROGS_s390x += set_memory_region_test +TEST_GEN_PROGS_s390x += kvm_binary_stats_test
TEST_GEN_PROGS += $(TEST_GEN_PROGS_$(UNAME_M)) LIBKVM += $(LIBKVM_$(UNAME_M)) diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h index 35739567189e..558b450f3d7c 100644 --- a/tools/testing/selftests/kvm/include/kvm_util.h +++ b/tools/testing/selftests/kvm/include/kvm_util.h @@ -391,4 +391,7 @@ uint64_t get_ucall(struct kvm_vm *vm, uint32_t vcpu_id, struct ucall *uc); #define GUEST_ASSERT_4(_condition, arg1, arg2, arg3, arg4) \ __GUEST_ASSERT((_condition), 4, (arg1), (arg2), (arg3), (arg4))
+int vm_get_stats_fd(struct kvm_vm *vm); +int vcpu_get_stats_fd(struct kvm_vm *vm, uint32_t vcpuid); + #endif /* SELFTEST_KVM_UTIL_H */ diff --git a/tools/testing/selftests/kvm/kvm_binary_stats_test.c b/tools/testing/selftests/kvm/kvm_binary_stats_test.c new file mode 100644 index 000000000000..f20870ae9296 --- /dev/null +++ b/tools/testing/selftests/kvm/kvm_binary_stats_test.c @@ -0,0 +1,225 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * kvm_binary_stats_test + * + * Copyright (C) 2021, Google LLC. + * + * Test the fd-based interface for KVM statistics. + */ + +#define _GNU_SOURCE /* for program_invocation_short_name */ +#include <fcntl.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <errno.h> + +#include "test_util.h" + +#include "kvm_util.h" +#include "asm/kvm.h" +#include "linux/kvm.h" + +static void stats_test(int stats_fd) +{ + ssize_t ret; + int i; + size_t size_desc, size_data = 0; + struct kvm_stats_header *header; + struct kvm_stats_desc *stats_desc, *pdesc; + struct kvm_stats_data *stats_data; + size_t size_header = sizeof(*header) + KVM_STATS_ID_MAXLEN; + + /* Read kvm stats header */ + header = malloc(size_header); + TEST_ASSERT(header, "Allocate memory for stats header"); + + ret = read(stats_fd, header, size_header); + TEST_ASSERT(ret == size_header, "Read stats header"); + size_desc = sizeof(*stats_desc) + header->name_size; + + /* Check id string in header, that should start with "kvm" */ + TEST_ASSERT(!strncmp(header->id, "kvm", 3) && + strlen(header->id) < KVM_STATS_ID_MAXLEN, + "Invalid KVM stats type, id: %s", header->id); + + /* Sanity check for other fields in header */ + if (header->count == 0) { + printf("No KVM stats defined!"); + return; + } + /* Check overlap */ + TEST_ASSERT(header->desc_offset > 0 && header->data_offset > 0 + && header->desc_offset >= size_header + && header->data_offset >= size_header, + "Invalid offset fields in header"); + TEST_ASSERT(header->desc_offset > header->data_offset + || (header->desc_offset + size_desc * header->count <= + header->data_offset), + "Descriptor block is overlapped with data block"); + + /* Allocate memory for stats descriptors */ + stats_desc = calloc(header->count, size_desc); + TEST_ASSERT(stats_desc, "Allocate memory for stats descriptors"); + /* Read kvm stats descriptors */ + ret = pread(stats_fd, stats_desc, + size_desc * header->count, header->desc_offset); + TEST_ASSERT(ret == size_desc * header->count, + "Read KVM stats descriptors"); + + /* Sanity check for fields in descriptors */ + for (i = 0; i < header->count; ++i) { + pdesc = (void *)stats_desc + i * size_desc; + /* Check type,unit,base boundaries */ + TEST_ASSERT((pdesc->flags & KVM_STATS_TYPE_MASK) + <= KVM_STATS_TYPE_MAX, "Unknown KVM stats type"); + TEST_ASSERT((pdesc->flags & KVM_STATS_UNIT_MASK) + <= KVM_STATS_UNIT_MAX, "Unknown KVM stats unit"); + TEST_ASSERT((pdesc->flags & KVM_STATS_BASE_MASK) + <= KVM_STATS_BASE_MAX, "Unknown KVM stats base"); + /* Check exponent for stats unit + * Exponent for counter should be greater than or equal to 0 + * Exponent for unit bytes should be greater than or equal to 0 + * Exponent for unit seconds should be less than or equal to 0 + * Exponent for unit clock cycles should be greater than or + * equal to 0 + */ + switch (pdesc->flags & KVM_STATS_UNIT_MASK) { + case KVM_STATS_UNIT_NONE: + case KVM_STATS_UNIT_BYTES: + case KVM_STATS_UNIT_CYCLES: + TEST_ASSERT(pdesc->exponent >= 0, + "Unsupported KVM stats unit"); + break; + case KVM_STATS_UNIT_SECONDS: + TEST_ASSERT(pdesc->exponent <= 0, + "Unsupported KVM stats unit"); + break; + } + /* Check name string */ + TEST_ASSERT(strlen(pdesc->name) < header->name_size, + "KVM stats name(%s) too long", pdesc->name); + /* Check size field, which should not be zero */ + TEST_ASSERT(pdesc->size, "KVM descriptor(%s) with size of 0", + pdesc->name); + size_data += pdesc->size * sizeof(stats_data->value[0]); + } + /* Check overlap */ + TEST_ASSERT(header->data_offset >= header->desc_offset + || header->data_offset + size_data <= header->desc_offset, + "Data block is overlapped with Descriptor block"); + /* Check validity of all stats data size */ + TEST_ASSERT(size_data >= header->count * sizeof(stats_data->value[0]), + "Data size is not correct"); + /* Check stats offset */ + for (i = 0; i < header->count; ++i) { + pdesc = (void *)stats_desc + i * size_desc; + TEST_ASSERT(pdesc->offset < size_data, + "Invalid offset (%u) for stats: %s", + pdesc->offset, pdesc->name); + } + + /* Allocate memory for stats data */ + stats_data = malloc(size_data); + TEST_ASSERT(stats_data, "Allocate memory for stats data"); + /* Read kvm stats data as a bulk */ + ret = pread(stats_fd, stats_data, size_data, header->data_offset); + TEST_ASSERT(ret == size_data, "Read KVM stats data"); + /* Read kvm stats data one by one */ + size_data = 0; + for (i = 0; i < header->count; ++i) { + pdesc = (void *)stats_desc + i * size_desc; + ret = pread(stats_fd, stats_data, + pdesc->size * sizeof(stats_data->value[0]), + header->data_offset + size_data); + TEST_ASSERT(ret == pdesc->size * sizeof(stats_data->value[0]), + "Read data of KVM stats: %s", pdesc->name); + size_data += pdesc->size * sizeof(stats_data->value[0]); + } + + free(stats_data); + free(stats_desc); +} + + +static void vm_stats_test(struct kvm_vm *vm) +{ + int stats_fd; + + /* Get fd for VM stats */ + stats_fd = vm_get_stats_fd(vm); + TEST_ASSERT(stats_fd >= 0, "Get VM stats fd"); + + stats_test(stats_fd); + close(stats_fd); + TEST_ASSERT(fcntl(stats_fd, F_GETFD) == -1, "Stats fd not freed"); +} + +static void vcpu_stats_test(struct kvm_vm *vm, int vcpu_id) +{ + int stats_fd; + + /* Get fd for VCPU stats */ + stats_fd = vcpu_get_stats_fd(vm, vcpu_id); + TEST_ASSERT(stats_fd >= 0, "Get VCPU stats fd"); + + stats_test(stats_fd); + close(stats_fd); + TEST_ASSERT(fcntl(stats_fd, F_GETFD) == -1, "Stats fd not freed"); +} + +#define DEFAULT_NUM_VM 4 +#define DEFAULT_NUM_VCPU 4 + +/* + * Usage: kvm_bin_form_stats [#vm] [#vcpu] + * The first parameter #vm set the number of VMs being created. + * The second parameter #vcpu set the number of VCPUs being created. + * By default, DEFAULT_NUM_VM VM and DEFAULT_NUM_VCPU VCPU for the VM would be + * created for testing. + */ + +int main(int argc, char *argv[]) +{ + int max_vm = DEFAULT_NUM_VM, max_vcpu = DEFAULT_NUM_VCPU, ret, i, j; + struct kvm_vm **vms; + + /* Get the number of VMs and VCPUs that would be created for testing. */ + if (argc > 1) { + max_vm = strtol(argv[1], NULL, 0); + if (max_vm <= 0) + max_vm = DEFAULT_NUM_VM; + } + if (argc > 2) { + max_vcpu = strtol(argv[2], NULL, 0); + if (max_vcpu <= 0) + max_vcpu = DEFAULT_NUM_VCPU; + } + + /* Check the extension for binary stats */ + ret = kvm_check_cap(KVM_CAP_BINARY_STATS_FD); + TEST_ASSERT(ret >= 0, + "Binary form statistics interface is not supported"); + + /* Create VMs and VCPUs */ + vms = malloc(sizeof(vms[0]) * max_vm); + TEST_ASSERT(vms, "Allocate memory for storing VM pointers"); + for (i = 0; i < max_vm; ++i) { + vms[i] = vm_create(VM_MODE_DEFAULT, + DEFAULT_GUEST_PHY_PAGES, O_RDWR); + for (j = 0; j < max_vcpu; ++j) + vm_vcpu_add(vms[i], j); + } + + /* Check stats read for every VM and VCPU */ + for (i = 0; i < max_vm; ++i) { + vm_stats_test(vms[i]); + for (j = 0; j < max_vcpu; ++j) + vcpu_stats_test(vms[i], j); + } + + for (i = 0; i < max_vm; ++i) + kvm_vm_free(vms[i]); + free(vms); + return 0; +} diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index 5c70596dd1b9..83c02cb0ae1e 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -2286,3 +2286,15 @@ unsigned int vm_calc_num_guest_pages(enum vm_guest_mode mode, size_t size) n = DIV_ROUND_UP(size, vm_guest_mode_params[mode].page_size); return vm_adjust_num_guest_pages(mode, n); } + +int vm_get_stats_fd(struct kvm_vm *vm) +{ + return ioctl(vm->fd, KVM_GET_STATS_FD, NULL); +} + +int vcpu_get_stats_fd(struct kvm_vm *vm, uint32_t vcpuid) +{ + struct vcpu *vcpu = vcpu_find(vm, vcpuid); + + return ioctl(vcpu->fd, KVM_GET_STATS_FD, NULL); +}
To remove code duplication, use the binary stats descriptors for debugfs interface for KVM stats. Then we only have one stats definitions for both binary and debugfs interface.
Signed-off-by: Jing Zhang jingzhangos@google.com --- arch/arm64/kvm/guest.c | 16 ------ arch/mips/kvm/mips.c | 39 -------------- arch/powerpc/kvm/book3s.c | 33 ------------ arch/powerpc/kvm/booke.c | 25 --------- arch/s390/kvm/kvm-s390.c | 108 -------------------------------------- arch/x86/kvm/x86.c | 49 +---------------- include/linux/kvm_host.h | 43 +++++---------- include/uapi/linux/kvm.h | 4 +- virt/kvm/kvm_main.c | 104 ++++++++++++++++++++++++++---------- 9 files changed, 94 insertions(+), 327 deletions(-)
diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 99842e29c2de..59486a9fef1a 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -66,22 +66,6 @@ struct _kvm_stats_header kvm_vcpu_stats_header = { } };
-struct kvm_stats_debugfs_item debugfs_entries[] = { - VCPU_STAT_GENERIC("halt_successful_poll", halt_successful_poll), - VCPU_STAT_GENERIC("halt_attempted_poll", halt_attempted_poll), - VCPU_STAT_GENERIC("halt_poll_invalid", halt_poll_invalid), - VCPU_STAT_GENERIC("halt_wakeup", halt_wakeup), - VCPU_STAT("hvc_exit_stat", hvc_exit_stat), - VCPU_STAT("wfe_exit_stat", wfe_exit_stat), - VCPU_STAT("wfi_exit_stat", wfi_exit_stat), - VCPU_STAT("mmio_exit_user", mmio_exit_user), - VCPU_STAT("mmio_exit_kernel", mmio_exit_kernel), - VCPU_STAT("exits", exits), - VCPU_STAT_GENERIC("halt_poll_success_ns", halt_poll_success_ns), - VCPU_STAT_GENERIC("halt_poll_fail_ns", halt_poll_fail_ns), - { NULL } -}; - static bool core_reg_offset_is_vreg(u64 off) { return off >= KVM_REG_ARM_CORE_REG(fp_regs.vregs) && diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c index 67404f0947aa..606fe3b47075 100644 --- a/arch/mips/kvm/mips.c +++ b/arch/mips/kvm/mips.c @@ -99,45 +99,6 @@ struct _kvm_stats_header kvm_vcpu_stats_header = { } };
-struct kvm_stats_debugfs_item debugfs_entries[] = { - VCPU_STAT("wait", wait_exits), - VCPU_STAT("cache", cache_exits), - VCPU_STAT("signal", signal_exits), - VCPU_STAT("interrupt", int_exits), - VCPU_STAT("cop_unusable", cop_unusable_exits), - VCPU_STAT("tlbmod", tlbmod_exits), - VCPU_STAT("tlbmiss_ld", tlbmiss_ld_exits), - VCPU_STAT("tlbmiss_st", tlbmiss_st_exits), - VCPU_STAT("addrerr_st", addrerr_st_exits), - VCPU_STAT("addrerr_ld", addrerr_ld_exits), - VCPU_STAT("syscall", syscall_exits), - VCPU_STAT("resvd_inst", resvd_inst_exits), - VCPU_STAT("break_inst", break_inst_exits), - VCPU_STAT("trap_inst", trap_inst_exits), - VCPU_STAT("msa_fpe", msa_fpe_exits), - VCPU_STAT("fpe", fpe_exits), - VCPU_STAT("msa_disabled", msa_disabled_exits), - VCPU_STAT("flush_dcache", flush_dcache_exits), - VCPU_STAT("vz_gpsi", vz_gpsi_exits), - VCPU_STAT("vz_gsfc", vz_gsfc_exits), - VCPU_STAT("vz_hc", vz_hc_exits), - VCPU_STAT("vz_grr", vz_grr_exits), - VCPU_STAT("vz_gva", vz_gva_exits), - VCPU_STAT("vz_ghfc", vz_ghfc_exits), - VCPU_STAT("vz_gpa", vz_gpa_exits), - VCPU_STAT("vz_resvd", vz_resvd_exits), -#ifdef CONFIG_CPU_LOONGSON64 - VCPU_STAT("vz_cpucfg", vz_cpucfg_exits), -#endif - VCPU_STAT_GENERIC("halt_successful_poll", halt_successful_poll), - VCPU_STAT_GENERIC("halt_attempted_poll", halt_attempted_poll), - VCPU_STAT_GENERIC("halt_poll_invalid", halt_poll_invalid), - VCPU_STAT_GENERIC("halt_wakeup", halt_wakeup), - VCPU_STAT_GENERIC("halt_poll_success_ns", halt_poll_success_ns), - VCPU_STAT_GENERIC("halt_poll_fail_ns", halt_poll_fail_ns), - {NULL} -}; - bool kvm_trace_guest_mode_change;
int kvm_guest_mode_change_trace_reg(void) diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c index 15436484e521..0c5bc4bf488c 100644 --- a/arch/powerpc/kvm/book3s.c +++ b/arch/powerpc/kvm/book3s.c @@ -102,39 +102,6 @@ struct _kvm_stats_header kvm_vcpu_stats_header = { } };
-struct kvm_stats_debugfs_item debugfs_entries[] = { - VCPU_STAT("exits", sum_exits), - VCPU_STAT("mmio", mmio_exits), - VCPU_STAT("sig", signal_exits), - VCPU_STAT("sysc", syscall_exits), - VCPU_STAT("inst_emu", emulated_inst_exits), - VCPU_STAT("dec", dec_exits), - VCPU_STAT("ext_intr", ext_intr_exits), - VCPU_STAT("queue_intr", queue_intr), - VCPU_STAT_GENERIC("halt_poll_success_ns", halt_poll_success_ns), - VCPU_STAT_GENERIC("halt_poll_fail_ns", halt_poll_fail_ns), - VCPU_STAT("halt_wait_ns", halt_wait_ns), - VCPU_STAT_GENERIC("halt_successful_poll", halt_successful_poll), - VCPU_STAT_GENERIC("halt_attempted_poll", halt_attempted_poll), - VCPU_STAT("halt_successful_wait", halt_successful_wait), - VCPU_STAT_GENERIC("halt_poll_invalid", halt_poll_invalid), - VCPU_STAT_GENERIC("halt_wakeup", halt_wakeup), - VCPU_STAT("pf_storage", pf_storage), - VCPU_STAT("sp_storage", sp_storage), - VCPU_STAT("pf_instruc", pf_instruc), - VCPU_STAT("sp_instruc", sp_instruc), - VCPU_STAT("ld", ld), - VCPU_STAT("ld_slow", ld_slow), - VCPU_STAT("st", st), - VCPU_STAT("st_slow", st_slow), - VCPU_STAT("pthru_all", pthru_all), - VCPU_STAT("pthru_host", pthru_host), - VCPU_STAT("pthru_bad_aff", pthru_bad_aff), - VM_STAT("largepages_2M", num_2M_pages, .mode = 0444), - VM_STAT("largepages_1G", num_1G_pages, .mode = 0444), - { NULL } -}; - static inline void kvmppc_update_int_pending(struct kvm_vcpu *vcpu, unsigned long pending_now, unsigned long old_pending) { diff --git a/arch/powerpc/kvm/booke.c b/arch/powerpc/kvm/booke.c index fbc9b7b6af57..888885b574cf 100644 --- a/arch/powerpc/kvm/booke.c +++ b/arch/powerpc/kvm/booke.c @@ -93,31 +93,6 @@ struct _kvm_stats_header kvm_vcpu_stats_header = { } };
-struct kvm_stats_debugfs_item debugfs_entries[] = { - VCPU_STAT("mmio", mmio_exits), - VCPU_STAT("sig", signal_exits), - VCPU_STAT("itlb_r", itlb_real_miss_exits), - VCPU_STAT("itlb_v", itlb_virt_miss_exits), - VCPU_STAT("dtlb_r", dtlb_real_miss_exits), - VCPU_STAT("dtlb_v", dtlb_virt_miss_exits), - VCPU_STAT("sysc", syscall_exits), - VCPU_STAT("isi", isi_exits), - VCPU_STAT("dsi", dsi_exits), - VCPU_STAT("inst_emu", emulated_inst_exits), - VCPU_STAT("dec", dec_exits), - VCPU_STAT("ext_intr", ext_intr_exits), - VCPU_STAT_GENERIC("halt_successful_poll", halt_successful_poll), - VCPU_STAT_GENERIC("halt_attempted_poll", halt_attempted_poll), - VCPU_STAT_GENERIC("halt_poll_invalid", halt_poll_invalid), - VCPU_STAT_GENERIC("halt_wakeup", halt_wakeup), - VCPU_STAT("doorbell", dbell_exits), - VCPU_STAT("guest doorbell", gdbell_exits), - VCPU_STAT_GENERIC("halt_poll_success_ns", halt_poll_success_ns), - VCPU_STAT_GENERIC("halt_poll_fail_ns", halt_poll_fail_ns), - VM_STAT_GENERIC("remote_tlb_flush", remote_tlb_flush), - { NULL } -}; - /* TODO: use vcpu_printf() */ void kvmppc_dump_vcpu(struct kvm_vcpu *vcpu) { diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index ba50dd775fa8..50dab32cbef1 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -188,114 +188,6 @@ struct _kvm_stats_header kvm_vcpu_stats_header = { } };
-struct kvm_stats_debugfs_item debugfs_entries[] = { - VCPU_STAT("userspace_handled", exit_userspace), - VCPU_STAT("exit_null", exit_null), - VCPU_STAT("pfault_sync", pfault_sync), - VCPU_STAT("exit_validity", exit_validity), - VCPU_STAT("exit_stop_request", exit_stop_request), - VCPU_STAT("exit_external_request", exit_external_request), - VCPU_STAT("exit_io_request", exit_io_request), - VCPU_STAT("exit_external_interrupt", exit_external_interrupt), - VCPU_STAT("exit_instruction", exit_instruction), - VCPU_STAT("exit_pei", exit_pei), - VCPU_STAT("exit_program_interruption", exit_program_interruption), - VCPU_STAT("exit_instr_and_program_int", exit_instr_and_program), - VCPU_STAT("exit_operation_exception", exit_operation_exception), - VCPU_STAT_GENERIC("halt_successful_poll", halt_successful_poll), - VCPU_STAT_GENERIC("halt_attempted_poll", halt_attempted_poll), - VCPU_STAT_GENERIC("halt_poll_invalid", halt_poll_invalid), - VCPU_STAT("halt_no_poll_steal", halt_no_poll_steal), - VCPU_STAT_GENERIC("halt_wakeup", halt_wakeup), - VCPU_STAT_GENERIC("halt_poll_success_ns", halt_poll_success_ns), - VCPU_STAT_GENERIC("halt_poll_fail_ns", halt_poll_fail_ns), - VCPU_STAT("instruction_lctlg", instruction_lctlg), - VCPU_STAT("instruction_lctl", instruction_lctl), - VCPU_STAT("instruction_stctl", instruction_stctl), - VCPU_STAT("instruction_stctg", instruction_stctg), - VCPU_STAT("deliver_ckc", deliver_ckc), - VCPU_STAT("deliver_cputm", deliver_cputm), - VCPU_STAT("deliver_emergency_signal", deliver_emergency_signal), - VCPU_STAT("deliver_external_call", deliver_external_call), - VCPU_STAT("deliver_service_signal", deliver_service_signal), - VCPU_STAT("deliver_virtio", deliver_virtio), - VCPU_STAT("deliver_stop_signal", deliver_stop_signal), - VCPU_STAT("deliver_prefix_signal", deliver_prefix_signal), - VCPU_STAT("deliver_restart_signal", deliver_restart_signal), - VCPU_STAT("deliver_program", deliver_program), - VCPU_STAT("deliver_io", deliver_io), - VCPU_STAT("deliver_machine_check", deliver_machine_check), - VCPU_STAT("exit_wait_state", exit_wait_state), - VCPU_STAT("inject_ckc", inject_ckc), - VCPU_STAT("inject_cputm", inject_cputm), - VCPU_STAT("inject_external_call", inject_external_call), - VM_STAT("inject_float_mchk", inject_float_mchk), - VCPU_STAT("inject_emergency_signal", inject_emergency_signal), - VM_STAT("inject_io", inject_io), - VCPU_STAT("inject_mchk", inject_mchk), - VM_STAT("inject_pfault_done", inject_pfault_done), - VCPU_STAT("inject_program", inject_program), - VCPU_STAT("inject_restart", inject_restart), - VM_STAT("inject_service_signal", inject_service_signal), - VCPU_STAT("inject_set_prefix", inject_set_prefix), - VCPU_STAT("inject_stop_signal", inject_stop_signal), - VCPU_STAT("inject_pfault_init", inject_pfault_init), - VM_STAT("inject_virtio", inject_virtio), - VCPU_STAT("instruction_epsw", instruction_epsw), - VCPU_STAT("instruction_gs", instruction_gs), - VCPU_STAT("instruction_io_other", instruction_io_other), - VCPU_STAT("instruction_lpsw", instruction_lpsw), - VCPU_STAT("instruction_lpswe", instruction_lpswe), - VCPU_STAT("instruction_pfmf", instruction_pfmf), - VCPU_STAT("instruction_ptff", instruction_ptff), - VCPU_STAT("instruction_stidp", instruction_stidp), - VCPU_STAT("instruction_sck", instruction_sck), - VCPU_STAT("instruction_sckpf", instruction_sckpf), - VCPU_STAT("instruction_spx", instruction_spx), - VCPU_STAT("instruction_stpx", instruction_stpx), - VCPU_STAT("instruction_stap", instruction_stap), - VCPU_STAT("instruction_iske", instruction_iske), - VCPU_STAT("instruction_ri", instruction_ri), - VCPU_STAT("instruction_rrbe", instruction_rrbe), - VCPU_STAT("instruction_sske", instruction_sske), - VCPU_STAT("instruction_ipte_interlock", instruction_ipte_interlock), - VCPU_STAT("instruction_essa", instruction_essa), - VCPU_STAT("instruction_stsi", instruction_stsi), - VCPU_STAT("instruction_stfl", instruction_stfl), - VCPU_STAT("instruction_tb", instruction_tb), - VCPU_STAT("instruction_tpi", instruction_tpi), - VCPU_STAT("instruction_tprot", instruction_tprot), - VCPU_STAT("instruction_tsch", instruction_tsch), - VCPU_STAT("instruction_sthyi", instruction_sthyi), - VCPU_STAT("instruction_sie", instruction_sie), - VCPU_STAT("instruction_sigp_sense", instruction_sigp_sense), - VCPU_STAT("instruction_sigp_sense_running", instruction_sigp_sense_running), - VCPU_STAT("instruction_sigp_external_call", instruction_sigp_external_call), - VCPU_STAT("instruction_sigp_emergency", instruction_sigp_emergency), - VCPU_STAT("instruction_sigp_cond_emergency", instruction_sigp_cond_emergency), - VCPU_STAT("instruction_sigp_start", instruction_sigp_start), - VCPU_STAT("instruction_sigp_stop", instruction_sigp_stop), - VCPU_STAT("instruction_sigp_stop_store_status", instruction_sigp_stop_store_status), - VCPU_STAT("instruction_sigp_store_status", instruction_sigp_store_status), - VCPU_STAT("instruction_sigp_store_adtl_status", instruction_sigp_store_adtl_status), - VCPU_STAT("instruction_sigp_set_arch", instruction_sigp_arch), - VCPU_STAT("instruction_sigp_set_prefix", instruction_sigp_prefix), - VCPU_STAT("instruction_sigp_restart", instruction_sigp_restart), - VCPU_STAT("instruction_sigp_cpu_reset", instruction_sigp_cpu_reset), - VCPU_STAT("instruction_sigp_init_cpu_reset", instruction_sigp_init_cpu_reset), - VCPU_STAT("instruction_sigp_unknown", instruction_sigp_unknown), - VCPU_STAT("instruction_diag_10", diagnose_10), - VCPU_STAT("instruction_diag_44", diagnose_44), - VCPU_STAT("instruction_diag_9c", diagnose_9c), - VCPU_STAT("diag_9c_ignored", diagnose_9c_ignored), - VCPU_STAT("diag_9c_forward", diagnose_9c_forward), - VCPU_STAT("instruction_diag_258", diagnose_258), - VCPU_STAT("instruction_diag_308", diagnose_308), - VCPU_STAT("instruction_diag_500", diagnose_500), - VCPU_STAT("instruction_diag_other", diagnose_other), - { NULL } -}; - /* allow nested virtualization in KVM (if enabled by user space) */ static int nested; module_param(nested, int, S_IRUGO); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 84438573b529..470a14c3c911 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -234,7 +234,7 @@ struct _kvm_stats_desc kvm_vm_stats_desc[] = { STATS_DESC_ICOUNTER(VM, mmu_unsync), STATS_DESC_ICOUNTER(VM, lpages), STATS_DESC_ICOUNTER(VM, nx_lpage_splits), - STATS_DESC_ICOUNTER(VM, max_mmu_page_hash_collisions) + STATS_DESC_PCOUNTER(VM, max_mmu_page_hash_collisions) }; static_assert(ARRAY_SIZE(kvm_vm_stats_desc) == sizeof(struct kvm_vm_stat) / sizeof(u64)); @@ -291,53 +291,6 @@ struct _kvm_stats_header kvm_vcpu_stats_header = { } };
-struct kvm_stats_debugfs_item debugfs_entries[] = { - VCPU_STAT("pf_fixed", pf_fixed), - VCPU_STAT("pf_guest", pf_guest), - VCPU_STAT("tlb_flush", tlb_flush), - VCPU_STAT("invlpg", invlpg), - VCPU_STAT("exits", exits), - VCPU_STAT("io_exits", io_exits), - VCPU_STAT("mmio_exits", mmio_exits), - VCPU_STAT("signal_exits", signal_exits), - VCPU_STAT("irq_window", irq_window_exits), - VCPU_STAT("nmi_window", nmi_window_exits), - VCPU_STAT("halt_exits", halt_exits), - VCPU_STAT_GENERIC("halt_successful_poll", halt_successful_poll), - VCPU_STAT_GENERIC("halt_attempted_poll", halt_attempted_poll), - VCPU_STAT_GENERIC("halt_poll_invalid", halt_poll_invalid), - VCPU_STAT_GENERIC("halt_wakeup", halt_wakeup), - VCPU_STAT("hypercalls", hypercalls), - VCPU_STAT("request_irq", request_irq_exits), - VCPU_STAT("irq_exits", irq_exits), - VCPU_STAT("host_state_reload", host_state_reload), - VCPU_STAT("fpu_reload", fpu_reload), - VCPU_STAT("insn_emulation", insn_emulation), - VCPU_STAT("insn_emulation_fail", insn_emulation_fail), - VCPU_STAT("irq_injections", irq_injections), - VCPU_STAT("nmi_injections", nmi_injections), - VCPU_STAT("req_event", req_event), - VCPU_STAT("l1d_flush", l1d_flush), - VCPU_STAT_GENERIC("halt_poll_success_ns", halt_poll_success_ns), - VCPU_STAT_GENERIC("halt_poll_fail_ns", halt_poll_fail_ns), - VCPU_STAT("nested_run", nested_run), - VCPU_STAT("directed_yield_attempted", directed_yield_attempted), - VCPU_STAT("directed_yield_successful", directed_yield_successful), - VCPU_STAT("guest_mode", guest_mode), - VM_STAT("mmu_shadow_zapped", mmu_shadow_zapped), - VM_STAT("mmu_pte_write", mmu_pte_write), - VM_STAT("mmu_pde_zapped", mmu_pde_zapped), - VM_STAT("mmu_flooded", mmu_flooded), - VM_STAT("mmu_recycled", mmu_recycled), - VM_STAT("mmu_cache_miss", mmu_cache_miss), - VM_STAT("mmu_unsync", mmu_unsync), - VM_STAT_GENERIC("remote_tlb_flush", remote_tlb_flush), - VM_STAT("largepages", lpages, .mode = 0444), - VM_STAT("nx_largepages_splitted", nx_lpage_splits, .mode = 0444), - VM_STAT("max_mmu_page_hash_collisions", max_mmu_page_hash_collisions), - { NULL } -}; - u64 __read_mostly host_xcr0; u64 __read_mostly supported_xcr0; EXPORT_SYMBOL_GPL(supported_xcr0); diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 692af9177c9f..6b353d6c50fd 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1262,14 +1262,8 @@ enum kvm_stat_kind {
struct kvm_stat_data { struct kvm *kvm; - struct kvm_stats_debugfs_item *dbgfs_item; -}; - -struct kvm_stats_debugfs_item { - const char *name; - int offset; + struct _kvm_stats_desc *desc; enum kvm_stat_kind kind; - int mode; };
struct _kvm_stats_header { @@ -1283,24 +1277,11 @@ struct _kvm_stats_desc { char name[KVM_STATS_NAME_LEN]; };
-#define KVM_DBGFS_GET_MODE(dbgfs_item) \ - ((dbgfs_item)->mode ? (dbgfs_item)->mode : 0644) - -#define VM_STAT(n, x, ...) \ - { n, offsetof(struct kvm, stat.x), KVM_STAT_VM, ## __VA_ARGS__ } -#define VCPU_STAT(n, x, ...) \ - { n, offsetof(struct kvm_vcpu, stat.x), KVM_STAT_VCPU, ## __VA_ARGS__ } -#define VM_STAT_GENERIC(n, x, ...) \ - { n, offsetof(struct kvm, stat.generic.x), KVM_STAT_VM, ## __VA_ARGS__ } -#define VCPU_STAT_GENERIC(n, x, ...) \ - { n, offsetof(struct kvm_vcpu, stat.generic.x), \ - KVM_STAT_VCPU, ## __VA_ARGS__ } - #define STATS_DESC_COMMON(type, unit, base, exp) \ .flags = type | unit | base | \ - BUILD_BUG_ON_ZERO(type & ~KVM_STATS_TYPE_MASK) | \ - BUILD_BUG_ON_ZERO(unit & ~KVM_STATS_UNIT_MASK) | \ - BUILD_BUG_ON_ZERO(base & ~KVM_STATS_BASE_MASK), \ + BUILD_BUG_ON_ZERO(type & ~KVM_STATS_TYPE_MASK) | \ + BUILD_BUG_ON_ZERO(unit & ~KVM_STATS_UNIT_MASK) | \ + BUILD_BUG_ON_ZERO(base & ~KVM_STATS_BASE_MASK), \ .exponent = exp, \ .size = 1
@@ -1341,19 +1322,24 @@ struct _kvm_stats_desc { SCOPE##_STATS_DESC(stat, type, unit, base, exp)
#define STATS_DESC_CUMULATIVE(SCOPE, name, unit, base, exponent) \ - STATS_DESC(SCOPE, name, KVM_STATS_TYPE_CUMULATIVE, \ - unit, base, exponent) + STATS_DESC(SCOPE, name, KVM_STATS_TYPE_CUMULATIVE, unit, base, exponent) #define STATS_DESC_INSTANT(SCOPE, name, unit, base, exponent) \ - STATS_DESC(SCOPE, name, KVM_STATS_TYPE_INSTANT, unit, base, exponent) \ + STATS_DESC(SCOPE, name, KVM_STATS_TYPE_INSTANT, unit, base, exponent) +#define STATS_DESC_PEAK(SCOPE, name, unit, base, exponent) \ + STATS_DESC(SCOPE, name, KVM_STATS_TYPE_PEAK, unit, base, exponent)
-/* Cumulative counter */ +/* Cumulative read/write counter */ #define STATS_DESC_COUNTER(SCOPE, name) \ STATS_DESC_CUMULATIVE(SCOPE, name, KVM_STATS_UNIT_NONE, \ KVM_STATS_BASE_POW10, 0) -/* Instantaneous counter */ +/* Instantaneous read only counter */ #define STATS_DESC_ICOUNTER(SCOPE, name) \ STATS_DESC_INSTANT(SCOPE, name, KVM_STATS_UNIT_NONE, \ KVM_STATS_BASE_POW10, 0) +/* Instantaneous read/write counter */ +#define STATS_DESC_PCOUNTER(SCOPE, name) \ + STATS_DESC_PEAK(SCOPE, name, KVM_STATS_UNIT_NONE, \ + KVM_STATS_BASE_POW10, 0)
/* Cumulative clock cycles */ #define STATS_DESC_CYCLE(SCOPE, name) \ @@ -1443,7 +1429,6 @@ struct _kvm_stats_desc { STATS_DESC_TIME_NSEC(VCPU_GENERIC, halt_poll_success_ns), \ STATS_DESC_TIME_NSEC(VCPU_GENERIC, halt_poll_fail_ns)
-extern struct kvm_stats_debugfs_item debugfs_entries[]; extern struct dentry *kvm_debugfs_dir; extern struct _kvm_stats_header kvm_vm_stats_header; extern struct _kvm_stats_header kvm_vcpu_stats_header; diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index d6e97b577d01..81ad814cdf24 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -1920,7 +1920,9 @@ struct kvm_stats_header { #define KVM_STATS_TYPE_MASK (0xF << KVM_STATS_TYPE_SHIFT) #define KVM_STATS_TYPE_CUMULATIVE (0x0 << KVM_STATS_TYPE_SHIFT) #define KVM_STATS_TYPE_INSTANT (0x1 << KVM_STATS_TYPE_SHIFT) -#define KVM_STATS_TYPE_MAX KVM_STATS_TYPE_INSTANT +/* Peak type is used as read/write instant counters */ +#define KVM_STATS_TYPE_PEAK (0x2 << KVM_STATS_TYPE_SHIFT) +#define KVM_STATS_TYPE_MAX KVM_STATS_TYPE_PEAK
#define KVM_STATS_UNIT_SHIFT 4 #define KVM_STATS_UNIT_MASK (0xF << KVM_STATS_UNIT_SHIFT) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index f5bea71ea61c..74f2d0567f3e 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -115,7 +115,6 @@ static DEFINE_PER_CPU(struct kvm_vcpu *, kvm_running_vcpu); struct dentry *kvm_debugfs_dir; EXPORT_SYMBOL_GPL(kvm_debugfs_dir);
-static int kvm_debugfs_num_entries; static const struct file_operations stat_fops_per_vm;
static long kvm_vcpu_ioctl(struct file *file, unsigned int ioctl, @@ -860,9 +859,24 @@ static void kvm_free_memslots(struct kvm *kvm, struct kvm_memslots *slots) kvfree(slots); }
+static umode_t kvm_stats_debugfs_mode(struct _kvm_stats_desc *pdesc) +{ + switch (pdesc->desc.flags & KVM_STATS_TYPE_MASK) { + case KVM_STATS_TYPE_INSTANT: + return 0444; + case KVM_STATS_TYPE_CUMULATIVE: + case KVM_STATS_TYPE_PEAK: + default: + return 0644; + } +} + + static void kvm_destroy_vm_debugfs(struct kvm *kvm) { int i; + int kvm_debugfs_num_entries = kvm_vm_stats_header.header.count + + kvm_vcpu_stats_header.header.count;
if (!kvm->debugfs_dentry) return; @@ -880,7 +894,10 @@ static int kvm_create_vm_debugfs(struct kvm *kvm, int fd) { char dir_name[ITOA_MAX_LEN * 2]; struct kvm_stat_data *stat_data; - struct kvm_stats_debugfs_item *p; + struct _kvm_stats_desc *pdesc; + int i; + int kvm_debugfs_num_entries = kvm_vm_stats_header.header.count + + kvm_vcpu_stats_header.header.count;
if (!debugfs_initialized()) return 0; @@ -894,15 +911,32 @@ static int kvm_create_vm_debugfs(struct kvm *kvm, int fd) if (!kvm->debugfs_stat_data) return -ENOMEM;
- for (p = debugfs_entries; p->name; p++) { + for (i = 0; i < kvm_vm_stats_header.header.count; ++i) { + pdesc = &kvm_vm_stats_desc[i]; stat_data = kzalloc(sizeof(*stat_data), GFP_KERNEL_ACCOUNT); if (!stat_data) return -ENOMEM;
stat_data->kvm = kvm; - stat_data->dbgfs_item = p; - kvm->debugfs_stat_data[p - debugfs_entries] = stat_data; - debugfs_create_file(p->name, KVM_DBGFS_GET_MODE(p), + stat_data->desc = pdesc; + stat_data->kind = KVM_STAT_VM; + kvm->debugfs_stat_data[i] = stat_data; + debugfs_create_file(pdesc->name, kvm_stats_debugfs_mode(pdesc), + kvm->debugfs_dentry, stat_data, + &stat_fops_per_vm); + } + + for (i = 0; i < kvm_vcpu_stats_header.header.count; ++i) { + pdesc = &kvm_vcpu_stats_desc[i]; + stat_data = kzalloc(sizeof(*stat_data), GFP_KERNEL_ACCOUNT); + if (!stat_data) + return -ENOMEM; + + stat_data->kvm = kvm; + stat_data->desc = pdesc; + stat_data->kind = KVM_STAT_VCPU; + kvm->debugfs_stat_data[i] = stat_data; + debugfs_create_file(pdesc->name, kvm_stats_debugfs_mode(pdesc), kvm->debugfs_dentry, stat_data, &stat_fops_per_vm); } @@ -4953,7 +4987,7 @@ static int kvm_debugfs_open(struct inode *inode, struct file *file, return -ENOENT;
if (simple_attr_open(inode, file, get, - KVM_DBGFS_GET_MODE(stat_data->dbgfs_item) & 0222 + kvm_stats_debugfs_mode(stat_data->desc) & 0222 ? set : NULL, fmt)) { kvm_put_kvm(stat_data->kvm); @@ -4976,14 +5010,14 @@ static int kvm_debugfs_release(struct inode *inode, struct file *file)
static int kvm_get_stat_per_vm(struct kvm *kvm, size_t offset, u64 *val) { - *val = *(u64 *)((void *)kvm + offset); + *val = *(u64 *)((void *)(&kvm->stat) + offset);
return 0; }
static int kvm_clear_stat_per_vm(struct kvm *kvm, size_t offset) { - *(u64 *)((void *)kvm + offset) = 0; + *(u64 *)((void *)(&kvm->stat) + offset) = 0;
return 0; } @@ -4996,7 +5030,7 @@ static int kvm_get_stat_per_vcpu(struct kvm *kvm, size_t offset, u64 *val) *val = 0;
kvm_for_each_vcpu(i, vcpu, kvm) - *val += *(u64 *)((void *)vcpu + offset); + *val += *(u64 *)((void *)(&vcpu->stat) + offset);
return 0; } @@ -5007,7 +5041,7 @@ static int kvm_clear_stat_per_vcpu(struct kvm *kvm, size_t offset) struct kvm_vcpu *vcpu;
kvm_for_each_vcpu(i, vcpu, kvm) - *(u64 *)((void *)vcpu + offset) = 0; + *(u64 *)((void *)(&vcpu->stat) + offset) = 0;
return 0; } @@ -5017,14 +5051,14 @@ static int kvm_stat_data_get(void *data, u64 *val) int r = -EFAULT; struct kvm_stat_data *stat_data = (struct kvm_stat_data *)data;
- switch (stat_data->dbgfs_item->kind) { + switch (stat_data->kind) { case KVM_STAT_VM: r = kvm_get_stat_per_vm(stat_data->kvm, - stat_data->dbgfs_item->offset, val); + stat_data->desc->desc.offset, val); break; case KVM_STAT_VCPU: r = kvm_get_stat_per_vcpu(stat_data->kvm, - stat_data->dbgfs_item->offset, val); + stat_data->desc->desc.offset, val); break; }
@@ -5039,14 +5073,14 @@ static int kvm_stat_data_clear(void *data, u64 val) if (val) return -EINVAL;
- switch (stat_data->dbgfs_item->kind) { + switch (stat_data->kind) { case KVM_STAT_VM: r = kvm_clear_stat_per_vm(stat_data->kvm, - stat_data->dbgfs_item->offset); + stat_data->desc->desc.offset); break; case KVM_STAT_VCPU: r = kvm_clear_stat_per_vcpu(stat_data->kvm, - stat_data->dbgfs_item->offset); + stat_data->desc->desc.offset); break; }
@@ -5103,6 +5137,7 @@ static int vm_stat_clear(void *_offset, u64 val) }
DEFINE_SIMPLE_ATTRIBUTE(vm_stat_fops, vm_stat_get, vm_stat_clear, "%llu\n"); +DEFINE_SIMPLE_ATTRIBUTE(vm_stat_readonly_fops, vm_stat_get, NULL, "%llu\n");
static int vcpu_stat_get(void *_offset, u64 *val) { @@ -5139,11 +5174,7 @@ static int vcpu_stat_clear(void *_offset, u64 val)
DEFINE_SIMPLE_ATTRIBUTE(vcpu_stat_fops, vcpu_stat_get, vcpu_stat_clear, "%llu\n"); - -static const struct file_operations *stat_fops[] = { - [KVM_STAT_VCPU] = &vcpu_stat_fops, - [KVM_STAT_VM] = &vm_stat_fops, -}; +DEFINE_SIMPLE_ATTRIBUTE(vcpu_stat_readonly_fops, vcpu_stat_get, NULL, "%llu\n");
static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm) { @@ -5197,15 +5228,32 @@ static void kvm_uevent_notify_change(unsigned int type, struct kvm *kvm)
static void kvm_init_debug(void) { - struct kvm_stats_debugfs_item *p; + const struct file_operations *fops; + struct _kvm_stats_desc *pdesc; + int i;
kvm_debugfs_dir = debugfs_create_dir("kvm", NULL);
- kvm_debugfs_num_entries = 0; - for (p = debugfs_entries; p->name; ++p, kvm_debugfs_num_entries++) { - debugfs_create_file(p->name, KVM_DBGFS_GET_MODE(p), - kvm_debugfs_dir, (void *)(long)p->offset, - stat_fops[p->kind]); + for (i = 0; i < kvm_vm_stats_header.header.count; ++i) { + pdesc = &kvm_vm_stats_desc[i]; + if (kvm_stats_debugfs_mode(pdesc) & 0222) + fops = &vm_stat_fops; + else + fops = &vm_stat_readonly_fops; + debugfs_create_file(pdesc->name, kvm_stats_debugfs_mode(pdesc), + kvm_debugfs_dir, + (void *)(long)pdesc->desc.offset, fops); + } + + for (i = 0; i < kvm_vcpu_stats_header.header.count; ++i) { + pdesc = &kvm_vcpu_stats_desc[i]; + if (kvm_stats_debugfs_mode(pdesc) & 0222) + fops = &vcpu_stat_fops; + else + fops = &vcpu_stat_readonly_fops; + debugfs_create_file(pdesc->name, kvm_stats_debugfs_mode(pdesc), + kvm_debugfs_dir, + (void *)(long)pdesc->desc.offset, fops); } }
On Mon, Jun 14, 2021 at 09:21:50PM +0000, Jing Zhang wrote:
This patchset provides a file descriptor for every VM and VCPU to read KVM statistics data in binary format. It is meant to provide a lightweight, flexible, scalable and efficient lock-free solution for user space telemetry applications to pull the statistics data periodically for large scale systems. The pulling frequency could be as high as a few times per second. In this patchset, every statistics data are treated to have some attributes as below:
- architecture dependent or generic
- VM statistics data or VCPU statistics data
- type: cumulative, instantaneous, peak
- unit: none for simple counter, nanosecond, microsecond, millisecond, second, Byte, KiByte, MiByte, GiByte. Clock Cycles
Since no lock/synchronization is used, the consistency between all the statistics data is not guaranteed. That means not all statistics data are read out at the exact same time, since the statistics date are still being updated by KVM subsystems while they are read out.
Sorry for my naive questions, but how does telemetry get statistics for hypervisors? Why is KVM different from hypervisors or NIC's statistics or any other high speed devices (RDMA) that generate tons of data?
Thanks
On 15/06/21 07:25, Leon Romanovsky wrote:
Sorry for my naive questions, but how does telemetry get statistics for hypervisors? Why is KVM different from hypervisors or NIC's statistics or any other high speed devices (RDMA) that generate tons of data?
Right now, the only way is debugfs but it's slow, and it's disabled when using lockdown mode; this series is a way to fix this.
I sense that there is another question in there; are you wondering if another mechanism should be used, for example netlink? The main issue there is how to identify a VM, since KVM file descriptors don't have a name. Using a pid works (sort of) for debugfs, but pids are not appropriate for a stable API. Using a file descriptor as in this series requires collaboration from the userspace program; howver, once the file descriptor has been transmitted via SCM_RIGHTS, telemetry can read it forever without further IPC, and there is proper privilege separation.
Paolo
On Tue, Jun 15, 2021 at 09:06:43AM +0200, Paolo Bonzini wrote:
On 15/06/21 07:25, Leon Romanovsky wrote:
Sorry for my naive questions, but how does telemetry get statistics for hypervisors? Why is KVM different from hypervisors or NIC's statistics or any other high speed devices (RDMA) that generate tons of data?
Right now, the only way is debugfs but it's slow, and it's disabled when using lockdown mode; this series is a way to fix this.
I sense that there is another question in there; are you wondering if another mechanism should be used, for example netlink? The main issue there is how to identify a VM, since KVM file descriptors don't have a name. Using a pid works (sort of) for debugfs, but pids are not appropriate for a stable API. Using a file descriptor as in this series requires collaboration from the userspace program; howver, once the file descriptor has been transmitted via SCM_RIGHTS, telemetry can read it forever without further IPC, and there is proper privilege separation.
Yeah, sorry for mixing different questions into one.
So the answer to the question "why KVM is different" is that it doesn't have any stable identification except file descriptor. While hypervisors have stable names, NICs and RDMA devices have interface indexes e.t.c. Did I get it right?
And this was second part of my question, the first part was my attempt to get on answer why current statistics like process info (/proc/xxx/*), NICs (netlink) and RDMA (sysfs) are not using binary format.
Thanks
Paolo
Hi Jing,
+int main(int argc, char *argv[]) +{
int max_vm = DEFAULT_NUM_VM, max_vcpu = DEFAULT_NUM_VCPU, ret, i, j;struct kvm_vm **vms;/* Get the number of VMs and VCPUs that would be created for testing. */if (argc > 1) {max_vm = strtol(argv[1], NULL, 0);if (max_vm <= 0)max_vm = DEFAULT_NUM_VM;}if (argc > 2) {max_vcpu = strtol(argv[2], NULL, 0);if (max_vcpu <= 0)max_vcpu = DEFAULT_NUM_VCPU;}/* Check the extension for binary stats */ret = kvm_check_cap(KVM_CAP_BINARY_STATS_FD);TEST_ASSERT(ret >= 0,"Binary form statistics interface is not supported");
kvm_check_cap returns the value of KVM_CHECK_EXTENSION, which is 0 if unsupported (-ERROR on an error). The assertion should be for ret > 0.
Made that change locally, and tested it with various configurations (vhe, nvhe), as well as kernel versions (with and without KVM_CAP_BINARY_STATS_FD), and it passes (or fails as expected). Without that fix and with a kernel that doesn't support KVM_CAP_BINARY_STATS_FD, it passes that assertion, but fails later at vcpu_stats_test().
With that fixed: Tested-by: Fuad Tabba tabba@google.com #arm64
Cheers, /fuad
/* Create VMs and VCPUs */vms = malloc(sizeof(vms[0]) * max_vm);TEST_ASSERT(vms, "Allocate memory for storing VM pointers");for (i = 0; i < max_vm; ++i) {vms[i] = vm_create(VM_MODE_DEFAULT,DEFAULT_GUEST_PHY_PAGES, O_RDWR);for (j = 0; j < max_vcpu; ++j)vm_vcpu_add(vms[i], j);}/* Check stats read for every VM and VCPU */for (i = 0; i < max_vm; ++i) {vm_stats_test(vms[i]);for (j = 0; j < max_vcpu; ++j)vcpu_stats_test(vms[i], j);}for (i = 0; i < max_vm; ++i)kvm_vm_free(vms[i]);free(vms);return 0;+} diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index 5c70596dd1b9..83c02cb0ae1e 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -2286,3 +2286,15 @@ unsigned int vm_calc_num_guest_pages(enum vm_guest_mode mode, size_t size) n = DIV_ROUND_UP(size, vm_guest_mode_params[mode].page_size); return vm_adjust_num_guest_pages(mode, n); }
+int vm_get_stats_fd(struct kvm_vm *vm) +{
return ioctl(vm->fd, KVM_GET_STATS_FD, NULL);+}
+int vcpu_get_stats_fd(struct kvm_vm *vm, uint32_t vcpuid) +{
struct vcpu *vcpu = vcpu_find(vm, vcpuid);return ioctl(vcpu->fd, KVM_GET_STATS_FD, NULL);+}
2.32.0.272.g935e593368-goog
On 14.06.21 23:21, Jing Zhang wrote:
Hi,
This patchset provides a file descriptor for every VM and VCPU to read KVM statistics data in binary format.
I've missed the discussions of previous versions, so please forgive my stupid questions:
* why is it binary instead of text ? is it so very high volume that it really matters ? * how will possible future extensions of the telemetry packets work ? * aren't there other means to get this fd instead of an ioctl() on the VM fd ? something more from the outside (eg. sysfs/procfs) * how will that relate to other hypervisors ?
Some notes from the operating perspective:
In typical datacenters we've got various monitoring tools that are able to catch up lots of data from different sources (especially files). If an operator e.g. is interested in something in happening in some file (e.g. in /proc of /sys), it's quite trivial - just configure yet another probe (maybe some regex for parsing) and done. Automatically fed in his $monitoring_solution (e.g. nagios, ELK, Splunk, whatsnot)
With your approach, it's not that simple: now the operator needs to create (and deploy and manage) a separate agent that somehow receives that fd from the VMM, reads and parses that specific binary stream and finally pushes it into the monitoring infrastructure. Or the VMM writes it into some file, where some monitoring agent can pick it up. In any case, not actually trivial from ops perspective.
In general I tend to like the fd approach (even though I don't like ioctls very much - I'd rather like to have it more Plan9-like ;-)). But it has the drawback of acquiring those fd's by separate processes isn't entirely easy and needs a lot of coordinated interaction.
That issue would be much easier if we had the ability to publish existing fd's into the file system (like Plan9's srvfs does), but we don't have that yet. (actually, I've hacked up some srvfs for Linux, but ... well ... it's just a hack, nowhere near to production).
Why not putting this into sysfs ?
I see two options:
a) if it's really kvm-specific (and no chance of using the same interface for other hypervisors), we could put it under the kvm device (/sys/class/misc/kvm).
b) have a generic VMM stats interface that theroretically could work with any hypervisor.
--mtx
On Tue, Jun 15, 2021 at 10:37:36AM +0200, Enrico Weigelt, metux IT consult wrote:
Why not putting this into sysfs ?
Because sysfs is "one value per file".
I see two options:
a) if it's really kvm-specific (and no chance of using the same interface for other hypervisors), we could put it under the kvm device (/sys/class/misc/kvm).
Again, that is NOT what sysfs is for.
b) have a generic VMM stats interface that theroretically could work with any hypervisor.
What other hypervisor matters?
greg k-h
On 15/06/21 09:53, Leon Romanovsky wrote:
Sorry for my naive questions, but how does telemetry get statistics for hypervisors? Why is KVM different from hypervisors or NIC's statistics or any other high speed devices (RDMA) that generate tons of data?
So the answer to the question "why KVM is different" is that it doesn't have any stable identification except file descriptor. While hypervisors have stable names, NICs and RDMA devices have interface indexes etc. Did I get it right?
Right.
And this was second part of my question, the first part was my attempt to get on answer why current statistics like process info (/proc/xxx/*), NICs (netlink) and RDMA (sysfs) are not using binary format.
NICs are using binary format (partly in struct ethtool_stats, partly in an array of u64). For KVM we decided to put the schema and the stats in the same file (though you can use pread to get only the stats) to have a single interface and avoid ioctls, unlike having both ETH_GSTRINGS and ETH_GSTATS.
I wouldn't say processes are using any specific format. There's a mix of "one value per file" (e.g. cpuset), human-readable tabular format (e.g. limits, sched), human- and machine-readable tabular format (e.g. status), and files that are ASCII but not human-readable (e.g. stat).
Paolo
On 15/06/21 10:37, Enrico Weigelt, metux IT consult wrote:
- why is it binary instead of text ? is it so very high volume that
it really matters ?
The main reason to have a binary format is not the high volume actually (though it also has its part). Rather, we would really like to include the schema to make the statistics self-describing. This includes stuff like whether the unit of measure of a statistic is clock cycles, nanoseconds, pages or whatnot; having this kind of information in text leads to awkwardness in the parsers. trace-cmd is another example where the data consists of a schema followed by binary data.
Text format could certainly be added if there's a usecase, but for developer use debugfs is usually a suitable replacement.
Last year we tried the opposite direction: we built a one-value-per-file filesystem with a common API that any subsystem could use (e.g. providing ethtool stats, /proc/interrupts, etc. in addition to KVM stats). We started with text, similar to sysfs, with the plan of extending it to a binary format later. However, other subsystems expressed very little interest in this, so instead we decided to go with something that is designed around KVM needs.
Still, the binary format that KVM uses is designed not to be KVM-specific. If other subsystems want to publish high-volume, self-describing statistic information, they are welcome to share the binary format and the code. Perhaps it may make sense in some cases to have them in sysfs, even (e.g. /sys/kernel/slab/*/.stats). As Greg said sysfs is currently one value per file, but perhaps that could be changed if the binary format is an additional way to access the information and not the only one (not that I'm planning to do it).
- how will possible future extensions of the telemetry packets work ?
The format includes a schema, so it's possible to add more statistics in the future. The exact list of statistics varies per architecture and is not part of the userspace API (obvious caveat: https://xkcd.com/1172/).
- aren't there other means to get this fd instead of an ioctl() on the
VM fd ? something more from the outside (eg. sysfs/procfs)
Not yet, but if there's a need it can be added. It'd be plausible to publish system-wide statistics via a ioctl on /dev/kvm, for example. We'd have to check how this compares with stuff that is world-readable in procfs and sysfs, but I don't think there are security concerns in exposing that.
There's also pidfd_getfd(2) which can be used to pull a VM file descriptor from another running process. That can be used to avoid the issue of KVM file descriptors being unnamed.
- how will that relate to other hypervisors ?
Other hypervisors do not run as part of the Linux kernel (at least they are not upstream). These statistics only apply to Linux *hosts*, not guests.
As far as I know, there is no standard that Xen or the proprietary hypervisors use to communicate their telemetry info to monitoring tools, and also no standard binary format used by exporters to talk to monitoring tools. If this format will be adopted by other hypervisors or any random software, I will be happy.
Some notes from the operating perspective:
In typical datacenters we've got various monitoring tools that are able to catch up lots of data from different sources (especially files). If an operator e.g. is interested in something in happening in some file (e.g. in /proc of /sys), it's quite trivial - just configure yet another probe (maybe some regex for parsing) and done. Automatically fed in his $monitoring_solution (e.g. nagios, ELK, Splunk, whatsnot)
... but in practice what you do is you have prebuilt exporters that talks to $monitoring_solution. Monitoring individual files is the exception, not the rule. But indeed Libvirt already has I/O and network statistics and there is an exporter for Prometheus, so we should add support for this new method as well to both QEMU (exporting the file descriptor) and Libvirt.
I hope this helps clarifying your doubts!
Paolo
With your approach, it's not that simple: now the operator needs to create (and deploy and manage) a separate agent that somehow receives that fd from the VMM, reads and parses that specific binary stream and finally pushes it into the monitoring infrastructure. Or the VMM writes it into some file, where some monitoring agent can pick it up. In any case, not actually trivial from ops perspective.
On Tue, Jun 15, 2021 at 01:03:34PM +0200, Paolo Bonzini wrote:
On 15/06/21 09:53, Leon Romanovsky wrote:
Sorry for my naive questions, but how does telemetry get statistics for hypervisors? Why is KVM different from hypervisors or NIC's statistics or any other high speed devices (RDMA) that generate tons of data?
So the answer to the question "why KVM is different" is that it doesn't have any stable identification except file descriptor. While hypervisors have stable names, NICs and RDMA devices have interface indexes etc. Did I get it right?
Right.
And this was second part of my question, the first part was my attempt to get on answer why current statistics like process info (/proc/xxx/*), NICs (netlink) and RDMA (sysfs) are not using binary format.
NICs are using binary format (partly in struct ethtool_stats, partly in an array of u64). For KVM we decided to put the schema and the stats in the same file (though you can use pread to get only the stats) to have a single interface and avoid ioctls, unlike having both ETH_GSTRINGS and ETH_GSTATS.
I wouldn't say processes are using any specific format. There's a mix of "one value per file" (e.g. cpuset), human-readable tabular format (e.g. limits, sched), human- and machine-readable tabular format (e.g. status), and files that are ASCII but not human-readable (e.g. stat).
I see, your explanation to Enrico cleared the mud.
Thanks
Paolo
On Mon, Jun 14, 2021 at 09:21:53PM +0000, Jing Zhang wrote:
Update KVM API documentation for binary statistics.
You should write more here. See my comment at the bottom...
Reviewed-by: David Matlack dmatlack@google.com Reviewed-by: Ricardo Koller ricarkol@google.com Reviewed-by: Krish Sadhukhan krish.sadhukhan@oracle.com Reviewed-by: Fuad Tabba tabba@google.com Signed-off-by: Jing Zhang jingzhangos@google.com
Documentation/virt/kvm/api.rst | 177 ++++++++++++++++++++++++++++++++- 1 file changed, 176 insertions(+), 1 deletion(-)
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index e328caa35d6c..35ee52dbec89 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -5034,7 +5034,6 @@ see KVM_XEN_VCPU_SET_ATTR above. The KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_ADJUST type may not be used with the KVM_XEN_VCPU_GET_ATTR ioctl.
4.131 KVM_GET_SREGS2
@@ -5081,6 +5080,174 @@ Writes special registers into the vcpu. See KVM_GET_SREGS2 for the data structures. This ioctl (when supported) replaces the KVM_SET_SREGS. +4.133 KVM_GET_STATS_FD +----------------------
+:Capability: KVM_CAP_STATS_BINARY_FD +:Architectures: all +:Type: vm ioctl, vcpu ioctl +:Parameters: none +:Returns: statistics file descriptor on success, < 0 on error
+Errors:
- ====== ======================================================
- ENOMEM if the fd could not be created due to lack of memory
- EMFILE if the number of opened files exceeds the limit
- ====== ======================================================
+The file descriptor can be used to read VM/vCPU statistics data in binary +format. The file data is organized into three blocks as below: ++-------------+ +| Header | ++-------------+ +| Descriptors | ++-------------+ +| Stats Data | ++-------------+
+The Header block is always at the start of the file. It is only needed to be +read one time for the lifetime of the file descriptor. +It is in the form of ``struct kvm_stats_header`` as below::
- #define KVM_STATS_ID_MAXLEN 64
- struct kvm_stats_header {
__u32 name_size;__u32 count;__u32 desc_offset;__u32 data_offset;char id[0];- };
+The ``id`` field is identification for the corresponding KVM statistics. For +VM statistics, it is in the form of "kvm-{kvm pid}", like "kvm-12345". For +VCPU statistics, it is in the form of "kvm-{kvm pid}/vcpu-{vcpu id}", like +"kvm-12345/vcpu-12".
+The ``name_size`` field is the size (byte) of the statistics name string +(including trailing '\0') appended to the end of every statistics descriptor.
+The ``count`` field is the number of statistics.
+The ``desc_offset`` field is the offset of the Descriptors block from the start +of the file indicated by the file descriptor.
+The ``data_offset`` field is the offset of the Stats Data block from the start +of the file indicated by the file descriptor.
+The Descriptors block is only needed to be read once for the lifetime of the +file descriptor. It is an array of ``struct kvm_stats_desc`` as shown in +below code block::
- #define KVM_STATS_TYPE_SHIFT 0
- #define KVM_STATS_TYPE_MASK (0xF << KVM_STATS_TYPE_SHIFT)
- #define KVM_STATS_TYPE_CUMULATIVE (0x0 << KVM_STATS_TYPE_SHIFT)
- #define KVM_STATS_TYPE_INSTANT (0x1 << KVM_STATS_TYPE_SHIFT)
- #define KVM_STATS_TYPE_MAX KVM_STATS_TYPE_INSTANT
- #define KVM_STATS_UNIT_SHIFT 4
- #define KVM_STATS_UNIT_MASK (0xF << KVM_STATS_UNIT_SHIFT)
- #define KVM_STATS_UNIT_NONE (0x0 << KVM_STATS_UNIT_SHIFT)
- #define KVM_STATS_UNIT_BYTES (0x1 << KVM_STATS_UNIT_SHIFT)
- #define KVM_STATS_UNIT_SECONDS (0x2 << KVM_STATS_UNIT_SHIFT)
- #define KVM_STATS_UNIT_CYCLES (0x3 << KVM_STATS_UNIT_SHIFT)
- #define KVM_STATS_UNIT_MAX KVM_STATS_UNIT_CYCLES
- #define KVM_STATS_BASE_SHIFT 8
- #define KVM_STATS_BASE_MASK (0xF << KVM_STATS_BASE_SHIFT)
- #define KVM_STATS_BASE_POW10 (0x0 << KVM_STATS_BASE_SHIFT)
- #define KVM_STATS_BASE_POW2 (0x1 << KVM_STATS_BASE_SHIFT)
- #define KVM_STATS_BASE_MAX KVM_STATS_BASE_POW2
- struct kvm_stats_desc {
__u32 flags;__s16 exponent;__u16 size;__u32 offset;__u32 unused;char name[0];- };
+The ``flags`` field contains the type and unit of the statistics data described +by this descriptor. The following flags are supported:
+Bits 0-3 of ``flags`` encode the type:
- ``KVM_STATS_TYPE_CUMULATIVE``
- The statistics data is cumulative. The value of data can only be increased.
- Most of the counters used in KVM are of this type.
- The corresponding ``count`` field for this type is always 1.
- ``KVM_STATS_TYPE_INSTANT``
- The statistics data is instantaneous. Its value can be increased or
- decreased. This type is usually used as a measurement of some resources,
- like the number of dirty pages, the number of large pages, etc.
- The corresponding ``count`` field for this type is always 1.
+Bits 4-7 of ``flags`` encode the unit:
- ``KVM_STATS_UNIT_NONE``
- There is no unit for the value of statistics data. This usually means that
- the value is a simple counter of an event.
- ``KVM_STATS_UNIT_BYTES``
- It indicates that the statistics data is used to measure memory size, in the
- unit of Byte, KiByte, MiByte, GiByte, etc. The unit of the data is
- determined by the ``exponent`` field in the descriptor. The
- ``KVM_STATS_BASE_POW2`` flag is valid in this case. The unit of the data is
- determined by ``pow(2, exponent)``. For example, if value is 10,
- ``exponent`` is 20, which means the unit of statistics data is MiByte, we
- can get the statistics data in the unit of Byte by
- ``value * pow(2, exponent) = 10 * pow(2, 20) = 10 MiByte`` which is
- 10 * 1024 * 1024 Bytes.
- ``KVM_STATS_UNIT_SECONDS``
- It indicates that the statistics data is used to measure time/latency, in
- the unit of nanosecond, microsecond, millisecond and second. The unit of the
- data is determined by the ``exponent`` field in the descriptor. The
- ``KVM_STATS_BASE_POW10`` flag is valid in this case. The unit of the data
- is determined by ``pow(10, exponent)``. For example, if value is 2000000,
- ``exponent`` is -6, which means the unit of statistics data is microsecond,
- we can get the statistics data in the unit of second by
- ``value * pow(10, exponent) = 2000000 * pow(10, -6) = 2 seconds``.
- ``KVM_STATS_UNIT_CYCLES``
- It indicates that the statistics data is used to measure CPU clock cycles.
- The ``KVM_STATS_BASE_POW10`` flag is valid in this case. For example, if
- value is 200, ``exponent`` is 4, we can get the number of CPU clock cycles
- by ``value * pow(10, exponent) = 200 * pow(10, 4) = 2000000``.
+Bits 8-11 of ``flags`` encode the base:
- ``KVM_STATS_BASE_POW10``
- The scale is based on power of 10. It is used for measurement of time and
- CPU clock cycles.
- ``KVM_STATS_BASE_POW2``
- The scale is based on power of 2. It is used for measurement of memory size.
+The ``exponent`` field is the scale of corresponding statistics data. For +example, if the unit is ``KVM_STATS_UNIT_BYTES``, the base is +``KVM_STATS_BASE_POW2``, the ``exponent`` is 10, then we know that the real +unit of the statistics data is KBytes a.k.a pow(2, 10) = 1024 bytes.
+The ``size`` field is the number of values (u64) of this statistics data. Its +value is usually 1 for most of simple statistics.
+The ``offset`` field is the offset from the start of Data Block to the start of +the corresponding statistics data.
+The ``unused`` fields are reserved for future support for other types of +statistics data, like log/linear histogram.
+The ``name`` field points to the name string of the statistics data. The name +string starts at the end of ``struct kvm_stats_desc``. +The maximum length (including trailing '\0') is indicated by ``name_size`` +in ``struct kvm_stats_header``.
+The Stats Data block contains an array of data values of type ``struct +kvm_vm_stats_data`` or ``struct kvm_vcpu_stats_data``. It would be read by +userspace periodically to pull statistics data. +The order of data value in Stats Data block is the same as the order of +descriptors in Descriptors block.
- Statistics data for VM/VCPU::
- struct kvm_stats_data {
__u64 value[0];- };
I forgot to comment on this one, sorry for the delay.
Why are you "inventing" your own schema format here for this? Why not use a well-known or at least well-designed/implemented one that we have in userspace already?
There are a few that I would love to see in the kernel, varlink being the best example. We have kernel examples of this and I would consider using that as a transport for sysfs-like data in the future, but never got around to it.
So again, why reinvent the wheel to create a custom api when you could use an existing one?
thanks,
greg k-h
On 16/06/21 17:21, Greg KH wrote:
I forgot to comment on this one, sorry for the delay.
Why are you "inventing" your own schema format here for this? Why not use a well-known or at least well-designed/implemented one that we have in userspace already?
There are a few that I would love to see in the kernel, varlink being the best example. We have kernel examples of this and I would consider using that as a transport for sysfs-like data in the future, but never got around to it.
Thanks, that's a good observation, and it's a problem that the rationale and the design process didn't end up in either the documentation or the commit message (only the outcome did). In order to fix that, this is going to be quite a long message.
Varlink comprises both a schema and an encoding, and each has its own problems. For the encoding of the data, varlink is really just using JSON and it is really just the wrong tool here. The first few problems that come to mind are:
- varlink structs are encoded as JSON dictionaries. Therefore, every time userspace reads the fields, the kernel has to include the field names as JSON dictionary keys. This means that a lot of time is spent writing buffers, and on the receiving side parsing them.
- because numeric data has to be converted to ASCII the output doesn't have fixed offsets, so it is not possible to make an efficient implementation of pread.
- even though Varlink specifies that int is "usually int64", a little-known gem is that JSON behavior for numbers not representable as a double (i.e. exceeding 2^53) is implementation-defined; implementations can and will mess up values outside that range. This problem is not specific to this KVM stats usecase; varlink's schema specification really is at odds with its encoding specification.
For the schema, there are some specific problems with varlink, but also a more generic issue. The specific problems are:
- the schema doesn't include the length of arrays. This makes it hard to compute in advance lengths and offsets of fields (even ignoring the fact that data is not binary, which I'll get to later)
- the schema also is not extensible with user annotations. In our case that would be mostly the unit in which the value is expressed.
The main problem with the various available serialization formats is that their schema languages are designed to be compiled with a code generator. Here instead the schema is transmitted from the kernel to userspace. Userspace is not really supposed to know what a value means or even that if it exists. Userspace takes care of collecting the data from the kernel, but ultimately there will be a human that knows what e.g. "io_exits" or "req_event" means, and they will ask for the current value, or a plot over time, of a specific statistic.
Now, unlike most other schema languages, varlink does not require precompiling the schema in its C bindings. However, this was mostly just the authors not bothering to write a C code generator---they did write one for Rust, for example. When using the "official" varlink C bindings, you hardly use the schema at all.
Something similar to the above issues is quite common in other formats. For example, FlatBuffers[2]'s schema language[1] does have annotations (which it calls metadata), but then we would still have to invent some standard annotations and teach programs about them. However, the deal breaker is that again there is no way to transmit the schema from the server to the client, and that users are supposed to precompile the schema using a code generator.
All that said, what we _could_ do is serialize the schema as JSON instead of using a binary format, like so:
[{ "name": "exits", "kind": "cumulative", "count": 1 }, { "name": "halt_poll_fail_ns", "kind": "cumulative", "unit": "seconds", "scale": {"base": "pow2", exponent: -9}, "count": 1 }, ...]
while keeping the actual statistics as an array of u64 values just like in these patches. The JSON representation of the schema would be always the same, so it could be treated as fixed-size and an efficient implementation of pread would be possible. And once the schema is JSON, there could be a *meta*-schema expressed using the varlink language:
# compare to struct kvm_stats_desc, included after my sig # for reference type StatsDescriptor { name: string, kind: (cumulative, instant, peak), unit: ?(bytes, seconds, cycles), scale: ?(base: (pow2, pow10), exponent: int), count: int }
type StatsSchema { name: string, stats: []StatsDescriptor }
Varlink would *not* be used in the kernel, because Varlink isreally just JSON. The above ten lines (plus comments) would be dropped in Documentation/ as a .varlink file; they would be a convenient way to describe (to either a human or a program) how to parse the JSON schema.
There would then be another problem, namely how to include the varlink schema in the generated documentation, since right now there are for example no sphinx bindings for varlink. Nevertheless, if you think it's better to have the schema as JSON instead of binary, according to the above varlink meta-schema, then we can look into it.
Thanks,
Paolo
- struct kvm_stats_desc {
__u32 flags;__s16 exponent;__u16 size;__u32 offset;__u32 unused;char name[0];- };
On 14/06/21 23:21, Jing Zhang wrote:
- /* Copy kvm stats values */
- copylen = header->header.data_offset + size_stats - pos;
- copylen = min(copylen, remain);
- if (copylen > 0) {
src = stats + pos - header->header.data_offset;if (copy_to_user(dest, src, copylen))return -EFAULT;remain -= copylen;pos += copylen;dest += copylen;- }
Hi Jing,
this code is causing usercopy warnings because the statistics are not part of the vcpu slab's usercopy region. You need to move struct kvm_vcpu_stat next to struct kvm_vcpu_arch, and adjust the call to kmem_cache_create_usercopy in kvm_init.
Can you post a new version of the series, and while you are at it explain the rationale for binary stats in the commit message for this patch? This should include:
- the problem statement (e.g. frequency of the accesses)
- what are the benefits compared to debugfs
- why the schema is included in the file descriptor as well
You can probably find a lot or all of the information in my emails from the last couple days, but you might also have other breadcrumbs from Google's internal implementation of binary stats.
Thanks,
Paolo
On Wed, Jun 16, 2021 at 12:12 PM Paolo Bonzini pbonzini@redhat.com wrote:
On 14/06/21 23:21, Jing Zhang wrote:
/* Copy kvm stats values */copylen = header->header.data_offset + size_stats - pos;copylen = min(copylen, remain);if (copylen > 0) {src = stats + pos - header->header.data_offset;if (copy_to_user(dest, src, copylen))return -EFAULT;remain -= copylen;pos += copylen;dest += copylen;}Hi Jing,
this code is causing usercopy warnings because the statistics are not part of the vcpu slab's usercopy region. You need to move struct kvm_vcpu_stat next to struct kvm_vcpu_arch, and adjust the call to kmem_cache_create_usercopy in kvm_init.
Can you post a new version of the series, and while you are at it explain the rationale for binary stats in the commit message for this patch? This should include:
the problem statement (e.g. frequency of the accesses)
what are the benefits compared to debugfs
why the schema is included in the file descriptor as well
You can probably find a lot or all of the information in my emails from the last couple days, but you might also have other breadcrumbs from Google's internal implementation of binary stats.
Thanks,
Paolo
Hi Paolo
Will fix the usercopy warnings, add more explanations and post another version. And thanks for all the information in your emails.
Jing
On Wed, Jun 16, 2021 at 06:59:15PM +0200, Paolo Bonzini wrote:
On 16/06/21 17:21, Greg KH wrote:
I forgot to comment on this one, sorry for the delay.
Why are you "inventing" your own schema format here for this? Why not use a well-known or at least well-designed/implemented one that we have in userspace already?
There are a few that I would love to see in the kernel, varlink being the best example. We have kernel examples of this and I would consider using that as a transport for sysfs-like data in the future, but never got around to it.
Thanks, that's a good observation, and it's a problem that the rationale and the design process didn't end up in either the documentation or the commit message (only the outcome did). In order to fix that, this is going to be quite a long message.
Varlink comprises both a schema and an encoding, and each has its own problems. For the encoding of the data, varlink is really just using JSON and it is really just the wrong tool here. The first few problems that come to mind are:
- varlink structs are encoded as JSON dictionaries. Therefore, every time
userspace reads the fields, the kernel has to include the field names as JSON dictionary keys. This means that a lot of time is spent writing buffers, and on the receiving side parsing them.
Has this been measured? Years ago when I messed with this, it was in the noise as JSON parsing is really fast these days.
- because numeric data has to be converted to ASCII the output doesn't have
fixed offsets, so it is not possible to make an efficient implementation of pread.
efficient where? In the kernel?
- even though Varlink specifies that int is "usually int64", a little-known
gem is that JSON behavior for numbers not representable as a double (i.e. exceeding 2^53) is implementation-defined; implementations can and will mess up values outside that range. This problem is not specific to this KVM stats usecase; varlink's schema specification really is at odds with its encoding specification.
That's interesting, do the varlink developers know this? And we can say "for the kernel int will be int64" and be done with it, so this shouldn't be that big of an issue.
For the schema, there are some specific problems with varlink, but also a more generic issue. The specific problems are:
- the schema doesn't include the length of arrays. This makes it hard to
compute in advance lengths and offsets of fields (even ignoring the fact that data is not binary, which I'll get to later)
Do you care in advance?
- the schema also is not extensible with user annotations. In our case that
would be mostly the unit in which the value is expressed.
units would be nice, I agree.
The main problem with the various available serialization formats is that their schema languages are designed to be compiled with a code generator. Here instead the schema is transmitted from the kernel to userspace. Userspace is not really supposed to know what a value means or even that if it exists. Userspace takes care of collecting the data from the kernel, but ultimately there will be a human that knows what e.g. "io_exits" or "req_event" means, and they will ask for the current value, or a plot over time, of a specific statistic.
Now, unlike most other schema languages, varlink does not require precompiling the schema in its C bindings. However, this was mostly just the authors not bothering to write a C code generator---they did write one for Rust, for example. When using the "official" varlink C bindings, you hardly use the schema at all.
I don't think the kernel implementation needs this, you can just specify the schema "on the fly" and it should work. Or maybe I'm wrong, it's been a few years.
Something similar to the above issues is quite common in other formats. For example, FlatBuffers[2]'s schema language[1] does have annotations (which it calls metadata), but then we would still have to invent some standard annotations and teach programs about them. However, the deal breaker is that again there is no way to transmit the schema from the server to the client, and that users are supposed to precompile the schema using a code generator.
Again, I didn't think this was an issue with the kernel implementation in that the userspace side could determine the schema by the data coming from the kernel, it wouldn't have to "know" about it ahead of time.
But I could be wrong.
All that said, what we _could_ do is serialize the schema as JSON instead of using a binary format, like so:
[{ "name": "exits", "kind": "cumulative", "count": 1 }, { "name": "halt_poll_fail_ns", "kind": "cumulative", "unit": "seconds", "scale": {"base": "pow2", exponent: -9}, "count": 1 }, ...]
while keeping the actual statistics as an array of u64 values just like in these patches. The JSON representation of the schema would be always the same, so it could be treated as fixed-size and an efficient implementation of pread would be possible. And once the schema is JSON, there could be a *meta*-schema expressed using the varlink language:
# compare to struct kvm_stats_desc, included after my sig # for reference type StatsDescriptor { name: string, kind: (cumulative, instant, peak), unit: ?(bytes, seconds, cycles), scale: ?(base: (pow2, pow10), exponent: int), count: int }
type StatsSchema { name: string, stats: []StatsDescriptor }
Varlink would *not* be used in the kernel, because Varlink isreally just JSON. The above ten lines (plus comments) would be dropped in Documentation/ as a .varlink file; they would be a convenient way to describe (to either a human or a program) how to parse the JSON schema.
There would then be another problem, namely how to include the varlink schema in the generated documentation, since right now there are for example no sphinx bindings for varlink. Nevertheless, if you think it's better to have the schema as JSON instead of binary, according to the above varlink meta-schema, then we can look into it.
It should be in some standard format. I'm not wed to varlink, but I thought it solved some of your issues here with userspace not knowing the schema ahead of time which is the key as it will change over time and the kernel should be 'self-describing'.
If not, and it sounds like you have looked into it, or at least the userspace side, then that's fine.
But you should write up a justification somewhere why you didn't use an existing format (what about the netlink format?) in at least the changelog, like you did here. As you will be stuck with this api for the next 20+ years, it's good to be confident in it :)
maybe someday we can get varlink into the tree, as I still think it would be better than the ad-hoc implementations we currently have.
thanks,
greg k-h
On 16/06/21 20:18, Greg KH wrote:
On Wed, Jun 16, 2021 at 06:59:15PM +0200, Paolo Bonzini wrote:
- varlink structs are encoded as JSON dictionaries. Therefore, every time
userspace reads the fields, the kernel has to include the field names as JSON dictionary keys. This means that a lot of time is spent writing buffers, and on the receiving side parsing them.
Has this been measured? Years ago when I messed with this, it was in the noise as JSON parsing is really fast these days.
Yes, parsing is really fast. However, the work doesn't end at building an in-memory representation. An efficient representation (and a schema that is negotiated in advance) makes it possible to do this work as late and as little as possible, instead of doing it on every fetch of the statistics.
For cloud vendors running virtual machines, they want to consolidate housekeeping tasks on as few CPUs as possible (because housekeeping CPUs cannot be billed to the customers), and every millisecond really counts. ASCII is inviting, but things like Protobufs, FlatBuffers, Cap'n'Proto are all binary because all those hidden costs do exist.
- because numeric data has to be converted to ASCII the output doesn't have
fixed offsets, so it is not possible to make an efficient implementation of pread.
efficient where? In the kernel?
Yes, Jing's patches can just do a quick copy_to_user if pread does not access the schema. And it's very simple code too.
- even though Varlink specifies that int is "usually int64", a little-known
gem is that JSON behavior for numbers not representable as a double (i.e. exceeding 2^53) is implementation-defined
That's interesting, do the varlink developers know this? And we can say "for the kernel int will be int64" and be done with it, so this shouldn't be that big of an issue.
Well yeah, but there's still the problem of what the other side thinks. In the end varlink's interesting because it's just JSON, meaning there's plenty of parsers available---but they all too often don't separate int vs. double. We had this issue with projects talking to QEMU (which has been using JSON the same way as varlink for ten years or so) and JSON parsers returning an overflow for 2^64-1 (because it rounds to 2^64) or an incorrect value. I'm not saying it's a showstopper, it's just an unavoidable ugliness if you pick JSON.
For the schema, there are some specific problems with varlink, but also a more generic issue. The specific problems are:
- the schema doesn't include the length of arrays. This makes it hard to
compute in advance lengths and offsets of fields (even ignoring the fact that data is not binary, which I'll get to later)
Do you care in advance?
Yes, once we add for example histograms we would like to include in the schema the size and number of the buckets.
Again, I didn't think this was an issue with the kernel implementation in that the userspace side could determine the schema by the data coming from the kernel, it wouldn't have to "know" about it ahead of time. But I could be wrong.
No, you're right. The C implementations are really just very thin wrappers over JSON. There's very little "Varlink"ness in them.
However the interesting part of the schema are the metadata--the unit, whether something is an instant vs. a cumulative value, the bucket size when we add histograms. These things are obviously not included in the data and must be communicated separately. Userspace tools could also use a schema to validate user requests ("record the halt_poll_fail_ns every second").
All that said, what we _could_ do is serialize the schema as JSON instead of using a binary format
It should be in some standard format. If not, and it sounds like you have looked into it, or at least the userspace side, then that's fine. But you should write up a justification somewhere why you didn't use an existing format (what about the netlink format?)
I guess you're talking about NETLINK_GENERIC, that also has the issue that the schema (the attributes) is not dynamic but rather part of the uAPI. We explicitly don't want them to be stable, they're like tracepoints in that respect and that's why we took ideas from trace-cmd. Anyway, as a start Jing will summarize all these discussions in v10.
Thanks,
Paolo
Hi Fuad,
On Tue, Jun 15, 2021 at 3:03 AM Fuad Tabba tabba@google.com wrote:
Hi Jing,
+int main(int argc, char *argv[]) +{
int max_vm = DEFAULT_NUM_VM, max_vcpu = DEFAULT_NUM_VCPU, ret, i, j;struct kvm_vm **vms;/* Get the number of VMs and VCPUs that would be created for testing. */if (argc > 1) {max_vm = strtol(argv[1], NULL, 0);if (max_vm <= 0)max_vm = DEFAULT_NUM_VM;}if (argc > 2) {max_vcpu = strtol(argv[2], NULL, 0);if (max_vcpu <= 0)max_vcpu = DEFAULT_NUM_VCPU;}/* Check the extension for binary stats */ret = kvm_check_cap(KVM_CAP_BINARY_STATS_FD);TEST_ASSERT(ret >= 0,"Binary form statistics interface is not supported");kvm_check_cap returns the value of KVM_CHECK_EXTENSION, which is 0 if unsupported (-ERROR on an error). The assertion should be for ret > 0.
Made that change locally, and tested it with various configurations (vhe, nvhe), as well as kernel versions (with and without KVM_CAP_BINARY_STATS_FD), and it passes (or fails as expected). Without that fix and with a kernel that doesn't support KVM_CAP_BINARY_STATS_FD, it passes that assertion, but fails later at vcpu_stats_test().
With that fixed: Tested-by: Fuad Tabba tabba@google.com #arm64
Cheers, /fuad
Thanks for the review and testing. Will fix it.
/* Create VMs and VCPUs */vms = malloc(sizeof(vms[0]) * max_vm);TEST_ASSERT(vms, "Allocate memory for storing VM pointers");for (i = 0; i < max_vm; ++i) {vms[i] = vm_create(VM_MODE_DEFAULT,DEFAULT_GUEST_PHY_PAGES, O_RDWR);for (j = 0; j < max_vcpu; ++j)vm_vcpu_add(vms[i], j);}/* Check stats read for every VM and VCPU */for (i = 0; i < max_vm; ++i) {vm_stats_test(vms[i]);for (j = 0; j < max_vcpu; ++j)vcpu_stats_test(vms[i], j);}for (i = 0; i < max_vm; ++i)kvm_vm_free(vms[i]);free(vms);return 0;+} diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index 5c70596dd1b9..83c02cb0ae1e 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -2286,3 +2286,15 @@ unsigned int vm_calc_num_guest_pages(enum vm_guest_mode mode, size_t size) n = DIV_ROUND_UP(size, vm_guest_mode_params[mode].page_size); return vm_adjust_num_guest_pages(mode, n); }
+int vm_get_stats_fd(struct kvm_vm *vm) +{
return ioctl(vm->fd, KVM_GET_STATS_FD, NULL);+}
+int vcpu_get_stats_fd(struct kvm_vm *vm, uint32_t vcpuid) +{
struct vcpu *vcpu = vcpu_find(vm, vcpuid);return ioctl(vcpu->fd, KVM_GET_STATS_FD, NULL);+}
2.32.0.272.g935e593368-goog
Thank, Jing
linux-kselftest-mirror@lists.linaro.org
-
Enrico Weigelt, metux IT consult -
Fuad Tabba -
Greg KH -
Jing Zhang -
Leon Romanovsky -
Paolo Bonzini