On Wed, Jan 31, 2024 at 6:19 AM Marco Elver elver@google.com wrote:
[...]
Signed-off-by: Marco Elver elver@google.com
include/linux/bpf_local_storage.h | 17 ++++++++++++++++- kernel/bpf/bpf_local_storage.c | 14 ++++---------- .../selftests/bpf/progs/cgrp_ls_recursion.c | 2 +- .../selftests/bpf/progs/task_ls_recursion.c | 2 +- 4 files changed, 22 insertions(+), 13 deletions(-)
diff --git a/include/linux/bpf_local_storage.h b/include/linux/bpf_local_storage.h index 173ec7f43ed1..c8cecf7fff87 100644 --- a/include/linux/bpf_local_storage.h +++ b/include/linux/bpf_local_storage.h @@ -130,9 +130,24 @@ bpf_local_storage_map_alloc(union bpf_attr *attr, bool bpf_ma);
struct bpf_local_storage_data * +bpf_local_storage_lookup_slowpath(struct bpf_local_storage *local_storage,
struct bpf_local_storage_map *smap,bool cacheit_lockit);+static inline struct bpf_local_storage_data * bpf_local_storage_lookup(struct bpf_local_storage *local_storage, struct bpf_local_storage_map *smap,
bool cacheit_lockit);
bool cacheit_lockit)+{
struct bpf_local_storage_data *sdata;/* Fast path (cache hit) */sdata = rcu_dereference_check(local_storage->cache[smap->cache_idx],bpf_rcu_lock_held());if (likely(sdata && rcu_access_pointer(sdata->smap) == smap))return sdata;
We have two changes here: 1) inlining; 2) likely() annotation. Could you please include in the commit log how much do the two contribute to the performance improvement?
Thanks, Song
return bpf_local_storage_lookup_slowpath(local_storage, smap, cacheit_lockit);+}
[...]