Similar to the previous fix for lru_hash maps, the lru_percpu_hash map implementation also suffers from unnecessary eviction when updating existing elements.
When updating a key that already exists in a full lru_percpu_hash map, the current code path calls prealloc_lru_pop() before checking for the existing key (unless map_flags is BPF_EXIST). This can evict an unrelated element even though the update is just modifying the per-CPU value of an existing entry.
Fix this by looking up the key first. If found, update the per-CPU value in-place using pcpu_copy_value(), refresh the LRU reference, and return early. Only proceed with node allocation if the key does not exist.
Fixes: 8f8449384ec3 ("bpf: Add BPF_MAP_TYPE_LRU_PERCPU_HASH") Signed-off-by: Leon Hwang leon.hwang@linux.dev --- kernel/bpf/hashtab.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index fb624aa76573..af54fc3a9ba9 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -1358,6 +1358,28 @@ static long __htab_lru_percpu_map_update_elem(struct bpf_map *map, void *key, b = __select_bucket(htab, hash); head = &b->head;
+ ret = htab_lock_bucket(b, &flags); + if (ret) + goto err_lock_bucket; + + l_old = lookup_elem_raw(head, hash, key, key_size); + + ret = check_flags(htab, l_old, map_flags); + if (ret) + goto err; + + if (l_old) { + bpf_lru_node_set_ref(&l_old->lru_node); + /* per-cpu hash map can update value in-place */ + pcpu_copy_value(htab, htab_elem_get_ptr(l_old, key_size), + value, onallcpus); + } + + htab_unlock_bucket(b, flags); + + if (l_old) + return 0; + /* For LRU, we need to alloc before taking bucket's * spinlock because LRU's elem alloc may need * to remove older elem from htab and this removal