bpf_object__relocate_core() allocates a CO-RE candidate cache before
processing relocation records. Two instruction offset validation
failures return -EINVAL directly instead of going through the common
cleanup path.
These checks were moved into bpf_object__relocate_core() when
bpf_core_apply_relo() was split, preserving direct returns that were
safe in the previous helper but now bypass cleanup.
Route both failures through the out path so the candidate cache and
target BTF override are released.
Signed-off-by: Luis Vieira <luisflavieira@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260929-libbpf-core-relo-cleanup-v1-1-c9cd0b5775a9@gmail.com
BTF_KIND_DECL_TAG is an attribute on a declaration: with kind_flag=0
the tag name is the argument of a btf_decl_tag(), and with kind_flag=1
it encodes a complete __attribute__.
When doing btf_dump, render both forms at every declaration btf_dump
emits - a record, a record member and a typedef:
struct foo { ... } __attribute__((bar));
struct foo { int a __attribute__((bar)); };
typedef struct { ... } __attribute__((bar)) foo_t;
A decl tag is a standalone type pointing at its target. Build an
index of decl tags in btf_dump_resize(). Keep it as a flat array
sorted by (target ID, tag ID). This allows for a stable emission order
in btf_dump_emit_decl_tags().
btf_dump_emit_decl_tags() binary searches for where the target's
entries would begin and walks tags while the target matches. A record
shares its target with its members, so the component_idx is matched
there.
A record attribute goes after the closing brace, where
__attribute__((packed)) already goes. A member attribute goes after
the bit-field width: clang rejects one between the declarator and the
':'. A typedef takes it after the declarator, so a typedef of an
anonymous record can carry two groups at once, one binding to the
record and one to the typedef.
This closes a generic gap in btf_dump, which silently dropped every
decl tag it was given.
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260929234608.48503-3-ihor.solodrai@linux.dev
btf_dump_mark_referenced() both walks every type added since the last
resize and decides what each one references. Move the walk out to
btf_dump_resize() and hand the function one type at a time, so a second
per-type job can share the same pass instead of adding another one.
No functional change.
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260929234608.48503-2-ihor.solodrai@linux.dev
Clang puts an extern variable that ends up with a section, an extern
map in .maps or an arena global in .addr_space.<N>, into the BTF
DATASEC of that section, next to the object's own definitions and
with offset 0. Unlike .kconfig and .ksyms, that section is a real
allocated ELF section whenever the same object also defines a
variable in it. The static linker handles such externs incorrectly in
two ways.
The output symbol of an unresolved extern gets the destination
section's index and offset instead of staying SHN_UNDEF with a zero
value. For ephemeral sections both are 0, so this only shows up for
allocated sections, where the extern turns into a size-zero NOTYPE
definition. Linking the output again then fails with "conflicting
non-weak symbol" against a strong definition of that variable, or,
against a weak one, keeps the bogus symbol and resolves every
reference to offset 0 of the section. And if the section of the
object with the extern doesn't start at offset 0 of the output
section, the output symbol fails linker_sanity_check_elf_symtab() on
the next link with "invalid extern symbol" before any of that.
When the extern is resolved by a later object in the same link, the
DATASEC entry appended for the extern keeps its placeholder offset,
as linker_append_btf() only updates the size of an existing entry.
The ELF symbol and the BTF secinfo then disagree and skeleton
generation fails with:
Error: Something is wrong for .addr_space.1's variable #1: need offset 0, already at 8.
Keep unresolved externs SHN_UNDEF with a zero value in the output
symbol table, and once an extern is resolved take the offset of its
DATASEC entry from the ELF symbol, which already accounts for weak
vs. strong resolution.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260926000243.2830819-1-andrii@kernel.org
CO-RE relocation of an ldimm64 instruction operates on two instruction
slots. A malformed BPF ELF can end a function after the first slot and
attach a CO-RE relocation to it. libbpf allocates the instruction array
according to the function symbol size, so the shared relocation code would
then access beyond the allocation.
Reject a terminal ldimm64 in libbpf's relocation loop, where the program
length is available, before resolving or applying the relocation. Both
resolved and unresolved relocations validate the absent second slot, and
unresolved relocation poisoning would additionally write past the array.
The in-kernel caller is protected by the verifier's early instruction-stream
check before it applies CO-RE relocations.
Fixes: eacaaed784e2 ("libbpf: Implement enum value-based CO-RE relocations")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/20260914140852.03DA21F0089B@smtp.kernel.org
Link: https://patch.msgid.link/20260917233222.2542500-11-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
CO-RE relocation records can name any instruction offset. When a
relocation cannot be resolved, bpf_core_patch_insn() currently poisons its
target before checking whether that instruction is a valid relocation
target. Malformed metadata can therefore replace jumps, calls, exits,
register-source arithmetic, or non-immediate loads instead of failing at
the relocation step.
Handle poisoning only after the instruction has passed the same class and
operand-form checks used for a resolved relocation. Route invalid forms
through the existing diagnostic and return a hard error. Keep poisoning
supported instructions, including both halves of a plain ldimm64, so an
unresolved relocation in dead code remains valid.
Extend bpf_core_poison_insn() to poison both halves of ldimm64, and return
its status directly from each validated instruction case. This avoids
routing the success path through a common label and leaves the helper free
to report errors.
The shared relocation code applies this restriction to both libbpf and
in-kernel CO-RE.
Fixes: d7a252708dbc ("libbpf: Improve handling of failed CO-RE relocations")
Reported-by: Nicholas Carlini <npc@anthropic.com>
Suggested-by: Nicholas Carlini <npc@anthropic.com>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260917233222.2542500-7-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add support for new kinds to libbpf. BTF_KIND_LOC_PARAM and
BTF_KIND_LOC_PROTO are dedup-able so add support for their
deduplication, whereas since BTF_KIND_LOCSEC contains a unique
offset it is not. LOC_PARAM is considered a primary type
since it contains no external references; LOC_PROTO is a
reference type consisting of LOC_PARAM references so they
are handled in the primary and reference dedup phases
respectively.
For BTF field iteration, BTF_KIND_LOCSEC needs 2 m_offs[] values
for the associated KIND_FUNC and KIND_LOC_PROTO type ids in
each LOCSEC entry.
Add APIs to add location param, location prototypes and location
sections and btf_is_* tests, data accessors for each.
For BTF distillation we add location info to split BTF.
Signed-off-by: Alan Maguire <alan.maguire@oracle.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260924111428.75957-3-alan.maguire@oracle.com
Light skeleton doesn't create maps. The loader prog does at run time.
Teach gen_loader to create a copy of .rodata map with offsets of
functions for a prog, see create_func_ptr_map().
- Such maps are not maps of the object and are not in the loader ctx.
Reserve slots in fd_array after the maps of the object for them and
close their fds when the loader is done or fails. The progs hold them.
Which progs have callx is not known when the loader is initialized,
so reserve for every prog if .text has callx and for the progs that
have callx otherwise.
- Add bpf_gen__func_ptr_map_create() that emits map create, update,
freeze and returns the index in fd_array for ld_imm64
BPF_PSEUDO_MAP_IDX_VALUE.
- User space can set .rodata before loading the skeleton, see
initial_value in bpf_map_desc. Copy it into the copy of the map as
well and store the offsets of functions on top, since user space
doesn't have them.
- Light skeleton can be generated for a target of another endianness.
Store the offsets in the byte order of the object.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260924031042.1690890-15-alexei.starovoitov@gmail.com
PIC code keeps constants with pointers, e.g. vtables, in .data.rel.ro
instead of .rodata. Nothing writes there after relocation.
libbpf treats every .data* section as writable, while the kernel looks
for pointers to functions in frozen read-only maps only.
Make .data.rel.ro and .data.rel.ro.* maps read-only and frozen like
.rodata and collect pointers to functions there.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260924031042.1690890-14-alexei.starovoitov@gmail.com
The kernel recognizes a pointer to a function in frozen read-only array
map by value: byte offset of a static function in the program.
Make it work for tables of functions, struct ops, vtables in .rodata:
static const struct shape_ops square_ops = { 1, area, 10, perimeter };
...
return ops->area(x) * ops->scale;
is compiled into
r3 = *(u64 *)(r1 + 8)
callx r3
where 'area' in .rodata is R_BPF_64_ABS64 relocation against .text.
- Collect relocations in .rodata* that are aligned pointers to
functions. Relocations in data were ignored. The rest still are.
- Append all static functions that the section points to, since any
of them can be called, if the prog refers to the section and has
callx. callx cannot call global functions. Don't pull them in. Such
pointer is NULL.
- Functions have different offsets in different progs, so create a copy
of the map per prog and store the offsets there. The kernel replaces
them with addresses at load and the prog must be the only user of
the map. The map of the section itself is left as-is.
- Progs without callx cannot call them and the kernel doesn't look for
pointers in their data. They keep using the map of the section and
nothing is appended to them.
- The kernel treats any aligned u64 that matches an offset of a static
function as a pointer. Warn if data has one that isn't.
Light skeleton is not supported yet.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260924031042.1690890-13-alexei.starovoitov@gmail.com
A pointer to a static function in data, e.g.
static int (* const handlers[])(void *ctx) = { foo, bar };
is R_BPF_64_ABS64 relocation against .text section symbol with the offset
of the function stored in place. The linker rejects it:
relocation against STT_SECTION in non-exec section is not supported!
so an object with a table of static functions cannot go through
'bpftool gen object', which all selftests do.
Functions move by the offset of input .text in the output section.
Add it to the pointer. Swap bytes if the object is of foreign endianness,
since data sections are kept in the byte order of the object.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260924031042.1690890-12-alexei.starovoitov@gmail.com
bpftool rounds memory-mapped data map sizes to the host page size when
generating a light skeleton. The generated code therefore uses a 64K
mapping size when bpftool runs on a 64K-page host, even if the skeleton
runs on a 4K-page target. The target rejects the oversized map mmap(),
causing failure of loading the light skeleton.
When try to run 64K-page selftests on 4K-page VM, the error message does
not provide the reason about page size.
test_atomics:PASS:atomics skeleton open 0 nsec
test_atomics:FAIL:atomics skeleton load unexpected error: -12 (errno 22)
#15 atomics:FAIL
Pass the original map value size and max entries to the generated code and
round mmap size to the runtime page size in the user-space light
skeleton helpers. This keeps generated light skeletons independent of the
build host page size.
Fixes: d510296d331a ("bpftool: Use syscall/loader program in "prog load" and "gen skeleton" command.")
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Quentin Monnet <qmo@kernel.org>
Link: https://lore.kernel.org/bpf/20260915161450.96249-1-leon.hwang@linux.dev
Add btf_module_allowlist and btf_module_allowlist_cnt fields to
bpf_object_open_opts to limit which kernel module BTFs libbpf is
allowed to load.
When the option is not specified, the existing behavior remains
unchanged. An explicitly specified empty allowlist prevents libbpf from
consulting any kernel module BTFs.
The allowlist limits which kernel module BTFs libbpf will consult
wherever module BTF might be needed.
Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Fuyu Zhao <zhaofuyu@vivo.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260915124104.77287-2-zhaofuyu@vivo.com
A local struct_ops type can contain a bitfield absent from the
corresponding kernel BTF type. bpf_map__init_kern_struct_ops() checks
whether data for an absent local member is zero.
For a bitfield, member->offset contains the bit offset and field width.
bpf_map__init_kern_struct_ops() used the encoded value to calculate the
member data pointer before rejecting the bitfield. The zero-data check
could therefore read outside st_ops->data and crash libbpf.
Reject local bitfields before calculating the member data pointer. Keep
the existing rejection for bitfields in the kernel type.
Fixes: c911fc61a7ce ("libbpf: Skip zeroed or null fields if not found in the kernel type.")
Signed-off-by: Mingpei CAO <caomingpei@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260915162252.473044-2-caomingpei@gmail.com
bpf_program__set_log_buf() checks the existing prog->log_size instead
of the incoming log_size argument when enforcing the UINT_MAX limit.
As a result, an oversized value can be accepted.
The setter also accepts a non-NULL log_buf with a zero log_size, while
bpf_prog_load() rejects mismatched log buffer and size values.
Validate the log buffer and size pair consistently with bpf_prog_load(),
and check the supplied log_size against UINT_MAX.
Signed-off-by: Luis Vieira <luisflavieira@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Mykyta Yatsenko <yatsenko@meta.com>
Link: https://lore.kernel.org/bpf/20260915-libbpf-log-buf-fix-v2-1-2feca3fa4842@gmail.com
When changing BPF program flags, applications often need to just add or
remove a single flag, without touching the existing ones. When using the
bpf_program__set_flags() function, this requires reading the existing
flags and doing bitwise manipulations before resetting the result, which
is slightly awkward to do, and easy to forget.
Add two new convenience helpers, bpf_program__add_flags() and
bpf_program__clear_flags(), which wraps bpf_program__set_flags() in the
bitwise operations required to modify flags without clobbering the
existing ones.
Suggested-by: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/bpf/20260912084109.432834-1-toke@redhat.com
bpftool's -d option is documented to enable bpf_trace_printk() messages
from the generated syscall loader when used with -L as specified in
commit d510296d331a ("bpftool: Use syscall/loader program in "prog load"
and "gen skeleton" command.")
However commit b59e4ce8bcaa ("bpftool: Switch bpf_object__load_xattr()
to bpf_object__load()") changed bpftool to use bpf_object__load() which
call the internal bpf_object_load with extra_log_level to 0 instead of
bpf_object__load_xattr with the user request log_level.
All the plumbing was still there to generate the bpf_trace_printk()
instructions from the generator but was now unreachable because
bpf_gen__init() was called with extra_log_level to 0, leaving
gen->log_level at 0.
This uses the obj->log_level field introduced in commit e0e3ea888c69
("libbpf: Allow passing user log setting through bpf_object_open_opts")
set from reading verifier_logs in do_skeleton(). This preserves both
object-level and explicit load-time logging settings.
Fixes: b59e4ce8bcaa ("bpftool: Switch bpf_object__load_xattr() to bpf_object__load()")
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com>
Link: https://lore.kernel.org/r/20260907170536.9996-1-mahe.tardy@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
init_arena_map_data() used the build-host page size while it opened an
object. This could reject skeleton generation when the build host and
target use different page sizes.
Validate the size during load, before libbpf calculates the arena data
offset. This uses the running kernel page size and rejects an
undersized map before relocation.
This problem manifested when cross compiling BPF selftests for
arm64 (64K page) on x86 (4K page). In this setup skeleton generation
fails.
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260908-libbpf_arena_thing-v1-1-0092c0e0d91b@meta.com
The libbpf section definition modifiers for XDP frags support and
sleepable programs stores the flags bits only in the private section
definition cookie from object open to load time. This has the
unfortunate consequence that API consumers cannot see (or manipulate)
the flag between object open and program load.
In particular, libxdp has special handling of frags-enabled programs to
make them compatible with the dispatcher. This doesn't work on XDP
programs that enable frags through the 'xdp.frags' section definition
because the flag is not visible through bpf_program__flags()[0].
Fix this by changing how libbpf loads the program flags from section
definitions: instead of using the private section definition cookie, add
a setup function to the default section definitions that stores the
flags for sleepable and XDP frags programs in the prog_flags field of
struct bpf_program.
Exposing the flags this way means that any use of
bpf_program__set_flags() will override the flags unless the caller takes
care of updating flags in a non-destructive way. This is unavoidable
with the set-only API, and any user setting flags unconditionally is
already broken in the sense that they will also override any other
current and future flags.
In addition, prog_flags survives program type changes through
bpf_program__set_type(). It is the responsibility of the caller to
ensure the flags are cleared if they are incompatible with the new
program type.
[0] https://github.com/xdp-project/xdp-tools/issues/587
Fixes: 082c4bfba4f7 ("libbpf: Add SEC name for xdp frags programs")
Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260901-libbpf-frags-flags-v3-3-4eb6f14968b0@redhat.com
When compiling libbpf with sanitation, ubsan will report the following:
```
left shift of 1 by 31 places cannot be represented in type 'int'
```
This may lead to undefined behavior according to the compiler implementation,
let's fix it by casting to unsigned counterpart before shifting.
Signed-off-by: Tan Wei <tw19881113@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260828003448.1684064-1-tw19881113@gmail.com
The kernel refuses to attach to a nop10 that crosses a page boundary,
since it can't be atomically rewritten:
/* can_optimize(), arch/x86/kernel/uprobes.c */
/* We can't do cross page atomic writes yet. */
return PAGE_SIZE - (vaddr & ~PAGE_MASK) >= OPT_INSN_SIZE;
Whether the nop10 crosses a page is purely up to the binary layout, so
this does happen in practice. libbpf doesn't check for it and blindly
shifts the uprobe onto the nop10, and the attach then fails with
-ENOTSUPP. Just keep the uprobe on the preceding 1-byte nop in that
case, it works everywhere as a regular int3 uprobe.
Fixes: ee2862439e5c ("libbpf: Change has_nop_combo to work on top of nop10")
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260825150444.31603-1-jiayuan.chen@linux.dev
ringbuf_process_ring() walks the records between the consumer and the
producer with an ordering comparison:
while (cons_pos < prod_pos) {
cons_pos and prod_pos mirror the kernel's ring positions and are
unsigned long here too, so on 32-bit they wrap at 2^32 bytes of traffic.
When producer_pos has wrapped and consumer_pos has not, prod_pos is the
smaller of the two, the loop body never runs and no record is consumed.
Since consumer_pos only advances inside that loop, it never wraps either
and the consumer stops delivering samples for good, with no error
returned to the caller: ring_buffer__poll() keeps reporting zero
records while the kernel side fills up and starts dropping.
Compare the distance instead. The consumer never runs ahead of the
producer, so prod_pos - cons_pos is the amount of unconsumed data and
stays correct across the wrap.
64-bit hosts are unaffected in practice: the counters would need 16 EiB
to wrap. This is the userspace counterpart of the kernel-side walk fixed
in "bpf: Fix pending_pos walk on 32-bit ring position wrap"; a 32-bit
consumer hits whichever of the two comes first.
Signed-off-by: Israel Téllez García <i.tellez@btesa.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814124843.22041-5-i.tellez@btesa.com
Use array_map_mmap_sz() for PERCPU_ARRAY like ARRAY in bpf_map_mmap_sz().
This lets bpf_map__set_value_size() skip mmap(), memcpy(), and munmap()
when the old and new value sizes occupy the same number of pages.
Fix some typos btw:
* mmapble -> mmapable
* satisified -> satisfied
* relocatin -> relocation
* atach_btf_obj_fd -> attach_btf_obj_fd
* len_secnd -> len_second
* precendence -> precedence
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814173206.93082-3-leon.hwang@linux.dev
Enhance bpftool to generate skeletons that properly handle global percpu
variables. The generated skeleton now includes a dedicated structure for
percpu data, allowing users to initialize and access percpu variables more
efficiently.
For global percpu variables, the skeleton now includes a nested
structure, e.g.:
struct test_global_percpu_data {
struct bpf_object_skeleton *skeleton;
struct bpf_object *obj;
struct {
struct bpf_map *percpu;
} maps;
// ...
struct test_global_percpu_data__percpu {
int data;
char run;
struct {
char set;
int i;
int nums[7];
} struct_data;
int nums[7];
} *percpu;
// ...
};
* The "struct test_global_percpu_data__percpu *percpu" points to
initialized data, which is actually "maps.percpu->mmaped".
* Before loading the skeleton, updating the
"struct test_global_percpu_data__percpu *percpu" modifies the initial
value of the corresponding global percpu variables.
* After loading the skeleton, "maps.percpu->mmaped" has been marked as
read-only in libbpf. If users want to update the global percpu
variables, they have to update the "maps.percpu" map instead.
* For lightweight skeleton, "lskel->percpu" will be protected by
"mprotect(p, sz, PROT_READ)".
* For subskeleton, those variables of global percpu data will be
skipped.
Assisted-by: Codex:gpt-5.5-xhigh
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Quentin Monnet <qmo@kernel.org>
Link: https://lore.kernel.org/bpf/20260813152324.97937-7-leon.hwang@linux.dev
Add support for global percpu data in libbpf by adding a new ".percpu"
section, similar to ".data". It enables efficient handling of percpu
global variables in bpf programs.
When generating loader for lightweight skeleton, update the percpu_array
map used for global percpu data using BPF_F_ALL_CPUS, in order to update
values across all CPUs using one value slot.
Unlike global data, the mmaped data for global percpu data will be marked
as read-only after populating the percpu_array map. Thereafter, users can
read those initialized percpu data after loading prog. If they want to
update the percpu data after loading prog, they have to update the
percpu_array map using key=0 instead.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260813152324.97937-6-leon.hwang@linux.dev
btf__find_by_name_kind() searches the base BTF before the split BTF,
so in case of a name collision between base and split it always
returns a base type. Tools that process split BTF may need to restrict
a lookup to the split's own types.
The internal helper btf__find_by_name_kind_own() already does exactly
that. Make it a public API.
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260722233518.778854-5-ihor.solodrai@linux.dev
attach_probe/uprobe-lib and uprobe_autoattach selftests fail with "failed
to resolve full path for libc.so.6" on older non-usrmerged distros, where
libc.so.6 lives under a top-level /lib64 or /lib rather than /usr/lib64 or
/usr/lib. Add /lib64:/lib to the search paths, alongside the existing
/usr/lib64:/usr/lib and Debian multiarch entries.
Fixes: 1ce3a60e3c28 ("libbpf: auto-resolve programs/libraries when necessary for uprobes")
Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/bpf/20260720-selftests-bpf_fixes-v2-3-b450eda93dfe@suse.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
In the previous optimized uprobe fix we changed the syscall
error used for its detection from ENXIO to EPROTO.
Changing related probe_uprobe_syscall detection check.
Fixes: 05738da0efa1 ("libbpf: Add uprobe syscall feature detection")
Fixes: 554ba38456da ("uprobes/x86: Move optimized uprobe from nop5 to nop10")
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://patch.msgid.link/20260703114917.238144-8-jolsa@kernel.org
When btf_parse_elf() is called without a caller-supplied base_btf (i.e.
via the public btf__parse_elf()) and the object file carries a .BTF.base
(distilled base) section, a dist_base_btf object is created and used as
the base of the split BTF built from the .BTF section. Because base_btf
is NULL, the relocation block that would otherwise free and clear
dist_base_btf is skipped, and ownership of dist_base_btf is instead
transferred to the split btf by setting btf->owns_base = true.
That ownership transfer was performed before the fallible btf_ext__new()
call that parses the .BTF.ext section. If .BTF.ext is malformed,
btf_ext__new() fails and the function jumps to the error path, which
frees dist_base_btf directly and then frees btf. Since owns_base is
already set, btf__free(btf) also frees btf->base_btf, which is the same
dist_base_btf object. The result is a use-after-free read followed by a
double free of the base BTF, driven entirely by a crafted object file
(a .BTF + .BTF.base + malformed .BTF.ext combination) passed to
btf__parse_elf(), as used by bpftool, pahole and similar tools.
Transfer ownership only after .BTF.ext has been parsed successfully, so
that any earlier failure leaves dist_base_btf owned solely by the local
cleanup path and it is freed exactly once.
Signed-off-by: Naveed Khan <naveed@digiscrypt.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/178345549172.94179.7948304165383170781@digiscrypt.com
The signed gen_loader used to police its own metadata map from within
BPF: emit_signature_match() read the kernel-cached map->sha[] back
through hardcoded struct bpf_map offsets and compared it against a hash
that compute_sha_update_offsets() baked into the signed instructions,
after a BPF_OBJ_GET_INFO_BY_FD round-trip to populate map->sha[].
The kernel now verifies the metadata at BPF_PROG_LOAD time by folding
the frozen contents of the loader's exclusive fd_array maps into the
signature, so the loader no longer checks anything itself. Generated
loaders thus carry no verification logic of their own anymore: Nothing
in the signing chain depends on emitted loader bytecode doing the right
thing.
On the loading side, skel_internal.h now sets fd_array_cnt for a signed
load so the kernel scans fd_array for the exclusive metadata map -
still frozen, as the kernel requires - and the BPF_OBJ_GET_INFO_BY_FD
round-trip to populate map->sha[] is gone. The struct bpf_map layout
BUILD_BUG_ON()s on the kernel side are removed as well: they only
pinned the ABI for the in-BPF read of map->sha[] that is no longer
needed. Same for the map->excl member. Note: gen_hash is retained; it
still marks a loader as signed so an untrusted host cannot re-dimension
maps or override initial values now covered by the signature.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260708075343.358712-4-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_object__probe_loading() tries to load trivial SOCKET_FILTER and
TRACEPOINT programs to verify the BPF environment works. When a
BPF token is in use with restricted program type permissions, these
probe loads may fail because the token does not allow the specific
program types, even though BPF loading is perfectly functional.
Fix by skipping the probe when a token FD is present: BPF token
creation itself proves the kernel has a working BPF subsystem.
Real BPF issues will be caught during actual program and map loading.
Signed-off-by: Yuan Chen <chenyuan@kylinos.cn>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260610145059.113412-2-chenyuan_fl@163.com
Building libbpf with -Wall (as happens via bpftool's bootstrap build)
surfaces ~120 -Wformat warnings where pr_warn/pr_debug format specifiers
don't match their argument types: %d for __u32/Elf64_Word, %u for signed
ints, %zd for size_t, %ld for unsigned long, and %x/%lx/%llx applied to
signed values.
Match each specifier to its argument's type where a correctly-signed
specifier exists (%d<->%u, %ld->%lu, %zd->%zu). For hex conversions,
which have no signed form, cast the argument instead (%x->(unsigned),
%lx->(unsigned long), %llx->(unsigned long long)). No functional change.
Note, the fdinfo map_flags sscanf used %i into a __u32 *, which warns.
The kernel prints map_flags as hex ("map_flags:\t%#x\n" in
bpf_map_show_fdinfo(), unchanged since the field was added to fdinfo), so
switch the conversion to %x: it parses the 0x-prefixed value and expects
unsigned int *, matching the destination, so the warning is gone with no
cast.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/r/20260624204946.2901178-1-andrii@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Adding bpf_program__attach_tracing_multi function for attaching
tracing program to multiple functions.
struct bpf_link *
bpf_program__attach_tracing_multi(const struct bpf_program *prog,
const char *pattern,
const struct bpf_tracing_multi_opts *opts);
User can specify functions to attach with 'pattern' argument that
allows wildcards (*?' supported) or provide BTF ids of functions
in array directly via opts argument. These options are mutually
exclusive.
When using BTF ids, user can also provide cookie value for each
provided id/function, that can be retrieved later in bpf program
with bpf_get_attach_cookie helper. Each cookie value is paired with
provided BTF id with the same array index.
Adding support to auto attach programs with following sections:
fsession.multi/<pattern>
fsession.multi.s/<pattern>
fentry.multi/<pattern>
fexit.multi/<pattern>
fentry.multi.s/<pattern>
fexit.multi.s/<pattern>
The provided <pattern> is used as 'pattern' argument in
bpf_program__attach_kprobe_multi_opts function.
The <pattern> allows to specify optional kernel module name with
following syntax:
<module>:<function_pattern>
In order to attach tracing_multi link to a module functions:
- program must be loaded with 'module' btf fd
(in attr::attach_btf_obj_fd)
- bpf_program__attach_tracing_multi must either have
pattern with module spec or BTF ids from the module
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260606123955.345967-21-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Adding btf_type_is_traceable_func function to perform same checks
as the kernel's btf_distill_func_proto function to prevent attachment
on some of the functions.
Exporting the function via libbpf_internal.h because it will be used
by benchmark test in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260606123955.345967-20-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Adding support to use session attachment with tracing_multi link.
Adding new BPF_TRACE_FSESSION_MULTI program attach type, that follows
the BPF_TRACE_FSESSION behaviour but on the tracing_multi link.
Such program is called on entry and exit of the attached function
and allows to pass cookie value from entry to exit execution.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260606123955.345967-16-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Adding new link to allow to attach program to multiple function
BTF IDs. The link is represented by struct bpf_tracing_multi_link.
To configure the link, new fields are added to bpf_attr::link_create
to pass array of BTF IDs;
struct {
__aligned_u64 ids;
__u32 cnt;
} tracing_multi;
Each BTF ID represents function (BTF_KIND_FUNC) that the link will
attach bpf program to.
We use previously added bpf_trampoline_multi_attach/detach functions
to attach/detach the link.
The linkinfo/fdinfo callbacks will be implemented in following changes.
Note this is supported only for archs (x86_64) with ftrace direct and
have single ops support.
CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS &&
CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
Note using sort_r (instead of plain sort) in check_dup_ids, because we
will use the swap callback in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260606123955.345967-14-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Adding new program attach types multi tracing attachment:
BPF_TRACE_FENTRY_MULTI
BPF_TRACE_FEXIT_MULTI
and their base support in verifier code.
Programs with such attach type will use specific link attachment
interface coming in following changes.
This was suggested by Andrii some (long) time ago and turned out
to be easier than having special program flag for that.
Bpf programs with such types have 'bpf_multi_func' function set as
their attach_btf_id and keep module reference when it's specified
by attach_prog_fd.
They are also accepted as sleepable programs during verification,
and the real validation for specific BTF_IDs/functions will happen
during the multi link attachment in following changes.
Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260606123955.345967-11-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Andrii Nakryiko
Luis Vieira
Ihor Solodrai
Kumar Kartikeya Dwivedi
Alan Maguire
Alexei Starovoitov
Leon Hwang
Fuyu Zhao
Martin KaFai Lau
Mingpei CAO
Toke Høiland-Jørgensen
Mahe Tardy
Mykyta Yatsenko
Sudip Mukherjee
Tw
Jiayuan Chen
Darren Carreras
Israel Téllez García
Ricardo B. Marlière
Jiri Olsa
Naveed Khan
Daniel Borkmann
Yuan Chen