This patch adds necessary infrastructure to attach a struct_ops
map to a cgroup. The initial need was to support migrating
the legacy BPF_PROG_TYPE_SOCK_OPS to a struct_ops.
Recently, there are other struct_ops use cases that
need to attach struct_ops to a cgroup. For example,
the recent BPF OOM and memcg discussion in LSFMMBPF 2026.
The motivation is to create a consistent expectation
for attaching struct_ops to cgroup instead of each subsystem
creating its own infrastructure. This logic includes
hierarchy expectation, ordering expectation,
attachment API, and rcu gp.
There is already an existing implementation for attaching
multiple bpf progs to a cgroup. There are also tools
built around it for querying. Attaching a struct_ops map
(which is a group of bpf programs) could also adhere to
a similar API and potentially reuse most of the existing
implementation.
A couple of ideas have been tried. One of them
is to use mprog.c. In terms of the amount of changes,
I eventually came to the same conclusion as in
commit 120933984460 ("bpf: Implement mprog API on top of existing cgroup progs").
I then shifted the focus to reusing the current
{update,compute,activate,purge}_effective_progs() which has
the main logic that implements the mprog API.
Since then, I tried to add a 'struct cgroup *cgroup' member
to the existing 'struct bpf_struct_ops_link' and link_create
will create a 'struct bpf_struct_ops_link' object to be stored
in the pl->link. This turns out to have more changes on
both cgroup.c and bpf_struct_ops.c than I like.
This patch directly reuses the 'struct bpf_cgroup_link' which
cgroup.c already understands. Add 'struct bpf_map *map'
to 'struct bpf_cgroup_link'. In the future, as more subsystems
are extended by struct_ops, we may consider to make
'struct bpf_map *map' as a primary citizen of a link
like 'struct bpf_prog *prog' and directly add
'struct bpf_map *map' to the generic 'struct bpf_link'.
The pl->link could be the traditional 'prog' link or the
new 'map' link. The places that need to handle them differently
have already been refactored into the new prog_list_*() added in
the earlier patch. In those new prog_list_*(), this patch will
check "pl->link && pl->link->map", learn that it is a 'map' link
and handle it correctly.
The bpf_prog_array also needs to handle that its item can store
the traditional 'prog' or it can store a struct_ops map.
The places that need to handle them differently have also
been refactored into the new bpf_cgroup_array_*() added
in the earlier patch. The two differences are:
- different sentinel (dummy_bpf_prog in prog vs cfi_stub in struct_ops)
- the array for struct_ops may need to go through different
rcu gp.
The bpf_cgroup_array_*() functions use the cgroup_bpf_attach_type (ie atype)
to distinguish the array is storing prog or storing struct_ops map.
This patch also implements a separate struct bpf_link_ops
"cgroup_struct_ops_link_ops" to have a separate link_ops implementation
that only handles the cgroup's struct_ops link.
Questions:
- Although this patch did not change it, it is not obvious to me how
the replace_effective_progs() and purge_effective_progs() handle
cases when there are existing BPF_F_PREORDER progs attached
in the hlist.
Misc notes:
- CGROUP_TCP_SOCK_OPS is added to the 'enum cgroup_bpf_attach_type'.
The actual implementation of the tcp_bpf_ops (a struct_ops)
will be added in the next patch.
- free_after_mult_rcu_gp is added to 'struct bpf_struct_ops' such that
the bpf_prog_array can have a mix of sleepable and
non-sleepable prog in a struct_ops. This can tell
how the bpf_prog_array should be freed.
- For a struct_ops that supports cgroup attachment, it does not need to
implement its own reg/unreg function. reg/unreg to a cgroup is
done by the common infrastructure added in this patch.
- The cgroup's struct_ops link only supports BPF_F_ALLOW_MULTI.
This is enforced internally in cgroup_bpf_struct_ops_attach.
This should be consistent with the current prog's link
behavior in cgroup_bpf_link_attach.
In the future, we may allow each subsystem to choose differently.
- A cgroup_atype member is added to 'struct bpf_struct_ops'.
When a subsystem struct_ops needs to support cgroup attachment,
it needs to add a value to 'enum cgroup_bpf_attach_type'
and then assign it to the newly added cgroup_atype member
in the bpf_struct_ops.
- During LINK_CREATE in syscall, the patch uses the same
BPF_STRUCT_OPS (in attr->link_create.attach_type).
The bpf_struct_ops_link_create learns the map and
from the map it learns the st_ops. If the st_ops->cgroup_atype
is not 0, it will create a cgroup's link.
- When a subsystem registers a struct_ops that supports cgroup
attachment, the struct_ops infrastructure will also ask the
cgroup infrastructure to remember a few things. This is done
by calling cgroup_bpf_struct_ops_register().
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260917200542.3689605-10-ameryhung@gmail.com
A local struct_ops type can contain a bitfield absent from the
corresponding kernel BTF type. bpf_map__init_kern_struct_ops() checks
whether data for an absent local member is zero.
For a bitfield, member->offset contains the bit offset and field width.
bpf_map__init_kern_struct_ops() used the encoded value to calculate the
member data pointer before rejecting the bitfield. The zero-data check
could therefore read outside st_ops->data and crash libbpf.
Reject local bitfields before calculating the member data pointer. Keep
the existing rejection for bitfields in the kernel type.
Fixes: c911fc61a7ce ("libbpf: Skip zeroed or null fields if not found in the kernel type.")
Signed-off-by: Mingpei CAO <caomingpei@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Amery Hung <ameryhung@gmail.com>
Link: https://lore.kernel.org/bpf/20260915162252.473044-2-caomingpei@gmail.com
bpf_program__set_log_buf() checks the existing prog->log_size instead
of the incoming log_size argument when enforcing the UINT_MAX limit.
As a result, an oversized value can be accepted.
The setter also accepts a non-NULL log_buf with a zero log_size, while
bpf_prog_load() rejects mismatched log buffer and size values.
Validate the log buffer and size pair consistently with bpf_prog_load(),
and check the supplied log_size against UINT_MAX.
Signed-off-by: Luis Vieira <luisflavieira@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Mykyta Yatsenko <yatsenko@meta.com>
Link: https://lore.kernel.org/bpf/20260915-libbpf-log-buf-fix-v2-1-2feca3fa4842@gmail.com
When changing BPF program flags, applications often need to just add or
remove a single flag, without touching the existing ones. When using the
bpf_program__set_flags() function, this requires reading the existing
flags and doing bitwise manipulations before resetting the result, which
is slightly awkward to do, and easy to forget.
Add two new convenience helpers, bpf_program__add_flags() and
bpf_program__clear_flags(), which wraps bpf_program__set_flags() in the
bitwise operations required to modify flags without clobbering the
existing ones.
Suggested-by: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/bpf/20260912084109.432834-1-toke@redhat.com
bpftool's -d option is documented to enable bpf_trace_printk() messages
from the generated syscall loader when used with -L as specified in
commit d510296d331a ("bpftool: Use syscall/loader program in "prog load"
and "gen skeleton" command.")
However commit b59e4ce8bcaa ("bpftool: Switch bpf_object__load_xattr()
to bpf_object__load()") changed bpftool to use bpf_object__load() which
call the internal bpf_object_load with extra_log_level to 0 instead of
bpf_object__load_xattr with the user request log_level.
All the plumbing was still there to generate the bpf_trace_printk()
instructions from the generator but was now unreachable because
bpf_gen__init() was called with extra_log_level to 0, leaving
gen->log_level at 0.
This uses the obj->log_level field introduced in commit e0e3ea888c69
("libbpf: Allow passing user log setting through bpf_object_open_opts")
set from reading verifier_logs in do_skeleton(). This preserves both
object-level and explicit load-time logging settings.
Fixes: b59e4ce8bcaa ("bpftool: Switch bpf_object__load_xattr() to bpf_object__load()")
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Mahe Tardy <mahe.tardy@gmail.com>
Link: https://lore.kernel.org/r/20260907170536.9996-1-mahe.tardy@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
init_arena_map_data() used the build-host page size while it opened an
object. This could reject skeleton generation when the build host and
target use different page sizes.
Validate the size during load, before libbpf calculates the arena data
offset. This uses the running kernel page size and rejects an
undersized map before relocation.
This problem manifested when cross compiling BPF selftests for
arm64 (64K page) on x86 (4K page). In this setup skeleton generation
fails.
Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260908-libbpf_arena_thing-v1-1-0092c0e0d91b@meta.com
Add explicit padding before launch_time so xsk_tx_metadata has the same
layout on 32-bit and 64-bit systems.
On several architectures (csky, i386, nios2, m65k, openrisc, sh), the old
native 32-bit layout put launch_time at offset 12 and had a natural size of
20 bytes. Using sizeof(struct xsk_tx_metadata) as tx_metadata_len was already
rejected because the length must be a multiple of eight, so the
straightforward use of the interface was broken on those ABIs. Userspace
could still register a padded length of 24 bytes, though; mixing the old and
new layouts then silently reads launch_time from the wrong offset and
misprograms packet launch times. This intentionally replaces that
incompatible layout because the affected architectures are unlikely to
have any notable users. (x86_64 and arm64 have the most users and are _not_
affected)
Fixes: ca4419f15abd ("xsk: Add launch time hardware offload support to XDP Tx metadata")
Reviewed-by: Simon Horman <horms@kernel.org>
Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Link: https://patch.msgid.link/20260819160535.1472459-2-sdf@fomichev.me
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
The libbpf section definition modifiers for XDP frags support and
sleepable programs stores the flags bits only in the private section
definition cookie from object open to load time. This has the
unfortunate consequence that API consumers cannot see (or manipulate)
the flag between object open and program load.
In particular, libxdp has special handling of frags-enabled programs to
make them compatible with the dispatcher. This doesn't work on XDP
programs that enable frags through the 'xdp.frags' section definition
because the flag is not visible through bpf_program__flags()[0].
Fix this by changing how libbpf loads the program flags from section
definitions: instead of using the private section definition cookie, add
a setup function to the default section definitions that stores the
flags for sleepable and XDP frags programs in the prog_flags field of
struct bpf_program.
Exposing the flags this way means that any use of
bpf_program__set_flags() will override the flags unless the caller takes
care of updating flags in a non-destructive way. This is unavoidable
with the set-only API, and any user setting flags unconditionally is
already broken in the sense that they will also override any other
current and future flags.
In addition, prog_flags survives program type changes through
bpf_program__set_type(). It is the responsibility of the caller to
ensure the flags are cleared if they are incompatible with the new
program type.
[0] https://github.com/xdp-project/xdp-tools/issues/587
Fixes: 082c4bfba4f7 ("libbpf: Add SEC name for xdp frags programs")
Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260901-libbpf-frags-flags-v3-3-4eb6f14968b0@redhat.com
When compiling libbpf with sanitation, ubsan will report the following:
```
left shift of 1 by 31 places cannot be represented in type 'int'
```
This may lead to undefined behavior according to the compiler implementation,
let's fix it by casting to unsigned counterpart before shifting.
Signed-off-by: Tan Wei <tw19881113@gmail.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260828003448.1684064-1-tw19881113@gmail.com
The kernel refuses to attach to a nop10 that crosses a page boundary,
since it can't be atomically rewritten:
/* can_optimize(), arch/x86/kernel/uprobes.c */
/* We can't do cross page atomic writes yet. */
return PAGE_SIZE - (vaddr & ~PAGE_MASK) >= OPT_INSN_SIZE;
Whether the nop10 crosses a page is purely up to the binary layout, so
this does happen in practice. libbpf doesn't check for it and blindly
shifts the uprobe onto the nop10, and the attach then fails with
-ENOTSUPP. Just keep the uprobe on the preceding 1-byte nop in that
case, it works everywhere as a regular int3 uprobe.
Fixes: ee2862439e5c ("libbpf: Change has_nop_combo to work on top of nop10")
Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260825150444.31603-1-jiayuan.chen@linux.dev
Add new bpf_skb_adjust_room() decapsulation flags:
- BPF_F_ADJ_ROOM_DECAP_L4_GRE
- BPF_F_ADJ_ROOM_DECAP_L4_UDP
- BPF_F_ADJ_ROOM_DECAP_IPXIP4
- BPF_F_ADJ_ROOM_DECAP_IPXIP6
These flags let BPF programs describe which tunnel layer is being
removed, so later changes can update tunnel-related GSO state
accordingly during decapsulation.
This patch only introduces the UAPI flag definitions and helper
documentation.
Co-developed-by: Max Tottenham <mtottenh@akamai.com>
Co-developed-by: Anna Glasgall <aglasgal@akamai.com>
Signed-off-by: Max Tottenham <mtottenh@akamai.com>
Signed-off-by: Anna Glasgall <aglasgal@akamai.com>
Signed-off-by: Nick Hudson <nhudson@akamai.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Reviewed-by: Willem de Bruijn <willemb@google.com>
Link: https://lore.kernel.org/bpf/20260812083115.73100-4-nhudson@akamai.com
ringbuf_process_ring() walks the records between the consumer and the
producer with an ordering comparison:
while (cons_pos < prod_pos) {
cons_pos and prod_pos mirror the kernel's ring positions and are
unsigned long here too, so on 32-bit they wrap at 2^32 bytes of traffic.
When producer_pos has wrapped and consumer_pos has not, prod_pos is the
smaller of the two, the loop body never runs and no record is consumed.
Since consumer_pos only advances inside that loop, it never wraps either
and the consumer stops delivering samples for good, with no error
returned to the caller: ring_buffer__poll() keeps reporting zero
records while the kernel side fills up and starts dropping.
Compare the distance instead. The consumer never runs ahead of the
producer, so prod_pos - cons_pos is the amount of unconsumed data and
stays correct across the wrap.
64-bit hosts are unaffected in practice: the counters would need 16 EiB
to wrap. This is the userspace counterpart of the kernel-side walk fixed
in "bpf: Fix pending_pos walk on 32-bit ring position wrap"; a 32-bit
consumer hits whichever of the two comes first.
Signed-off-by: Israel Téllez García <i.tellez@btesa.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814124843.22041-5-i.tellez@btesa.com
Use array_map_mmap_sz() for PERCPU_ARRAY like ARRAY in bpf_map_mmap_sz().
This lets bpf_map__set_value_size() skip mmap(), memcpy(), and munmap()
when the old and new value sizes occupy the same number of pages.
Fix some typos btw:
* mmapble -> mmapable
* satisified -> satisfied
* relocatin -> relocation
* atach_btf_obj_fd -> attach_btf_obj_fd
* len_secnd -> len_second
* precendence -> precedence
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814173206.93082-3-leon.hwang@linux.dev
Enhance bpftool to generate skeletons that properly handle global percpu
variables. The generated skeleton now includes a dedicated structure for
percpu data, allowing users to initialize and access percpu variables more
efficiently.
For global percpu variables, the skeleton now includes a nested
structure, e.g.:
struct test_global_percpu_data {
struct bpf_object_skeleton *skeleton;
struct bpf_object *obj;
struct {
struct bpf_map *percpu;
} maps;
// ...
struct test_global_percpu_data__percpu {
int data;
char run;
struct {
char set;
int i;
int nums[7];
} struct_data;
int nums[7];
} *percpu;
// ...
};
* The "struct test_global_percpu_data__percpu *percpu" points to
initialized data, which is actually "maps.percpu->mmaped".
* Before loading the skeleton, updating the
"struct test_global_percpu_data__percpu *percpu" modifies the initial
value of the corresponding global percpu variables.
* After loading the skeleton, "maps.percpu->mmaped" has been marked as
read-only in libbpf. If users want to update the global percpu
variables, they have to update the "maps.percpu" map instead.
* For lightweight skeleton, "lskel->percpu" will be protected by
"mprotect(p, sz, PROT_READ)".
* For subskeleton, those variables of global percpu data will be
skipped.
Assisted-by: Codex:gpt-5.5-xhigh
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Quentin Monnet <qmo@kernel.org>
Link: https://lore.kernel.org/bpf/20260813152324.97937-7-leon.hwang@linux.dev
Add support for global percpu data in libbpf by adding a new ".percpu"
section, similar to ".data". It enables efficient handling of percpu
global variables in bpf programs.
When generating loader for lightweight skeleton, update the percpu_array
map used for global percpu data using BPF_F_ALL_CPUS, in order to update
values across all CPUs using one value slot.
Unlike global data, the mmaped data for global percpu data will be marked
as read-only after populating the percpu_array map. Thereafter, users can
read those initialized percpu data after loading prog. If they want to
update the percpu data after loading prog, they have to update the
percpu_array map using key=0 instead.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260813152324.97937-6-leon.hwang@linux.dev
btf__find_by_name_kind() searches the base BTF before the split BTF,
so in case of a name collision between base and split it always
returns a base type. Tools that process split BTF may need to restrict
a lookup to the split's own types.
The internal helper btf__find_by_name_kind_own() already does exactly
that. Make it a public API.
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260722233518.778854-5-ihor.solodrai@linux.dev
attach_probe/uprobe-lib and uprobe_autoattach selftests fail with "failed
to resolve full path for libc.so.6" on older non-usrmerged distros, where
libc.so.6 lives under a top-level /lib64 or /lib rather than /usr/lib64 or
/usr/lib. Add /lib64:/lib to the search paths, alongside the existing
/usr/lib64:/usr/lib and Debian multiarch entries.
Fixes: 1ce3a60e3c28 ("libbpf: auto-resolve programs/libraries when necessary for uprobes")
Signed-off-by: Ricardo B. Marlière <rbm@suse.com>
Acked-by: Ihor Solodrai <ihor.solodrai@linux.dev>
Link: https://lore.kernel.org/bpf/20260720-selftests-bpf_fixes-v2-3-b450eda93dfe@suse.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
BPF_FIB_LOOKUP_VLAN resolves a VLAN egress. The reverse is also
useful: an XDP program receiving a VLAN-tagged frame on a physical
device wants the lookup to behave as if the packet had arrived on the
corresponding VLAN subinterface, so iif-based policy routing and VRF
table selection use the right ingress.
Add BPF_FIB_LOOKUP_VLAN_INPUT. When set, params->h_vlan_proto and
params->h_vlan_TCI are read as an input VLAN tag and the matching VLAN
device of params->ifindex is resolved with __vlan_find_dev_deep_rcu().
The device must be up and in the same network namespace as
params->ifindex (a VLAN device can be moved to another netns while
registered on its parent; receive would deliver into that other
namespace, which a lookup here cannot represent). If params->ifindex
is itself a VLAN device, its inner (QinQ) subinterface is matched.
For a bond or team, a tag on a port matches no device and returns
NOT_FWDED; pass the master's ifindex.
The lookup then runs with the resolved device as the ingress;
params->ifindex itself is not modified on the input side. When the
resolved device is enslaved to a VRF, both the full lookup (via the
l3mdev rule) and BPF_FIB_LOOKUP_DIRECT (via l3mdev_fib_table_rcu())
select the VRF's table from the resolved ingress. That follows from
feeding the resolved device to the flow as the ingress
(fl4.flowi4_iif = dev->ifindex), which is what makes l3mdev resolve
the VRF master from the subinterface rather than from
params->ifindex.
The two failure classes get different treatment on purpose. A
h_vlan_proto other than 802.1Q/802.1ad is API misuse and returns
-EINVAL, since it would otherwise reach the WARN in vlan_proto_idx()
with a program-controlled value. An unmatched VID, a device that is
down, or one in another namespace is a data outcome and returns
BPF_FIB_LKUP_RET_NOT_FWDED, matching the DIRECT path when
fib_get_table() finds no table and mirroring real ingress, where the
receive path drops such frames. A VID of 0 (a priority tag) is looked
up literally and normally fails the same way; receive instead
processes such frames untagged, so callers should not set the flag for
priority tags. Proceeding on the physical device for any of these
would be fail-open for the policy-routing cases above.
The h_vlan fields share a union with tbid, so the flag cannot be
combined with BPF_FIB_LOOKUP_TBID. It describes ingress, so it also
cannot be combined with BPF_FIB_LOOKUP_OUTPUT. Both combinations
return -EINVAL; restricting now keeps a later relaxation backward
compatible. Combining with BPF_FIB_LOOKUP_VLAN is allowed: the tag is
consumed on the ingress side and the egress tag is written on
success.
Under !CONFIG_VLAN_8021Q the __vlan_find_dev_deep_rcu() stub returns
NULL, so every lookup with a valid proto returns NOT_FWDED, which is
correct since no VLAN device can exist.
Suggested-by: Toke Høiland-Jørgensen <toke@redhat.com>
Signed-off-by: Avinash Duduskar <avinash.duduskar@gmail.com>
Reviewed-by: Toke Høiland-Jørgensen <toke@redhat.com>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://lore.kernel.org/bpf/20260713162305.1237211-3-avinash.duduskar@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_fib_lookup() returns the FIB-resolved egress ifindex straight
from the fib result. When the egress is a VLAN device, the returned
ifindex is the VLAN netdev's, which has no XDP xmit handler; XDP
programs that want to forward the frame (e.g. xdp-forward) must
instead target the underlying physical device and push the VLAN tag
themselves. Today the program has no way to learn either the
underlying ifindex or the VLAN tag without maintaining its own
VLAN-to-ifindex map in userspace and refreshing it on netlink
events.
Add BPF_FIB_LOOKUP_VLAN. When the caller sets this flag and the fib
result is a VLAN device whose immediate parent is a real (non-VLAN)
device in the same network namespace, populate the existing output
fields params->h_vlan_proto and params->h_vlan_TCI from the VLAN
device and replace params->ifindex with the parent's ifindex.
params->h_vlan_TCI carries the VID only, with PCP and DEI bits zero; a
consumer wanting to set egress priority writes PCP itself.
params->smac is the VLAN device's own address, which can differ from
the parent's.
Only the immediate parent is resolved, via vlan_dev_priv(dev)->real_dev
and not vlan_dev_real_dev(), which walks to the bottom of a stack. When
the immediate parent is not a real device in the same namespace, the
lookup returns BPF_FIB_LKUP_RET_VLAN_FAILURE and leaves params->ifindex
at the input. This covers a stacked VLAN (QinQ), where the immediate
parent is itself a VLAN device and one h_vlan_proto/h_vlan_TCI pair
cannot describe two tags, and a parent in another network namespace (a
VLAN device can be moved while its parent stays), whose ifindex would
be meaningless in the caller's namespace. A program that wants the
VLAN device's own ifindex re-issues the lookup, with a re-initialized
params, without BPF_FIB_LOOKUP_VLAN, so the unreducible case stays
distinct from a physical egress. That distinction matters for XDP: a
program cannot xmit on a VLAN device, so a success carrying the VLAN
ifindex would make it redirect to a device with no ndo_xdp_xmit and
drop the frame at xdp_do_flush(). The swap and the vlan fields are
written only on the reduce path; other output fields keep their
existing behaviour, so a frag-needed result still reports the route
mtu in params->mtu_result.
BPF_FIB_LOOKUP_VLAN is only useful to XDP, which cannot redirect to a
VLAN device. A tc program can redirect to the VLAN device directly, so
bpf_skb_fib_lookup() rejects the flag with -EINVAL; bpf_xdp_fib_lookup()
accepts it. When the flag is not set, behaviour is unchanged:
h_vlan_proto and h_vlan_TCI are zeroed and ifindex is left at the FIB
result.
The new block is compiled only under CONFIG_VLAN_8021Q since
vlan_dev_priv() is not defined otherwise; without that config
is_vlan_dev() is constant false and the flag is accepted but never
acts. That is safe because no VLAN device can exist there, so every
egress is already physical.
This lets an XDP redirect target the physical device and learn the
tag to push in a single lookup, which xdp-forward's optional VLAN
mode (xdp-project/xdp-tools#504) wants from the kernel side.
The helper's input semantics are unchanged; the reverse direction
(supplying a tag as lookup input) is added in the following patch.
Suggested-by: Toke Høiland-Jørgensen <toke@redhat.com>
Signed-off-by: Avinash Duduskar <avinash.duduskar@gmail.com>
Reviewed-by: Toke Høiland-Jørgensen <toke@redhat.com>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Acked-by: David Ahern <dsahern@kernel.org>
Link: https://lore.kernel.org/bpf/20260713162305.1237211-2-avinash.duduskar@gmail.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
In the previous optimized uprobe fix we changed the syscall
error used for its detection from ENXIO to EPROTO.
Changing related probe_uprobe_syscall detection check.
Fixes: 05738da0efa1 ("libbpf: Add uprobe syscall feature detection")
Fixes: 554ba38456da ("uprobes/x86: Move optimized uprobe from nop5 to nop10")
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://patch.msgid.link/20260703114917.238144-8-jolsa@kernel.org
Add a new file descriptor sentinel FD_FAILFS_ROOT following
FD_PIDFS_ROOT and FD_NSFS_ROOT and teach fchdir() to accept it. A
process calling fchdir(FD_FAILFS_ROOT) moves its working directory
into failfs. Every AT_FDCWD-relative lookup afterwards fails with
EOPNOTSUPP including "." and ".." and getcwd() reports the working
directory as unreachable from the process root by returning a path
prefixed with "(unreachable)". Lookups relative to explicit directory
file descriptors are unaffected.
The sentinel is the only way in. No privilege or gating is required.
Setting the working directory to a directory in which every operation
fails grants nothing and loses nothing that closing file descriptors
couldn't lose. An unlinked working directory behaves the same way today
modulo errno. The working directory also plays no role in confining ".."
resolution so no boundary is weakened.
Link: https://patch.msgid.link/20260724-work-failfs-v2-2-485dabbae185@kernel.org
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Linux 7.3 declares the fpsimd registers in the arm64 sigcontext as
__u128 instead of __uint128_t, see kernel commit d4bf9e08d412 ("arm64:
uapi: Use __u128 instead of __uint128_t in UAPI headers"). The private
linux/types.h in include/ shadows the kernel's linux/types.h while
building libbpf, so with glibc, whose signal.h pulls in
asm/sigcontext.h, the build fails with an unknown type name __u128.
Add the __s128 and __u128 typedefs as the kernel's uapi and tools
headers define them.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Automatically comment on and close master-targeted pull requests that
touch kernel-synchronized libbpf paths. Keep mirror-only and unknown
paths open, with a github-only label for maintainer overrides.
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
This was copied over from the Linux kernel's LICENSES/preferred/LGPL-2.1
a long time ago, and did not have the newer SPDX 3.0 identifiers (where
LGPL-2.1 is deprecated in favor of LGPL-2.1-only) and has the old FSF
address listed (FSF has gone remote only:
https://www.fsf.org/about/contact/)
Copy in the new text from commit 89373f5695dc918a0118fa71ee4dc423bc8c8476
and refer to it using the new LGPL-2.1-only identifier
The source code itself needs to be modified in the Linux kernel tree,
that will be done separate from this but fixing the files here first
will make it easier for downstream packagers of libbpf
Signed-off-by: Michel Lind <salimma@fedoraproject.org>
When btf_parse_elf() is called without a caller-supplied base_btf (i.e.
via the public btf__parse_elf()) and the object file carries a .BTF.base
(distilled base) section, a dist_base_btf object is created and used as
the base of the split BTF built from the .BTF section. Because base_btf
is NULL, the relocation block that would otherwise free and clear
dist_base_btf is skipped, and ownership of dist_base_btf is instead
transferred to the split btf by setting btf->owns_base = true.
That ownership transfer was performed before the fallible btf_ext__new()
call that parses the .BTF.ext section. If .BTF.ext is malformed,
btf_ext__new() fails and the function jumps to the error path, which
frees dist_base_btf directly and then frees btf. Since owns_base is
already set, btf__free(btf) also frees btf->base_btf, which is the same
dist_base_btf object. The result is a use-after-free read followed by a
double free of the base BTF, driven entirely by a crafted object file
(a .BTF + .BTF.base + malformed .BTF.ext combination) passed to
btf__parse_elf(), as used by bpftool, pahole and similar tools.
Transfer ownership only after .BTF.ext has been parsed successfully, so
that any earlier failure leaves dist_base_btf owned solely by the local
cleanup path and it is freed exactly once.
Signed-off-by: Naveed Khan <naveed@digiscrypt.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/178345549172.94179.7948304165383170781@digiscrypt.com
The signed gen_loader used to police its own metadata map from within
BPF: emit_signature_match() read the kernel-cached map->sha[] back
through hardcoded struct bpf_map offsets and compared it against a hash
that compute_sha_update_offsets() baked into the signed instructions,
after a BPF_OBJ_GET_INFO_BY_FD round-trip to populate map->sha[].
The kernel now verifies the metadata at BPF_PROG_LOAD time by folding
the frozen contents of the loader's exclusive fd_array maps into the
signature, so the loader no longer checks anything itself. Generated
loaders thus carry no verification logic of their own anymore: Nothing
in the signing chain depends on emitted loader bytecode doing the right
thing.
On the loading side, skel_internal.h now sets fd_array_cnt for a signed
load so the kernel scans fd_array for the exclusive metadata map -
still frozen, as the kernel requires - and the BPF_OBJ_GET_INFO_BY_FD
round-trip to populate map->sha[] is gone. The struct bpf_map layout
BUILD_BUG_ON()s on the kernel side are removed as well: they only
pinned the ABI for the in-BPF read of map->sha[] that is no longer
needed. Same for the map->excl member. Note: gen_hash is retained; it
still marks a loader as signed so an untrusted host cannot re-dimension
maps or override initial values now covered by the signature.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260708075343.358712-4-daniel@iogearbox.net
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
bpf_object__probe_loading() tries to load trivial SOCKET_FILTER and
TRACEPOINT programs to verify the BPF environment works. When a
BPF token is in use with restricted program type permissions, these
probe loads may fail because the token does not allow the specific
program types, even though BPF loading is perfectly functional.
Fix by skipping the probe when a token FD is present: BPF token
creation itself proves the kernel has a working BPF subsystem.
Real BPF issues will be caught during actual program and map loading.
Signed-off-by: Yuan Chen <chenyuan@kylinos.cn>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260610145059.113412-2-chenyuan_fl@163.com
Adding BPF_OBJ_GET_INFO_BY_FD support for tracing_multi links.
We expose following tracing_multi link data:
- attach_type of the program
- number of ids
- array of BTF ids
- array of its related kernel addresses
- array of cookies
The change follows the kprobe_multi and uprobe_multi link-info convention
of optional output arrays with an in/out count,
On top of standard tracing link data we also expose addresses, because they
are useful info for user (especially when the attachment was done via pattern).
This data is hidden when kallsyms does not allow exposing kernel pointer values.
Assisted-by: Codex:GPT-5
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Leon Hwang <leon.hwang@linux.dev>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260629212208.895962-2-jolsa@kernel.org
Building libbpf with -Wall (as happens via bpftool's bootstrap build)
surfaces ~120 -Wformat warnings where pr_warn/pr_debug format specifiers
don't match their argument types: %d for __u32/Elf64_Word, %u for signed
ints, %zd for size_t, %ld for unsigned long, and %x/%lx/%llx applied to
signed values.
Match each specifier to its argument's type where a correctly-signed
specifier exists (%d<->%u, %ld->%lu, %zd->%zu). For hex conversions,
which have no signed form, cast the argument instead (%x->(unsigned),
%lx->(unsigned long), %llx->(unsigned long long)). No functional change.
Note, the fdinfo map_flags sscanf used %i into a __u32 *, which warns.
The kernel prints map_flags as hex ("map_flags:\t%#x\n" in
bpf_map_show_fdinfo(), unchanged since the field was added to fdinfo), so
switch the conversion to %x: it parses the 0x-prefixed value and expects
unsigned int *, matching the destination, so the warning is gone with no
cast.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/r/20260624204946.2901178-1-andrii@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
We have several use cases where a pod injects traffic into the datapath
of another so that the traffic appears to have originated from that
pod. One such use case is a synthetic flow generator which injects
synthetic traffic into a pod's datapath to enable dynamic probing and
debugging. Another is a transparent proxy where connections originating
from one pod are redirected towards another which proxies that
connection. The new connection is bound to the IP of the original pod
using IP_TRANSPARENT and its traffic is injected into that pod's
datapath and handled as if it had originated there. This can be used for
mTLS, etc.
We use bpf_redirect(BPF_F_INGRESS) to direct traffic leaving the proxy,
flow generator, etc. towards the target pod, ensuring that eBPF programs
that are meant to intercept traffic leaving that pod are executed.
However, this doesn't work with netkit.
With netkit, an ingress redirection from proxy to workload skips eBPF
programs that are meant to intercept traffic leaving the pod, since they
reside on the netkit peer device. One workaround is to attach the
same program to both the netkit peer device and the TCX ingress hook for
the netkit pair's primary interface, but
a) This seems hacky and we need to be careful not to run the same
program twice for the same skb in cases where we want to pass that
traffic to the host stack.
b) We're trying to keep the proxy redirection / traffic injection
systems as modular and separated from Cilium as possible, the system
that manages netkit setup and core eBPF programming.
It would be handy if instead we could redirect traffic directly from
one netkit peer device to another. This patch proposes an extension
to bpf_redirect_peer to allow us to do just that.
With this patch, the BPF_F_EGRESS flag tells bpf_redirect_peer to emit
the skb in the egress direction of the target interface's peer device
While the main use case is netkit, I suppose you could also use this
mode with veth as well if, e.g., there were some eBPF programs attached
to that side of the veth pair that needed to intercept traffic.
+---------------------------------------------------------------------+
| +-------------------------+ 6. bpf_redirect_neigh(eth0) |
| | pod (10.244.0.10) | ------------------------ |
| | | | | |
| | +--------+ | | +---------+ | |
| | 1. packet -->| | | | | | | |
| | leaves ^ | netkit |<===========|======| netkit | | |
| | | | peer |=======(eBPF)=====>| primary | | |
| | | | | | | | | | |
| | | +--------+ | | +---------+ | |
| | | | | 2. bpf_redirect v |
| +-----------|-------------+ |___________________ +-------|
| | | | eth0 |
| | 5. bpf_redirect_peer(BPF_F_EGRESS) | +-------|
| |________________________ | |
| +-------------------------+ | | |
| | proxy (10.244.0.11) | | | |
| | IP_TRANSPARENT | | | |
| | +--------+ | | +---------+ | |
| | 3. packet <--| | | | | |<-- |
| | enters | netkit |<===========|======| netkit | |
| | [proxy] | peer |=======(eBPF)=====>| primary | |
| | 4. packet -->| | | | | |
| | leaves +--------+ | +---------+ |
| | sip=10.244.0.10 | |
| +-------------------------+ |
+---------------------------------------------------------------------+
Using the proxy use case as an example, in step 5 we would redirect
traffic leaving the proxy towards the pod's peer device using
bpf_redirect_peer(BPF_F_EGRESS).
As a bonus, since the skb doesn't have to go through the backlog queue
it can take full advantage of netkit's performance benefits. I set up a
test where outgoing iperf3 traffic is injected into the datapath of
another pod using either bpf_redirect_peer(BPF_F_EGRESS) or
bpf_redirect(BPF_F_INGRESS). I used Cilium's eBPF host routing mode
which skips the host stack and uses BPF redirect helpers to do all the
routing.
(net.ipv4.tcp_congestion_control=cubic,mtu=1500,100GiB link,Cilium
eBPF host routing mode)
BASELINE [bpf_redirect(BPF_F_INGRESS)]
1. [iperf pod] ==bpf_redirect([pod b], BPF_F_INGRESS)==> [pod b]
2. [pod b] ==bpf_redirect_neigh([eth0])==> eth0
3. eth0 ==over network==> [host b]
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 231 GBytes 33.0 Gbits/sec 12060 sender
[ 5] 0.00-60.00 sec 230 GBytes 33.0 Gbits/sec receiver
TEST [bpf_redirect_peer(BPF_F_EGRESS)]
1. [iperf pod] ==bpf_redirect_peer([pod b], BPF_F_EGRESS)==> [pod b]
2. [pod b] ==bpf_redirect_neigh([eth0])==> eth0
3. eth0 ==over network==> [host b]
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-60.00 sec 272 GBytes 38.9 Gbits/sec 0 sender
[ 5] 0.00-60.00 sec 272 GBytes 38.9 Gbits/sec receiver
In this test, using bpf_redirect_peer(BPF_F_EGRESS) for the hop from
[iperf pod] to [pod b] led to ~18% more throughput compared to
bpf_redirect(BPF_F_INGRESS).
Signed-off-by: Jordan Rife <jordan@jrife.io>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Paul Chaignon <paul.chaignon@gmail.com>
Reviewed-by: Jiayuan Chen <jiayuan.chen@linux.dev>
Link: https://lore.kernel.org/r/20260618182035.43811-2-jordan@jrife.io
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Allow uprobe_multi link to identify the target binary by an already
opened file descriptor.
Adding new BPF_F_UPROBE_MULTI_PATH_FD flag and the path_fd field for
the attr.link_create.uprobe_multi struct.
When the flag is set, we resolve the target from path_fd, without the
flag, we keep the existing string path behavior.
I don't see a use case for supporting O_PATH file descriptors, because
we need to read the binary first to get probes offsets, so I'm using
the CLASS(fd, f), which fails for O_PATH fds.
Assisted-by: Codex:GPT-5.4
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260611114230.950379-4-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Adding bpf_program__attach_tracing_multi function for attaching
tracing program to multiple functions.
struct bpf_link *
bpf_program__attach_tracing_multi(const struct bpf_program *prog,
const char *pattern,
const struct bpf_tracing_multi_opts *opts);
User can specify functions to attach with 'pattern' argument that
allows wildcards (*?' supported) or provide BTF ids of functions
in array directly via opts argument. These options are mutually
exclusive.
When using BTF ids, user can also provide cookie value for each
provided id/function, that can be retrieved later in bpf program
with bpf_get_attach_cookie helper. Each cookie value is paired with
provided BTF id with the same array index.
Adding support to auto attach programs with following sections:
fsession.multi/<pattern>
fsession.multi.s/<pattern>
fentry.multi/<pattern>
fexit.multi/<pattern>
fentry.multi.s/<pattern>
fexit.multi.s/<pattern>
The provided <pattern> is used as 'pattern' argument in
bpf_program__attach_kprobe_multi_opts function.
The <pattern> allows to specify optional kernel module name with
following syntax:
<module>:<function_pattern>
In order to attach tracing_multi link to a module functions:
- program must be loaded with 'module' btf fd
(in attr::attach_btf_obj_fd)
- bpf_program__attach_tracing_multi must either have
pattern with module spec or BTF ids from the module
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260606123955.345967-21-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Adding btf_type_is_traceable_func function to perform same checks
as the kernel's btf_distill_func_proto function to prevent attachment
on some of the functions.
Exporting the function via libbpf_internal.h because it will be used
by benchmark test in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260606123955.345967-20-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Adding support to use session attachment with tracing_multi link.
Adding new BPF_TRACE_FSESSION_MULTI program attach type, that follows
the BPF_TRACE_FSESSION behaviour but on the tracing_multi link.
Such program is called on entry and exit of the attached function
and allows to pass cookie value from entry to exit execution.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260606123955.345967-16-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Add support to specify cookies for tracing_multi link.
Cookies are provided in array where each value is paired with provided
BTF ID value with the same array index.
Such cookie can be retrieved by bpf program with bpf_get_attach_cookie
helper call.
We need to sort cookies array together with ids array in check_dup_ids,
to keep the id->cookie relation.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260606123955.345967-15-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Adding new link to allow to attach program to multiple function
BTF IDs. The link is represented by struct bpf_tracing_multi_link.
To configure the link, new fields are added to bpf_attr::link_create
to pass array of BTF IDs;
struct {
__aligned_u64 ids;
__u32 cnt;
} tracing_multi;
Each BTF ID represents function (BTF_KIND_FUNC) that the link will
attach bpf program to.
We use previously added bpf_trampoline_multi_attach/detach functions
to attach/detach the link.
The linkinfo/fdinfo callbacks will be implemented in following changes.
Note this is supported only for archs (x86_64) with ftrace direct and
have single ops support.
CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS &&
CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
Note using sort_r (instead of plain sort) in check_dup_ids, because we
will use the swap callback in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260606123955.345967-14-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Martin KaFai Lau
Mingpei CAO
Luis Vieira
Toke Høiland-Jørgensen
Mahe Tardy
Mykyta Yatsenko
Stanislav Fomichev
Sudip Mukherjee
Tw
Jiayuan Chen
Darren Carreras
Bobby Eshleman
Nick Hudson
Israel Téllez García
Leon Hwang
Ihor Solodrai
Ricardo B. Marlière
Jianlin Shi
Avinash Duduskar
Jiri Olsa
Christian Brauner
Daniel Golle
Michel Lind
Song Liu
Naveed Khan
Daniel Borkmann
Yuan Chen
Andrii Nakryiko
Jordan Rife