mirror of
https://github.com/netdata/libbpf.git
synced 2026-11-06 01:40:15 +08:00
bc48d4e164a6c1c0f7b4a2367a7932bc42aeafa1
575
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Amery Hung
|
bc48d4e164 |
bpf: tcp: Support parse/len/write header option hooks in bpf_tcp_ops
Add the TCP header option callbacks to the bpf_tcp_ops struct_ops type:
parse_hdr - parse the options of an incoming skb on an established
connection
hdr_opt_len - reserve space in the TCP header for bpf options
write_hdr_opt - write the reserved bpf options
These mirror the BPF_SOCK_OPS_PARSE_HDR_OPT_CB, _HDR_OPT_LEN_CB and
_WRITE_HDR_OPT_CB legacy sockops callbacks, but are exposed as struct_ops
members so a program can implement them with normal function signatures
and per-member helper sets.
The reserved header window is shared between the legacy sockops and
bpf_tcp_ops paths. tcp_{syn,synack,established}_options() first run the
legacy BPF_SOCK_OPS_HDR_OPT_LEN_CB and then call hdr_opt_len, so both
sources accumulate into opts->bpf_opt_len; at write time the legacy
options are emitted first and bpf_tcp_ops writes after them.
API design
bpf_tcp_ops overloads the sock_ops header-option helpers rather than
introducing a new API: bpf_reserve_hdr_opt(), bpf_store_hdr_opt() and
bpf_load_hdr_opt() are exposed per-member (reserve for hdr_opt_len,
store/load for write_hdr_opt, load for parse_hdr) and share the existing
kernel option-walking core via _bpf_sock_ops{store,load}hdr_opt(), with
the bpf_tcp_ops wrappers synthesizing a temporary bpf_sock_ops_kern from
the program ctx. This keeps a port from the legacy
BPF_SOCK_OPS*_HDR_OPT_CB callbacks mechanical (same helper calls) and
adds no new UAPI helper/kfunc surface.
An alternative considered was to drop the option helpers entirely: have
hdr_opt_len reserve space purely through its return value, and introduce
a dedicated TCP-header-option dynptr used for both reading and writing.
That is a cleaner, more self-contained interface, but it is a larger
change and does not reuse the legacy helpers, making a port from sockops
less mechanical. It can be pursued as a follow-up; the helper-based
interface here keeps this series focused on moving the hooks to
struct_ops.
The hdr_opt_len fast path in tcp_established_options() is gated by
cgroup_bpf_enabled(CGROUP_TCP_SOCK_OPS). Note this is a global,
per-attach-type static branch: it is enabled whenever any bpf_tcp_ops is
attached, even one that does not implement hdr_opt_len or that is attached
to a different cgroup. In those cases the block still runs but
bpf_tcp_ops_hdr_opt_len() no-ops via the per-member check in the dispatch
macro. A per-member/per-cgroup gate could be added later if the extra
fast-path work proves measurable.
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com>
Link: https://patch.msgid.link/20260917200542.3689605-13-ameryhung@gmail.com
|
||
 Martin KaFai Lau
|
a7ee927845 |
bpf: Add infrastructure to support attaching struct_ops to cgroups
This patch adds necessary infrastructure to attach a struct_ops
map to a cgroup. The initial need was to support migrating
the legacy BPF_PROG_TYPE_SOCK_OPS to a struct_ops.
Recently, there are other struct_ops use cases that
need to attach struct_ops to a cgroup. For example,
the recent BPF OOM and memcg discussion in LSFMMBPF 2026.
The motivation is to create a consistent expectation
for attaching struct_ops to cgroup instead of each subsystem
creating its own infrastructure. This logic includes
hierarchy expectation, ordering expectation,
attachment API, and rcu gp.
There is already an existing implementation for attaching
multiple bpf progs to a cgroup. There are also tools
built around it for querying. Attaching a struct_ops map
(which is a group of bpf programs) could also adhere to
a similar API and potentially reuse most of the existing
implementation.
A couple of ideas have been tried. One of them
is to use mprog.c. In terms of the amount of changes,
I eventually came to the same conclusion as in
commit 120933984460 ("bpf: Implement mprog API on top of existing cgroup progs").
I then shifted the focus to reusing the current
{update,compute,activate,purge}_effective_progs() which has
the main logic that implements the mprog API.
Since then, I tried to add a 'struct cgroup *cgroup' member
to the existing 'struct bpf_struct_ops_link' and link_create
will create a 'struct bpf_struct_ops_link' object to be stored
in the pl->link. This turns out to have more changes on
both cgroup.c and bpf_struct_ops.c than I like.
This patch directly reuses the 'struct bpf_cgroup_link' which
cgroup.c already understands. Add 'struct bpf_map *map'
to 'struct bpf_cgroup_link'. In the future, as more subsystems
are extended by struct_ops, we may consider to make
'struct bpf_map *map' as a primary citizen of a link
like 'struct bpf_prog *prog' and directly add
'struct bpf_map *map' to the generic 'struct bpf_link'.
The pl->link could be the traditional 'prog' link or the
new 'map' link. The places that need to handle them differently
have already been refactored into the new prog_list_*() added in
the earlier patch. In those new prog_list_*(), this patch will
check "pl->link && pl->link->map", learn that it is a 'map' link
and handle it correctly.
The bpf_prog_array also needs to handle that its item can store
the traditional 'prog' or it can store a struct_ops map.
The places that need to handle them differently have also
been refactored into the new bpf_cgroup_array_*() added
in the earlier patch. The two differences are:
- different sentinel (dummy_bpf_prog in prog vs cfi_stub in struct_ops)
- the array for struct_ops may need to go through different
rcu gp.
The bpf_cgroup_array_*() functions use the cgroup_bpf_attach_type (ie atype)
to distinguish the array is storing prog or storing struct_ops map.
This patch also implements a separate struct bpf_link_ops
"cgroup_struct_ops_link_ops" to have a separate link_ops implementation
that only handles the cgroup's struct_ops link.
Questions:
- Although this patch did not change it, it is not obvious to me how
the replace_effective_progs() and purge_effective_progs() handle
cases when there are existing BPF_F_PREORDER progs attached
in the hlist.
Misc notes:
- CGROUP_TCP_SOCK_OPS is added to the 'enum cgroup_bpf_attach_type'.
The actual implementation of the tcp_bpf_ops (a struct_ops)
will be added in the next patch.
- free_after_mult_rcu_gp is added to 'struct bpf_struct_ops' such that
the bpf_prog_array can have a mix of sleepable and
non-sleepable prog in a struct_ops. This can tell
how the bpf_prog_array should be freed.
- For a struct_ops that supports cgroup attachment, it does not need to
implement its own reg/unreg function. reg/unreg to a cgroup is
done by the common infrastructure added in this patch.
- The cgroup's struct_ops link only supports BPF_F_ALLOW_MULTI.
This is enforced internally in cgroup_bpf_struct_ops_attach.
This should be consistent with the current prog's link
behavior in cgroup_bpf_link_attach.
In the future, we may allow each subsystem to choose differently.
- A cgroup_atype member is added to 'struct bpf_struct_ops'.
When a subsystem struct_ops needs to support cgroup attachment,
it needs to add a value to 'enum cgroup_bpf_attach_type'
and then assign it to the newly added cgroup_atype member
in the bpf_struct_ops.
- During LINK_CREATE in syscall, the patch uses the same
BPF_STRUCT_OPS (in attr->link_create.attach_type).
The bpf_struct_ops_link_create learns the map and
from the map it learns the st_ops. If the st_ops->cgroup_atype
is not 0, it will create a cgroup's link.
- When a subsystem registers a struct_ops that supports cgroup
attachment, the struct_ops infrastructure will also ask the
cgroup infrastructure to remember a few things. This is done
by calling cgroup_bpf_struct_ops_register().
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Signed-off-by: Amery Hung <ameryhung@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20260917200542.3689605-10-ameryhung@gmail.com
|
||
 Stanislav Fomichev
|
6de1eac4cd |
xsk: align TX metadata layout across ABIs
Add explicit padding before launch_time so xsk_tx_metadata has the same
layout on 32-bit and 64-bit systems.
On several architectures (csky, i386, nios2, m65k, openrisc, sh), the old
native 32-bit layout put launch_time at offset 12 and had a natural size of
20 bytes. Using sizeof(struct xsk_tx_metadata) as tx_metadata_len was already
rejected because the length must be a multiple of eight, so the
straightforward use of the interface was broken on those ABIs. Userspace
could still register a padded length of 24 bytes, though; mixing the old and
new layouts then silently reads launch_time from the wrong offset and
misprograms packet launch times. This intentionally replaces that
incompatible layout because the affected architectures are unlikely to
have any notable users. (x86_64 and arm64 have the most users and are _not_
affected)
Fixes: ca4419f15abd ("xsk: Add launch time hardware offload support to XDP Tx metadata")
Reviewed-by: Simon Horman <horms@kernel.org>
Signed-off-by: Stanislav Fomichev <sdf@fomichev.me>
Link: https://patch.msgid.link/20260819160535.1472459-2-sdf@fomichev.me
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
||
 Bobby Eshleman
|
7183e9519f |
net: devmem: allow rx-page-size > PAGE_SIZE per dmabuf binding
Every devmem dmabuf binding today hands the page_pool PAGE_SIZE niovs.
This caps a single RX descriptor at PAGE_SIZE, burning CPU on buffer
churn for large flows.
Add a bind-time netlink attribute, NETDEV_A_DMABUF_RX_PAGE_SIZE, that
lets userspace request a larger niov size. The value must be a power of
two >= PAGE_SIZE.
The TX path is changed to always pass PAGE_SIZE.
Measurements:
Setup: kperf in devmem RX/TX cuda mode, 4 flows, 64 MB messages, 60s,
dctcp, num-rx-queues=4, dmabuf-rx/tx-size-mb=2048, 10 runs per niov
size, mlx5.
CPU Util:
niov net sirq % net idle % app sys % app idle %
----- ---------------- ---------------- ---------------- ----------------
4K 62.38 +/- 8.27 33.40 +/- 7.51 54.15 +/- 10.23 43.67 +/- 10.53
16K 58.91 +/- 5.35 35.23 +/- 5.88 41.05 +/- 8.87 56.42 +/- 9.24
32K 64.12 +/- 0.68 31.09 +/- 1.48 44.54 +/- 3.51 52.63 +/- 3.65
64K 54.69 +/- 5.54 39.67 +/- 5.81 35.47 +/- 3.11 61.97 +/- 3.27
RX app sys % drops ~19% from 4K to 64K.
Throughput:
niov RX dev Gbps RX flow avg Gbps
----- ---------------- -----------------
4K 300.63 +/- 53.21 75.16 +/- 13.30
16K 321.35 +/- 28.20 80.34 +/- 7.05
32K 347.63 +/- 2.20 86.91 +/- 0.55
64K 332.11 +/- 14.26 83.03 +/- 3.56
Throughput seems to increase, but the stdev is pretty wide so could just
be noise.
kperf support (not yet merged):
https://github.com/facebookexperimental/kperf/commit/8837577f920876bce6986ec18869ac04439ebcd2
Acked-by: Stanislav Fomichev <sdf@fomichev.me>
Reviewed-by: Mina Almasry <almasrymina@google.com>
Reviewed-by: Nikolay Aleksandrov <razor@blackwall.org>
Signed-off-by: Bobby Eshleman <bobbyeshleman@meta.com>
Link: https://patch.msgid.link/20260805-tcpdm-large-niovs-v8-1-3e0225e2808c@meta.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
||
 Nick Hudson
|
19e98d95e9 |
bpf: Add BPF_F_ADJ_ROOM_DECAP_* flags for tunnel decapsulation
Add new bpf_skb_adjust_room() decapsulation flags: - BPF_F_ADJ_ROOM_DECAP_L4_GRE - BPF_F_ADJ_ROOM_DECAP_L4_UDP - BPF_F_ADJ_ROOM_DECAP_IPXIP4 - BPF_F_ADJ_ROOM_DECAP_IPXIP6 These flags let BPF programs describe which tunnel layer is being removed, so later changes can update tunnel-related GSO state accordingly during decapsulation. This patch only introduces the UAPI flag definitions and helper documentation. Co-developed-by: Max Tottenham <mtottenh@akamai.com> Co-developed-by: Anna Glasgall <aglasgal@akamai.com> Signed-off-by: Max Tottenham <mtottenh@akamai.com> Signed-off-by: Anna Glasgall <aglasgal@akamai.com> Signed-off-by: Nick Hudson <nhudson@akamai.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Reviewed-by: Willem de Bruijn <willemb@google.com> Link: https://lore.kernel.org/bpf/20260812083115.73100-4-nhudson@akamai.com |
||
|
Nick Hudson
|
aaafcbda3b |
bpf: Name the enum for BPF_FUNC_skb_adjust_room flags
The existing anonymous enum for BPF_FUNC_skb_adjust_room flags is named to enum bpf_adj_room_flags to enable CO-RE (Compile Once - Run Everywhere) lookups in BPF programs. Co-developed-by: Max Tottenham <mtottenh@akamai.com> Co-developed-by: Anna Glasgall <aglasgal@akamai.com> Signed-off-by: Max Tottenham <mtottenh@akamai.com> Signed-off-by: Anna Glasgall <aglasgal@akamai.com> Signed-off-by: Nick Hudson <nhudson@akamai.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Reviewed-by: Willem de Bruijn <willemb@google.com> Link: https://lore.kernel.org/bpf/20260812083115.73100-2-nhudson@akamai.com |
||
 Jianlin Shi
|
cd08bc2420 |
docs: bpf: Document BPF_RB_OVERWRITE_POS in bpf_ringbuf_query
BPF_RB_OVERWRITE_POS is supported by bpf_ringbuf_query() but was missing from the helper documentation. Add it to the flags list in both the kernel UAPI header and its tools/ mirror. Signed-off-by: Jianlin Shi <shijianlin11@foxmail.com> Acked-by: Xu Kuohai <xukuohai@huawei.com> Link: https://lore.kernel.org/bpf/tencent_22134645443B75ED907D2A85A47AD554A709@qq.com Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> |
||
 Avinash Duduskar
|
1810ef6cdb |
bpf: Add BPF_FIB_LOOKUP_VLAN_INPUT flag to bpf_fib_lookup() helper
BPF_FIB_LOOKUP_VLAN resolves a VLAN egress. The reverse is also useful: an XDP program receiving a VLAN-tagged frame on a physical device wants the lookup to behave as if the packet had arrived on the corresponding VLAN subinterface, so iif-based policy routing and VRF table selection use the right ingress. Add BPF_FIB_LOOKUP_VLAN_INPUT. When set, params->h_vlan_proto and params->h_vlan_TCI are read as an input VLAN tag and the matching VLAN device of params->ifindex is resolved with __vlan_find_dev_deep_rcu(). The device must be up and in the same network namespace as params->ifindex (a VLAN device can be moved to another netns while registered on its parent; receive would deliver into that other namespace, which a lookup here cannot represent). If params->ifindex is itself a VLAN device, its inner (QinQ) subinterface is matched. For a bond or team, a tag on a port matches no device and returns NOT_FWDED; pass the master's ifindex. The lookup then runs with the resolved device as the ingress; params->ifindex itself is not modified on the input side. When the resolved device is enslaved to a VRF, both the full lookup (via the l3mdev rule) and BPF_FIB_LOOKUP_DIRECT (via l3mdev_fib_table_rcu()) select the VRF's table from the resolved ingress. That follows from feeding the resolved device to the flow as the ingress (fl4.flowi4_iif = dev->ifindex), which is what makes l3mdev resolve the VRF master from the subinterface rather than from params->ifindex. The two failure classes get different treatment on purpose. A h_vlan_proto other than 802.1Q/802.1ad is API misuse and returns -EINVAL, since it would otherwise reach the WARN in vlan_proto_idx() with a program-controlled value. An unmatched VID, a device that is down, or one in another namespace is a data outcome and returns BPF_FIB_LKUP_RET_NOT_FWDED, matching the DIRECT path when fib_get_table() finds no table and mirroring real ingress, where the receive path drops such frames. A VID of 0 (a priority tag) is looked up literally and normally fails the same way; receive instead processes such frames untagged, so callers should not set the flag for priority tags. Proceeding on the physical device for any of these would be fail-open for the policy-routing cases above. The h_vlan fields share a union with tbid, so the flag cannot be combined with BPF_FIB_LOOKUP_TBID. It describes ingress, so it also cannot be combined with BPF_FIB_LOOKUP_OUTPUT. Both combinations return -EINVAL; restricting now keeps a later relaxation backward compatible. Combining with BPF_FIB_LOOKUP_VLAN is allowed: the tag is consumed on the ingress side and the egress tag is written on success. Under !CONFIG_VLAN_8021Q the __vlan_find_dev_deep_rcu() stub returns NULL, so every lookup with a valid proto returns NOT_FWDED, which is correct since no VLAN device can exist. Suggested-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Avinash Duduskar <avinash.duduskar@gmail.com> Reviewed-by: Toke Høiland-Jørgensen <toke@redhat.com> Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com> Link: https://lore.kernel.org/bpf/20260713162305.1237211-3-avinash.duduskar@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> |
||
|
Avinash Duduskar
|
a432251e60 |
bpf: Add BPF_FIB_LOOKUP_VLAN flag to bpf_fib_lookup() helper
bpf_fib_lookup() returns the FIB-resolved egress ifindex straight from the fib result. When the egress is a VLAN device, the returned ifindex is the VLAN netdev's, which has no XDP xmit handler; XDP programs that want to forward the frame (e.g. xdp-forward) must instead target the underlying physical device and push the VLAN tag themselves. Today the program has no way to learn either the underlying ifindex or the VLAN tag without maintaining its own VLAN-to-ifindex map in userspace and refreshing it on netlink events. Add BPF_FIB_LOOKUP_VLAN. When the caller sets this flag and the fib result is a VLAN device whose immediate parent is a real (non-VLAN) device in the same network namespace, populate the existing output fields params->h_vlan_proto and params->h_vlan_TCI from the VLAN device and replace params->ifindex with the parent's ifindex. params->h_vlan_TCI carries the VID only, with PCP and DEI bits zero; a consumer wanting to set egress priority writes PCP itself. params->smac is the VLAN device's own address, which can differ from the parent's. Only the immediate parent is resolved, via vlan_dev_priv(dev)->real_dev and not vlan_dev_real_dev(), which walks to the bottom of a stack. When the immediate parent is not a real device in the same namespace, the lookup returns BPF_FIB_LKUP_RET_VLAN_FAILURE and leaves params->ifindex at the input. This covers a stacked VLAN (QinQ), where the immediate parent is itself a VLAN device and one h_vlan_proto/h_vlan_TCI pair cannot describe two tags, and a parent in another network namespace (a VLAN device can be moved while its parent stays), whose ifindex would be meaningless in the caller's namespace. A program that wants the VLAN device's own ifindex re-issues the lookup, with a re-initialized params, without BPF_FIB_LOOKUP_VLAN, so the unreducible case stays distinct from a physical egress. That distinction matters for XDP: a program cannot xmit on a VLAN device, so a success carrying the VLAN ifindex would make it redirect to a device with no ndo_xdp_xmit and drop the frame at xdp_do_flush(). The swap and the vlan fields are written only on the reduce path; other output fields keep their existing behaviour, so a frag-needed result still reports the route mtu in params->mtu_result. BPF_FIB_LOOKUP_VLAN is only useful to XDP, which cannot redirect to a VLAN device. A tc program can redirect to the VLAN device directly, so bpf_skb_fib_lookup() rejects the flag with -EINVAL; bpf_xdp_fib_lookup() accepts it. When the flag is not set, behaviour is unchanged: h_vlan_proto and h_vlan_TCI are zeroed and ifindex is left at the FIB result. The new block is compiled only under CONFIG_VLAN_8021Q since vlan_dev_priv() is not defined otherwise; without that config is_vlan_dev() is constant false and the flag is accepted but never acts. That is safe because no VLAN device can exist there, so every egress is already physical. This lets an XDP redirect target the physical device and learn the tag to push in a single lookup, which xdp-forward's optional VLAN mode (xdp-project/xdp-tools#504) wants from the kernel side. The helper's input semantics are unchanged; the reverse direction (supplying a tag as lookup input) is added in the following patch. Suggested-by: Toke Høiland-Jørgensen <toke@redhat.com> Signed-off-by: Avinash Duduskar <avinash.duduskar@gmail.com> Reviewed-by: Toke Høiland-Jørgensen <toke@redhat.com> Reviewed-by: Emil Tsalapatis <emil@etsalapatis.com> Acked-by: David Ahern <dsahern@kernel.org> Link: https://lore.kernel.org/bpf/20260713162305.1237211-2-avinash.duduskar@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> |
||
 Christian Brauner
|
d8b7bfc9f6 |
fs: support FD_FAILFS_ROOT in fchdir()
Add a new file descriptor sentinel FD_FAILFS_ROOT following FD_PIDFS_ROOT and FD_NSFS_ROOT and teach fchdir() to accept it. A process calling fchdir(FD_FAILFS_ROOT) moves its working directory into failfs. Every AT_FDCWD-relative lookup afterwards fails with EOPNOTSUPP including "." and ".." and getcwd() reports the working directory as unreachable from the process root by returning a path prefixed with "(unreachable)". Lookups relative to explicit directory file descriptors are unaffected. The sentinel is the only way in. No privilege or gating is required. Setting the working directory to a directory in which every operation fails grants nothing and loses nothing that closing file descriptors couldn't lose. An unlinked working directory behaves the same way today modulo errno. The working directory also plays no role in confining ".." resolution so no boundary is weakened. Link: https://patch.msgid.link/20260724-work-failfs-v2-2-485dabbae185@kernel.org Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org> |
||
 Jiri Olsa
|
d36d49e72c |
bpf: Add tracing_multi link info support
Adding BPF_OBJ_GET_INFO_BY_FD support for tracing_multi links. We expose following tracing_multi link data: - attach_type of the program - number of ids - array of BTF ids - array of its related kernel addresses - array of cookies The change follows the kprobe_multi and uprobe_multi link-info convention of optional output arrays with an in/out count, On top of standard tracing link data we also expose addresses, because they are useful info for user (especially when the attachment was done via pattern). This data is hidden when kallsyms does not allow exposing kernel pointer values. Assisted-by: Codex:GPT-5 Signed-off-by: Jiri Olsa <jolsa@kernel.org> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Leon Hwang <leon.hwang@linux.dev> Acked-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260629212208.895962-2-jolsa@kernel.org |
||
 Jordan Rife
|
8dae7836fb |
bpf: Support BPF_F_EGRESS with bpf_redirect_peer
We have several use cases where a pod injects traffic into the datapath of another so that the traffic appears to have originated from that pod. One such use case is a synthetic flow generator which injects synthetic traffic into a pod's datapath to enable dynamic probing and debugging. Another is a transparent proxy where connections originating from one pod are redirected towards another which proxies that connection. The new connection is bound to the IP of the original pod using IP_TRANSPARENT and its traffic is injected into that pod's datapath and handled as if it had originated there. This can be used for mTLS, etc. We use bpf_redirect(BPF_F_INGRESS) to direct traffic leaving the proxy, flow generator, etc. towards the target pod, ensuring that eBPF programs that are meant to intercept traffic leaving that pod are executed. However, this doesn't work with netkit. With netkit, an ingress redirection from proxy to workload skips eBPF programs that are meant to intercept traffic leaving the pod, since they reside on the netkit peer device. One workaround is to attach the same program to both the netkit peer device and the TCX ingress hook for the netkit pair's primary interface, but a) This seems hacky and we need to be careful not to run the same program twice for the same skb in cases where we want to pass that traffic to the host stack. b) We're trying to keep the proxy redirection / traffic injection systems as modular and separated from Cilium as possible, the system that manages netkit setup and core eBPF programming. It would be handy if instead we could redirect traffic directly from one netkit peer device to another. This patch proposes an extension to bpf_redirect_peer to allow us to do just that. With this patch, the BPF_F_EGRESS flag tells bpf_redirect_peer to emit the skb in the egress direction of the target interface's peer device While the main use case is netkit, I suppose you could also use this mode with veth as well if, e.g., there were some eBPF programs attached to that side of the veth pair that needed to intercept traffic. +---------------------------------------------------------------------+ | +-------------------------+ 6. bpf_redirect_neigh(eth0) | | | pod (10.244.0.10) | ------------------------ | | | | | | | | | +--------+ | | +---------+ | | | | 1. packet -->| | | | | | | | | | leaves ^ | netkit |<===========|======| netkit | | | | | | | peer |=======(eBPF)=====>| primary | | | | | | | | | | | | | | | | | +--------+ | | +---------+ | | | | | | | 2. bpf_redirect v | | +-----------|-------------+ |___________________ +-------| | | | | eth0 | | | 5. bpf_redirect_peer(BPF_F_EGRESS) | +-------| | |________________________ | | | +-------------------------+ | | | | | proxy (10.244.0.11) | | | | | | IP_TRANSPARENT | | | | | | +--------+ | | +---------+ | | | | 3. packet <--| | | | | |<-- | | | enters | netkit |<===========|======| netkit | | | | [proxy] | peer |=======(eBPF)=====>| primary | | | | 4. packet -->| | | | | | | | leaves +--------+ | +---------+ | | | sip=10.244.0.10 | | | +-------------------------+ | +---------------------------------------------------------------------+ Using the proxy use case as an example, in step 5 we would redirect traffic leaving the proxy towards the pod's peer device using bpf_redirect_peer(BPF_F_EGRESS). As a bonus, since the skb doesn't have to go through the backlog queue it can take full advantage of netkit's performance benefits. I set up a test where outgoing iperf3 traffic is injected into the datapath of another pod using either bpf_redirect_peer(BPF_F_EGRESS) or bpf_redirect(BPF_F_INGRESS). I used Cilium's eBPF host routing mode which skips the host stack and uses BPF redirect helpers to do all the routing. (net.ipv4.tcp_congestion_control=cubic,mtu=1500,100GiB link,Cilium eBPF host routing mode) BASELINE [bpf_redirect(BPF_F_INGRESS)] 1. [iperf pod] ==bpf_redirect([pod b], BPF_F_INGRESS)==> [pod b] 2. [pod b] ==bpf_redirect_neigh([eth0])==> eth0 3. eth0 ==over network==> [host b] [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-60.00 sec 231 GBytes 33.0 Gbits/sec 12060 sender [ 5] 0.00-60.00 sec 230 GBytes 33.0 Gbits/sec receiver TEST [bpf_redirect_peer(BPF_F_EGRESS)] 1. [iperf pod] ==bpf_redirect_peer([pod b], BPF_F_EGRESS)==> [pod b] 2. [pod b] ==bpf_redirect_neigh([eth0])==> eth0 3. eth0 ==over network==> [host b] [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-60.00 sec 272 GBytes 38.9 Gbits/sec 0 sender [ 5] 0.00-60.00 sec 272 GBytes 38.9 Gbits/sec receiver In this test, using bpf_redirect_peer(BPF_F_EGRESS) for the hop from [iperf pod] to [pod b] led to ~18% more throughput compared to bpf_redirect(BPF_F_INGRESS). Signed-off-by: Jordan Rife <jordan@jrife.io> Acked-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Paul Chaignon <paul.chaignon@gmail.com> Reviewed-by: Jiayuan Chen <jiayuan.chen@linux.dev> Link: https://lore.kernel.org/r/20260618182035.43811-2-jordan@jrife.io Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
|
Jiri Olsa
|
82d3303c39 |
bpf: Add support to specify uprobe_multi target via file descriptor
Allow uprobe_multi link to identify the target binary by an already opened file descriptor. Adding new BPF_F_UPROBE_MULTI_PATH_FD flag and the path_fd field for the attr.link_create.uprobe_multi struct. When the flag is set, we resolve the target from path_fd, without the flag, we keep the existing string path behavior. I don't see a use case for supporting O_PATH file descriptors, because we need to read the binary first to get probes offsets, so I'm using the CLASS(fd, f), which fails for O_PATH fds. Assisted-by: Codex:GPT-5.4 Signed-off-by: Jiri Olsa <jolsa@kernel.org> Link: https://lore.kernel.org/r/20260611114230.950379-4-jolsa@kernel.org Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
|
Jiri Olsa
|
ed2a823a50 |
bpf: Add support for tracing_multi link session
Adding support to use session attachment with tracing_multi link. Adding new BPF_TRACE_FSESSION_MULTI program attach type, that follows the BPF_TRACE_FSESSION behaviour but on the tracing_multi link. Such program is called on entry and exit of the attached function and allows to pass cookie value from entry to exit execution. Signed-off-by: Jiri Olsa <jolsa@kernel.org> Link: https://lore.kernel.org/r/20260606123955.345967-16-jolsa@kernel.org Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
|
Jiri Olsa
|
1ba74665c7 |
bpf: Add support for tracing_multi link cookies
Add support to specify cookies for tracing_multi link. Cookies are provided in array where each value is paired with provided BTF ID value with the same array index. Such cookie can be retrieved by bpf program with bpf_get_attach_cookie helper call. We need to sort cookies array together with ids array in check_dup_ids, to keep the id->cookie relation. Signed-off-by: Jiri Olsa <jolsa@kernel.org> Link: https://lore.kernel.org/r/20260606123955.345967-15-jolsa@kernel.org Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
|
Jiri Olsa
|
622bb934a9 |
bpf: Add support for tracing multi link
Adding new link to allow to attach program to multiple function
BTF IDs. The link is represented by struct bpf_tracing_multi_link.
To configure the link, new fields are added to bpf_attr::link_create
to pass array of BTF IDs;
struct {
__aligned_u64 ids;
__u32 cnt;
} tracing_multi;
Each BTF ID represents function (BTF_KIND_FUNC) that the link will
attach bpf program to.
We use previously added bpf_trampoline_multi_attach/detach functions
to attach/detach the link.
The linkinfo/fdinfo callbacks will be implemented in following changes.
Note this is supported only for archs (x86_64) with ftrace direct and
have single ops support.
CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS &&
CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
Note using sort_r (instead of plain sort) in check_dup_ids, because we
will use the swap callback in following changes.
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Link: https://lore.kernel.org/r/20260606123955.345967-14-jolsa@kernel.org
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
||
|
Jiri Olsa
|
4e48554f9f |
bpf: Add multi tracing attach types
Adding new program attach types multi tracing attachment: BPF_TRACE_FENTRY_MULTI BPF_TRACE_FEXIT_MULTI and their base support in verifier code. Programs with such attach type will use specific link attachment interface coming in following changes. This was suggested by Andrii some (long) time ago and turned out to be easier than having special program flag for that. Bpf programs with such types have 'bpf_multi_func' function set as their attach_btf_id and keep module reference when it's specified by attach_prog_fd. They are also accepted as sleepable programs during verification, and the real validation for specific BTF_IDs/functions will happen during the multi link attachment in following changes. Suggested-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Jiri Olsa <jolsa@kernel.org> Link: https://lore.kernel.org/r/20260606123955.345967-11-jolsa@kernel.org Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
 Leon Hwang
|
c25c836734 |
bpf: Check tail zero of bpf_prog_info
Since there're 4 bytes padding at the end of struct bpf_prog_info, they
won't be checked by bpf_check_uarg_tail_zero().
pahole -C bpf_prog_info ./vmlinux
struct bpf_prog_info {
...
__u32 attach_btf_obj_id; /* 220 4 */
__u32 attach_btf_id; /* 224 4 */
/* size: 232, cachelines: 4, members: 38 */
/* sum members: 224 */
/* sum bitfield members: 1 bits, bit holes: 1, sum bit holes: 31 bits */
/* padding: 4 */
/* forced alignments: 9 */
/* last cacheline: 40 bytes */
} __attribute__((__aligned__(8)));
If a future kernel extension adds a new 4-byte field, older userspace
programs allocating this structure on the stack might inadvertently pass
uninitialized stack garbage into the new field, permanently breaking
backward compatibility. -- sashiko [1]
Fix it by changing sizeof(info) to
offsetofend(struct bpf_prog_info, attach_btf_id).
And, add "__u32 :32" to the tail of struct bpf_prog_info.
[1] https://lore.kernel.org/bpf/20260513224823.6494FC19425@smtp.kernel.org/
Fixes: aba64c7da983 ("bpf: Add verified_insns to bpf_prog_info and fdinfo")
Acked-by: Mykyta Yatsenko <yatsenko@meta.com>
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/r/20260605155249.20772-3-leon.hwang@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
||
|
Leon Hwang
|
b8612dd587 |
bpf: Check tail zero of bpf_map_info
Since there're 4 bytes padding at the end of struct bpf_map_info, they
won't be checked by bpf_check_uarg_tail_zero().
pahole -C bpf_map_info ./vmlinux
struct bpf_map_info {
...
__u64 hash __attribute__((__aligned__(8))); /* 88 8 */
__u32 hash_size; /* 96 4 */
/* size: 104, cachelines: 2, members: 18 */
/* padding: 4 */
/* forced alignments: 1 */
/* last cacheline: 40 bytes */
} __attribute__((__aligned__(8)));
If a future kernel extension adds a new 4-byte field, older userspace
programs allocating this structure on the stack might inadvertently pass
uninitialized stack garbage into the new field, permanently breaking
backward compatibility. -- sashiko [1]
Fix it by changing sizeof(info) to
offsetofend(struct bpf_map_info, hash_size).
And, add "__u32 :32" to the tail of struct bpf_map_info.
[1] https://lore.kernel.org/bpf/20260513224823.6494FC19425@smtp.kernel.org/
Fixes: ea2e6467ac36 ("bpf: Return hashes of maps in BPF_OBJ_GET_INFO_BY_FD")
Acked-by: Mykyta Yatsenko <yatsenko@meta.com>
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/r/20260605155249.20772-2-leon.hwang@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
||
 Mykyta Yatsenko
|
1369dc002d |
bpf: Implement resizable hashmap basic functions
Use rhashtable_lookup_likely() for lookups, rhashtable_remove_fast() for deletes, and rhashtable_lookup_get_insert_fast() for inserts. Updates modify values in place under RCU rather than allocating a new element and swapping the pointer (as regular htab does). This trades read consistency for performance: concurrent readers may see partial updates. BPF_F_LOCK support and special-field handling (timers, kptrs, etc.) follow in a later commit. Initialize rhashtable with bpf_mem_alloc element cache. Require BPF_F_NO_PREALLOC. Limit max_entries to 2^31. Free elements via rhashtable_free_and_destroy(). Signed-off-by: Mykyta Yatsenko <yatsenko@meta.com> Link: https://lore.kernel.org/r/20260605-rhash-v7-4-5b8e05f8630d@meta.com Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
 Dragos Tatulea
|
04e0c4cdbf |
netdev: expose io_uring rx_page_order order via netlink
This adds observability for the io_uring zcrx rx-buf-len configuration. Signed-off-by: Dragos Tatulea <dtatulea@nvidia.com> Reviewed-by: Yael Chemla <ychemla@nvidia.com> Reviewed-by: Tariq Toukan <tariqt@nvidia.com> Acked-by: Pavel Begunkov <asml.silence@gmail.com> Link: https://patch.msgid.link/20260612211709.1456966-3-dtatulea@nvidia.com Signed-off-by: Jakub Kicinski <kuba@kernel.org> |
||
 Louis Scalbert
|
1c1b4bdf74 |
bonding: 3ad: add lacp_strict configuration knob
When an 802.3ad (LACP) bonding interface has no slaves in the
collecting/distributing state, the bonding master still reports
carrier as up as long as at least 'min_links' slaves have carrier.
In this situation, only one slave is effectively used for TX/RX,
while traffic received on other slaves is dropped. Upper-layer
daemons therefore consider the interface operational, even though
traffic may be blackholed if the lack of LACP negotiation means
the partner is not ready to deal with traffic.
Introduce a configuration knob to control this behavior. It allows
the bonding master to assert carrier only when at least 'min_links'
slaves are in Collecting_Distributing state.
The default mode preserves the existing behavior. This patch only
introduces the knob; its behavior is implemented in the subsequent
commit.
Fixes: 655f8919d549 ("bonding: add min links parameter to 802.3ad")
Signed-off-by: Louis Scalbert <louis.scalbert@6wind.com>
Acked-by: Jay Vosburgh <jv@jvosburgh.net>
Link: https://patch.msgid.link/20260603150331.1919611-4-louis.scalbert@6wind.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
||
 Kuniyuki Iwashima
|
494ba2a44d |
geneve: Introduce IFLA_GENEVE_LOCAL and IFLA_GENEVE_LOCAL6.
By default, a GENEVE device bind()s its underlying UDP socket(s) to
the IPv4 or IPv6 wildcard address because there is no way to specify
a specific local IP address to bind() to.
This prevents deploying multiple GENEVE devices on a multi-homed host
where each device should be isolated and bound to a different local IP
address on the same UDP port.
Let's introduce new options, IFLA_GENEVE_LOCAL and IFLA_GENEVE_LOCAL6,
to allow specifying a local IPv4/IPv6 address for the backend UDP
socket.
By default, when collect metadata mode (IFLA_GENEVE_COLLECT_METADATA)
is enabled, both IPv4 and IPv6 sockets are created. However, if a
source address is specified via the new attributes, only a single
socket corresponding to that specific address family is created.
Accordingly, geneve_find_sock() and geneve_find_dev() are updated to
take the source address into account, ensuring that multiple devices
and sockets configured with different source addresses can coexist
without conflict.
In addition, the source address is validated in geneve_xmit_skb()
and geneve6_xmit_skb(), so the BPF prog must set it in bpf_tunnel_key.
With this change, multiple GENEVE devices can be successfully created
and bound to their respective local IP addresses:
(*) "local" is the keyword for IFLA_GENEVE_LOCAL / IFLA_GENEVE_LOCAL6
# for i in $(seq 1 2);
do
ip link add geneve4_${i} type geneve local 192.168.0.${i} external
ip addr add 192.168.0.${i}/24 dev geneve4_${i}
ip link set geneve4_${i} up
ip link add geneve6_${i} type geneve local 2001:9292::${i} external
ip addr add 2001:9292::${i}/64 dev geneve6_${i} nodad
ip link set geneve6_${i} up
done
# ip -d l | grep geneve
9: geneve4_1: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
geneve external id 0 local 192.168.0.1 ...
10: geneve6_1: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
geneve external id 0 local 2001:9292::1 ...
11: geneve4_2: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
geneve external id 0 local 192.168.0.2 ...
12: geneve6_2: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
geneve external id 0 local 2001:9292::2 ...
# ss -ua | grep geneve
UNCONN 0 0 192.168.0.2:geneve 0.0.0.0:*
UNCONN 0 0 192.168.0.1:geneve 0.0.0.0:*
UNCONN 0 0 [2001:9292::2]:geneve *:*
UNCONN 0 0 [2001:9292::1]:geneve *:*
Note that even if the local address is explicitly configured with
the wildcard address, kernel does not dump it except for devices with
IFLA_GENEVE_COLLECT_METADATA. This is consistent with the behaviour
of is_tnl_info_zero(), which treats the wildcard remote address as not
configured.
## ynl example.
# ./tools/net/ynl/pyynl/cli.py \
--spec ./Documentation/netlink/specs/rt-link.yaml \
--do newlink --create \
--json '{"ifname": "geneve0",
"linkinfo": {"kind":"geneve",
"data": {"local": "0.0.0.0",
"collect-metadata": true}}}'
# ./tools/net/ynl/pyynl/cli.py \
--spec ./Documentation/netlink/specs/rt-link.yaml \
--do getlink \
--json '{"ifname": "geneve0"}' --output-json | \
jq .linkinfo.data.local
"0.0.0.0"
Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
Link: https://patch.msgid.link/20260602190436.139591-6-kuniyu@google.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
||
 Danielle Ratson
|
cc1ee3742d |
bridge: uapi: Add neigh_forward_grat netlink attributes
Add netlink attributes for controlling gratuitous ARP and unsolicited NA forwarding when neighbor suppression is enabled. Add IFLA_BRPORT_NEIGH_FORWARD_GRAT for port-level control and BRIDGE_VLANDB_ENTRY_NEIGH_FORWARD_GRAT for per-VLAN control. The new attributes provide independent control of gratuitous ARP and unsolicited NA packets. Operators can enable forwarding for those packets for fast mobility across VTEPs while keeping general neighbor suppression active. Reviewed-by: Ido Schimmel <idosch@nvidia.com> Signed-off-by: Danielle Ratson <danieller@nvidia.com> Link: https://patch.msgid.link/20260511065936.4173106-2-danieller@nvidia.com Signed-off-by: Paolo Abeni <pabeni@redhat.com> |
||
 Stephen Hemminger
|
1729da885e |
net/sched: netem: add per-impairment extended statistics
Add 64-bit counters for each impairment netem applies (delay, loss, ECN marking, corruption, duplication, reordering) and for skb allocation failures during enqueue. Exposed through TCA_STATS_APP as struct tc_netem_xstats. Counters increment when an impairment is occurs, independent of later events that may mask its on-wire effect. Added allocation_errors (similar to sch_fq) to account for when impairment could not be applied due to memory pressure, etc. Signed-off-by: Stephen Hemminger <stephen@networkplumber.org> Link: https://patch.msgid.link/20260509171123.307549-6-stephen@networkplumber.org Signed-off-by: Paolo Abeni <pabeni@redhat.com> |
||
 Dorjoy Chowdhury
|
b03713be6f |
openat2: new OPENAT2_REGULAR flag support
This flag indicates the path should be opened if it's a regular file. This is useful to write secure programs that want to avoid being tricked into opening device nodes with special semantics while thinking they operate on regular files. This is a requested feature from the uapi-group[1]. The previously introduced EFTYPE error code is returned when the path doesn't refer to a regular file. For example, if openat2 is called on path /dev/null with OPENAT2_REGULAR in the flag param, it will return -EFTYPE. When used in combination with O_CREAT, either the regular file is created, or if the path already exists, it is opened if it's a regular file. Otherwise, -EFTYPE is returned. When OPENAT2_REGULAR is combined with O_DIRECTORY, -EINVAL is returned as it doesn't make sense to open a path that is both a directory and a regular file. The UAPI bit lives in the upper 32 bits of open_how::flags (((__u64)1 << 32)) so that open(2) and openat(2) -- whose @flags argument is a C int -- cannot physically express it. This is a structural guarantee, not a runtime mask: the bit is unrepresentable in 32 bits. Because the rest of the VFS open path narrows to 32 bits in several places (op->open_flag, f->f_flags, the unsigned open_flag argument of i_op->atomic_open()), build_open_flags() translates OPENAT2_REGULAR into a kernel-internal lower-32-bit carrier __O_REGULAR (bit 4, unused as an O_* on every architecture) before the assignment to op->open_flag. __O_REGULAR then rides through the existing channels exactly like __FMODE_EXEC. do_dentry_open() strips it so it cannot leak back to userspace via fcntl(F_GETFL). Four BUILD_BUG_ON_MSG() invariants in build_open_flags() prevent any future bit collision or accidental low-32 redefinition: - VALID_OPEN_FLAGS fits in 32 bits. - OPENAT2_REGULAR lives in the upper 32 bits. - OPENAT2_REGULAR does not alias any open()/openat() flag. - __O_REGULAR does not alias any user-visible flag. [1]: https://uapi-group.org/kernel-features/#ability-to-only-open-regular-files Christian Brauner <brauner@kernel.org> says: Move OPENAT2_REGULAR to the upper 32 bits of open_how::flags with a kernel-internal __O_REGULAR carrier so that open(2)/openat(2) cannot encode the flag; add BUILD_BUG_ON_MSG() invariants and register __O_REGULAR in the fcntl_init() allocation-uniqueness BUILD_BUG_ON() (bit count 21 -> 22). Signed-off-by: Dorjoy Chowdhury <dorjoychy111@gmail.com> Link: https://patch.msgid.link/20260328172314.45807-2-dorjoychy111@gmail.com Reviewed-by: Jeff Layton <jlayton@kernel.org> Reviewed-by: Aleksa Sarai <aleksa@amutable.com> Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org> |
||
|
Leon Hwang
|
978fc80ec7 |
bpf: Extend BPF syscall with common attributes support
Add generic BPF syscall support for passing common attributes.
The initial set of common attributes includes:
1. 'log_buf': User-provided buffer for storing logs.
2. 'log_size': Size of the log buffer.
3. 'log_level': Log verbosity level.
4. 'log_true_size': Actual log size reported by kernel.
The common-attribute pointer and its size are passed as the 4th and 5th
syscall arguments. A new command bit, 'BPF_COMMON_ATTRS' ('1 << 16'),
indicates that common attributes are supplied.
This commit adds syscall and uapi plumbing. Command-specific handling is
added in follow-up patches.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/r/20260512153157.28382-2-leon.hwang@linux.dev
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
|
||
 Alan Maguire
|
9801e02973 |
bpf: Extend BTF UAPI vlen, kinds to use unused bits
BTF maximum vlen is encoded using 16 bits with a maximum vlen of 65535. This has sufficed for structs, function parameters and enumerated type values. However, with upcoming BTF location information - in particular information about inline sites - this limit is surpassed. Use bits 16-23 - currently unused in BTF info - to extend to 24 bits, giving a max vlen of (2^24 - 1), or 16 million. Also extend BTF kind encoding from 5 to 7 bits, giving a maximum available number of kinds of 128. Since with the BTF location work we use another 3 kinds, we are fast approaching the current limit of 32. Convert BTF_MAX_* values to enums to allow them to be encoded in kernel BTF; this will allow us to detect if the running kernel supports a 24-bit vlen or not. Add one for max _possible_ (not used) kind. Fix up a few places in the kernel where a 16-bit vlen is assumed; remove BTF_INFO_MASK as now all bits are used. The vlen expansion was suggested by Andrii in [1]; the kind expansion is tackled here too as it may be needed also to support new kinds in BTF. [1] https://lore.kernel.org/bpf/CAEf4BzZx=X6vGqcA8SPU6D+v6k+TR=ZewebXMuXtpmML058piw@mail.gmail.com/ Suggested-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Alan Maguire <alan.maguire@oracle.com> Acked-by: Mykyta Yatsenko <yatsenko@meta.com> Link: https://lore.kernel.org/r/20260417143023.1551481-2-alan.maguire@oracle.com Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
 Andy Roulin
|
86525c190b |
net: bridge: add stp_mode attribute for STP mode selection
The bridge-stp usermode helper is currently restricted to the initial
network namespace, preventing userspace STP daemons (e.g. mstpd) from
operating on bridges in other network namespaces. Since commit
ff62198553e4 ("bridge: Only call /sbin/bridge-stp for the initial
network namespace"), bridges in non-init namespaces silently fall
back to kernel STP with no way to use userspace STP.
Add a new bridge attribute IFLA_BR_STP_MODE that allows explicit
per-bridge control over STP mode selection:
BR_STP_MODE_AUTO (default) - Existing behavior: invoke the
/sbin/bridge-stp helper in init_net only; fall back to kernel STP
if it fails or in non-init namespaces.
BR_STP_MODE_USER - Directly enable userspace STP (BR_USER_STP)
without invoking the helper. Works in any network namespace.
Userspace is responsible for ensuring an STP daemon manages the
bridge.
BR_STP_MODE_KERNEL - Directly enable kernel STP (BR_KERNEL_STP)
without invoking the helper.
The mode can only be changed while STP is disabled, or set to the
same value (-EBUSY otherwise). IFLA_BR_STP_MODE is processed before
IFLA_BR_STP_STATE in br_changelink(), so both can be set atomically
in a single netlink message. The mode can also be changed in the
same message that disables STP.
The stp_mode struct field is u8 since all possible values fit, while
NLA_U32 is used for the netlink attribute since it occupies the same
space in the netlink message as NLA_U8.
A new stp_helper_active boolean tracks whether the /sbin/bridge-stp
helper was invoked during br_stp_start(), so that br_stp_stop() only
calls the helper for stop when it was called for start. This avoids
calling the helper asymmetrically when stp_mode changes between
start and stop.
Suggested-by: Ido Schimmel <idosch@nvidia.com>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Ido Schimmel <idosch@nvidia.com>
Acked-by: Nikolay Aleksandrov <nikolay@nvidia.com>
Signed-off-by: Andy Roulin <aroulin@nvidia.com>
Link: https://patch.msgid.link/20260405205224.3163000-2-aroulin@nvidia.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
||
 Daniel Borkmann
|
8159a62fa8 |
netkit: Add single device mode for netkit
Add a single device mode for netkit instead of netkit pairs. The primary target for the paired devices is to connect network namespaces, of course, and support has been implemented in projects like Cilium [0]. For the rxq leasing the plan is to support two main scenarios related to single device mode: * For the use-case of io_uring zero-copy, the control plane can either set up a netkit pair where the peer device can perform rxq leasing which is then tied to the lifetime of the peer device, or the control plane can use a regular netkit pair to connect the hostns to a Pod/container and dynamically add/remove rxq leasing through a single device without having to interrupt the device pair. In the case of io_uring, the memory pool is used as skb non-linear pages, and thus the skb will go its way through the regular stack into netkit. Things like the netkit policy when no BPF is attached or skb scrubbing etc apply as-is in case the paired devices are used, or if the backend memory is tied to the single device and traffic goes through a paired device. * For the use-case of AF_XDP, the control plane needs to use netkit in the single device mode. The single device mode currently enforces only a pass policy when no BPF is attached, and does not yet support BPF link attachments for AF_XDP. skbs sent to that device get dropped at the moment. Given AF_XDP operates at a lower layer of the stack tying this to the netkit pair did not make sense. In future, the plan is to allow BPF at the XDP layer which can: i) process traffic coming from the AF_XDP application (e.g. QEMU with AF_XDP backend) to filter egress traffic or to push selected egress traffic up to the single netkit device to the local stack (e.g. DHCP requests), and ii) vice-versa skbs sent to the single netkit into the AF_XDP application (e.g. DHCP replies). Also, the control-plane can dynamically manage rxq leasing for the single netkit device without having to interrupt (e.g. down/up cycle) the main netkit pair for the Pod which has traffic going in and out. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Co-developed-by: David Wei <dw@davidwei.uk> Signed-off-by: David Wei <dw@davidwei.uk> Reviewed-by: Jordan Rife <jordan@jrife.io> Reviewed-by: Nikolay Aleksandrov <razor@blackwall.org> Link: https://docs.cilium.io/en/stable/operations/performance/tuning/#netkit-device-mode [0] Link: https://patch.msgid.link/20260402231031.447597-11-daniel@iogearbox.net Signed-off-by: Jakub Kicinski <kuba@kernel.org> |
||
|
Daniel Borkmann
|
c361e88771 |
net: Add queue-create operation
Add a ynl netdev family operation called queue-create that creates a
new queue on a netdevice:
name: queue-create
attribute-set: queue
flags: [admin-perm]
do:
request:
attributes:
- ifindex
- type
- lease
reply: &queue-create-op
attributes:
- id
This is a generic operation such that it can be extended for various
use cases in future. Right now it is mandatory to specify ifindex,
the queue type which is enforced to rx and a lease. The newly created
queue id is returned to the caller.
A queue from a virtual device can have a lease which refers to another
queue from a physical device. This is useful for memory providers
and AF_XDP operations which take an ifindex and queue id to allow
applications to bind against virtual devices in containers. The lease
couples both queues together and allows to proxy the operations from
a virtual device in a container to the physical device.
In future, the nested lease attribute can be lifted and made optional
for other use-cases such as dynamic queue creation for physical
netdevs. The lack of lease and the specification of the physical
device as an ifindex will imply that we need a real queue to be
allocated. Similarly, the queue type enforcement to rx can then be
lifted as well to support tx.
An early implementation had only driver-specific integration [0], but
in order for other virtual devices to reuse, it makes sense to have
this as a generic API in core net.
For leasing queues, the virtual netdev must have real_num_rx_queues
less than num_rx_queues at the time of calling queue-create. The
queue-type must be rx as only rx queues are supported for leasing
for now. We also enforce that the queue-create ifindex must point
to a virtual device, and that the nested lease attribute's ifindex
must point to a physical device. The nested lease attribute set
contains a netns-id attribute which is optional and can specify a
netns-id relative to the caller's netns. It requires cap_net_admin
and if the netns-id attribute is not specified, the lease ifindex
will be retrieved from the current netns. Also, it is modeled as
an s32 type similarly as done elsewhere in the stack.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Co-developed-by: David Wei <dw@davidwei.uk>
Signed-off-by: David Wei <dw@davidwei.uk>
Acked-by: Stanislav Fomichev <sdf@fomichev.me>
Reviewed-by: Nikolay Aleksandrov <razor@blackwall.org>
Link: https://bpfconf.ebpf.io/bpfconf2025/bpfconf2025_material/lsfmmbpf_2025_netkit_borkmann.pdf [0]
Link: https://patch.msgid.link/20260402231031.447597-2-daniel@iogearbox.net
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
||
 Hangbin Liu
|
b6f2d5c84d |
bonding: print churn state via netlink
Currently, the churn state is printed only in sysfs. Add netlink support so users could get the state via netlink. Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> Link: https://patch.msgid.link/20260224020215.6012-1-liuhangbin@gmail.com Signed-off-by: Paolo Abeni <pabeni@redhat.com> |
||
 Eyal Birger
|
c2838bdfe5 |
bpf: Clarify BPF_RB_NO_WAKEUP behavior for bpf_ringbuf_discard()
Clarify bpf_ringbuf_discard() documentation for BPF_RB_NO_WAKEUP. Discarded ring buffer records are still left in the ring buffer and are only skipped when user space consumes them. This can matter when BPF_RB_NO_WAKEUP is used: a later submit relying on adaptive wakeup might not wake the consumer, because the discarded record still needs to be consumed first. Scenario: epoll_wait(rb_fd); // blocks rec = bpf_ringbuf_reserve(&rb, ...); bpf_ringbuf_discard(rec, BPF_RB_NO_WAKEUP); rec = bpf_ringbuf_reserve(&rb, ...); bpf_ringbuf_submit(rec, 0); // valid record, but no wakeup Document this in bpf_ringbuf_discard() to make the interaction between discarded records, user-space consumption, and adaptive wakeups explicit. Reported-by: Shmulik Ladkani <shmulik.ladkani@gmail.com> Signed-off-by: Eyal Birger <eyal.birger@gmail.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260331130612.3762433-1-eyal.birger@gmail.com ---- v2: adapt wording per feedback from Andrii. |
||
|
Alan Maguire
|
462733ecb8 |
btf: Add BTF kind layout encoding to UAPI
BTF kind layouts provide information to parse BTF kinds. By separating parsing BTF from using all the information it provides, we allow BTF to encode new features even if they cannot be used by readers. This will be helpful in particular for cases where older tools are used to parse newer BTF with kinds the older tools do not recognize; the BTF can still be parsed in such cases using kind layout. The intent is to support encoding of kind layouts optionally so that tools like pahole can add this information. For each kind, we record - length of singular element following struct btf_type - length of each of the btf_vlen() elements following - a (currently unused) flags field The ideas here were discussed at [1], [2]; hence Suggested-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Alan Maguire <alan.maguire@oracle.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20260326145444.2076244-2-alan.maguire@oracle.com [1] https://lore.kernel.org/bpf/CAEf4BzYjWHRdNNw4B=eOXOs_ONrDwrgX4bn=Nuc1g8JPFC34MA@mail.gmail.com/ [2] https://lore.kernel.org/bpf/20230531201936.1992188-1-alan.maguire@oracle.com/ |
||
 Paolo Abeni
|
a61d018702 |
geneve: add netlink support for GRO hint
Allow configuring and dumping the new device option, and cache its value into the geneve socket itself. The new option is not tie to it any code yet. Signed-off-by: Paolo Abeni <pabeni@redhat.com> Link: https://patch.msgid.link/2295d4e4d1e919a3189425141bbc71c7850a2de0.1769011015.git.pabeni@redhat.com Signed-off-by: Jakub Kicinski <kuba@kernel.org> |
||
 Jakub Kicinski
|
13ae2d1841 |
Revert "Merge branch 'netkit-support-for-io_uring-zero-copy-and-af_xdp'"
This reverts commit 77b9c4a438fc66e2ab004c411056b3fb71a54f2c, reversing
changes made to 4515ec4ad58a37e70a9e1256c0b993958c9b7497:
931420a2fc36 ("selftests/net: Add netkit container tests")
ab771c938d9a ("selftests/net: Make NetDrvContEnv support queue leasing")
6be87fbb2776 ("selftests/net: Add env for container based tests")
61d99ce3dfc2 ("selftests/net: Add bpf skb forwarding program")
920da3634194 ("netkit: Add xsk support for af_xdp applications")
eef51113f8af ("netkit: Add netkit notifier to check for unregistering devices")
b5ef109d22d4 ("netkit: Implement rtnl_link_ops->alloc and ndo_queue_create")
b5c3fa4a0b16 ("netkit: Add single device mode for netkit")
0073d2fd679d ("xsk: Proxy pool management for leased queues")
1ecea95dd3b5 ("xsk: Extend xsk_rcv_check validation")
804bf334d08a ("net: Proxy netdev_queue_get_dma_dev for leased queues")
0caa9a8ddec3 ("net: Proxy net_mp_{open,close}_rxq for leased queues")
ff8889ff9107 ("net, ethtool: Disallow leased real rxqs to be resized")
9e2103f36110 ("net: Add lease info to queue-get response")
31127deddef4 ("net: Implement netdev_nl_queue_create_doit")
a5546e18f77c ("net: Add queue-create operation")
The series will conflict with io_uring work, and the code needs more
polish.
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
|
||
|
Daniel Borkmann
|
05121e3fd9 |
netkit: Add single device mode for netkit
Add a single device mode for netkit instead of netkit pairs. The primary target for the paired devices is to connect network namespaces, of course, and support has been implemented in projects like Cilium [0]. For the rxq leasing the plan is to support two main scenarios related to single device mode: * For the use-case of io_uring zero-copy, the control plane can either set up a netkit pair where the peer device can perform rxq leasing which is then tied to the lifetime of the peer device, or the control plane can use a regular netkit pair to connect the hostns to a Pod/container and dynamically add/remove rxq leasing through a single device without having to interrupt the device pair. In the case of io_uring, the memory pool is used as skb non-linear pages, and thus the skb will go its way through the regular stack into netkit. Things like the netkit policy when no BPF is attached or skb scrubbing etc apply as-is in case the paired devices are used, or if the backend memory is tied to the single device and traffic goes through a paired device. * For the use-case of AF_XDP, the control plane needs to use netkit in the single device mode. The single device mode currently enforces only a pass policy when no BPF is attached, and does not yet support BPF link attachments for AF_XDP. skbs sent to that device get dropped at the moment. Given AF_XDP operates at a lower layer of the stack tying this to the netkit pair did not make sense. In future, the plan is to allow BPF at the XDP layer which can: i) process traffic coming from the AF_XDP application (e.g. QEMU with AF_XDP backend) to filter egress traffic or to push selected egress traffic up to the single netkit device to the local stack (e.g. DHCP requests), and ii) vice-versa skbs sent to the single netkit into the AF_XDP application (e.g. DHCP replies). Also, the control-plane can dynamically manage rxq leasing for the single netkit device without having to interrupt (e.g. down/up cycle) the main netkit pair for the Pod which has traffic going in and out. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Co-developed-by: David Wei <dw@davidwei.uk> Signed-off-by: David Wei <dw@davidwei.uk> Reviewed-by: Jordan Rife <jordan@jrife.io> Reviewed-by: Nikolay Aleksandrov <razor@blackwall.org> Link: https://docs.cilium.io/en/stable/operations/performance/tuning/#netkit-device-mode [0] Link: https://patch.msgid.link/20260115082603.219152-10-daniel@iogearbox.net Signed-off-by: Paolo Abeni <pabeni@redhat.com> |
||
|
Daniel Borkmann
|
a623828cdc |
net: Add queue-create operation
Add a ynl netdev family operation called queue-create that creates a
new queue on a netdevice:
name: queue-create
attribute-set: queue
flags: [admin-perm]
do:
request:
attributes:
- ifindex
- type
- lease
reply: &queue-create-op
attributes:
- id
This is a generic operation such that it can be extended for various
use cases in future. Right now it is mandatory to specify ifindex,
the queue type which is enforced to rx and a lease. The newly created
queue id is returned to the caller.
A queue from a virtual device can have a lease which refers to another
queue from a physical device. This is useful for memory providers
and AF_XDP operations which take an ifindex and queue id to allow
applications to bind against virtual devices in containers. The lease
couples both queues together and allows to proxy the operations from
a virtual device in a container to the physical device.
In future, the nested lease attribute can be lifted and made optional
for other use-cases such as dynamic queue creation for physical
netdevs. The lack of lease and the specification of the physical
device as an ifindex will imply that we need a real queue to be
allocated. Similarly, the queue type enforcement to rx can then be
lifted as well to support tx.
An early implementation had only driver-specific integration [0], but
in order for other virtual devices to reuse, it makes sense to have
this as a generic API in core net.
For leasing queues, the virtual netdev must have real_num_rx_queue
less than num_rx_queues at the time of calling queue-create. The
queue-type must be rx as only rx queues are supported for leasing
for now. We also enforce that the queue-create ifindex must point
to a virtual device, and that the nested lease attribute's ifindex
must point to a physical device. The nested lease attribute set
contains a netns-id attribute which is currently only intended for
dumping as part of the queue-get operation. Also, it is modeled as
an s32 type similarly as done elsewhere in the stack.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Co-developed-by: David Wei <dw@davidwei.uk>
Signed-off-by: David Wei <dw@davidwei.uk>
Link: https://bpfconf.ebpf.io/bpfconf2025/bpfconf2025_material/lsfmmbpf_2025_netkit_borkmann.pdf [0]
Acked-by: Stanislav Fomichev <sdf@fomichev.me>
Reviewed-by: Nikolay Aleksandrov <razor@blackwall.org>
Link: https://patch.msgid.link/20260115082603.219152-2-daniel@iogearbox.net
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
|
||
 Jonas Köppeler
|
4f8bdd9ce3 |
net/sched: sch_cake: share shaper state across sub-instances of cake_mq
This commit adds shared shaper state across the cake instances beneath a cake_mq qdisc. It works by periodically tracking the number of active instances, and scaling the configured rate by the number of active queues. The scan is lockless and simply reads the qlen and the last_active state variable of each of the instances configured beneath the parent cake_mq instance. Locking is not required since the values are only updated by the owning instance, and eventual consistency is sufficient for the purpose of estimating the number of active queues. The interval for scanning the number of active queues is set to 200 us. We found this to be a good tradeoff between overhead and response time. For a detailed analysis of this aspect see the Netdevconf talk: https://netdevconf.info/0x19/docs/netdev-0x19-paper16-talk-paper.pdf Reviewed-by: Jamal Hadi Salim <jhs@mojatatu.com> Signed-off-by: Jonas Köppeler <j.koeppeler@tu-berlin.de> Signed-off-by: Toke Høiland-Jørgensen <toke@redhat.com> Link: https://patch.msgid.link/20260109-mq-cake-sub-qdisc-v8-5-8d613fece5d8@redhat.com Signed-off-by: Paolo Abeni <pabeni@redhat.com> |
||
 Dapeng Mi
|
ad0d0e5112 |
perf/x86/intel: Add support for PEBS memory auxiliary info field in DMR
With the introduction of the OMR feature, the PEBS memory auxiliary info field for load and store latency events has been restructured for DMR. The memory auxiliary info field's bit[8] indicates whether a L2 cache miss occurred for a memory load or store instruction. If bit[8] is 0, it signifies no L2 cache miss, and bits[7:0] specify the exact cache data source (up to the L2 cache level). If bit[8] is 1, bits[7:0] represent the OMR encoding, indicating the specific L3 cache or memory region involved in the memory access. A significant enhancement is OMR encoding provides up to 8 fine-grained memory regions besides the cache region. A significant enhancement for OMR encoding is the ability to provide up to 8 fine-grained memory regions in addition to the cache region, offering more detailed insights into memory access regions. For detailed information on the memory auxiliary info encoding, please refer to section 16.2 "PEBS LOAD LATENCY AND STORE LATENCY FACILITY" in the ISE documentation. This patch ensures that the PEBS memory auxiliary info field is correctly interpreted and utilized in DMR. Signed-off-by: Dapeng Mi <dapeng1.mi@linux.intel.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://patch.msgid.link/20260114011750.350569-3-dapeng1.mi@linux.intel.com |
||
 Bill Wendling
|
1317132162 |
compiler_types.h: Attributes: Add __counted_by_ptr macro
Introduce __counted_by_ptr(), which works like __counted_by(), but for
pointer struct members.
struct foo {
int a, b, c;
char *buffer __counted_by_ptr(bytes);
short nr_bars;
struct bar *bars __counted_by_ptr(nr_bars);
size_t bytes;
};
Because "counted_by" can only be applied to pointer members in very
recent compiler versions, its application ends up needing to be distinct
from flexibe array "counted_by" annotations, hence a separate macro.
This is a reworking of Kees' previous patch [1].
Link: https://lore.kernel.org/all/20251020220118.1226740-1-kees@kernel.org/ [1]
Co-developed-by: Kees Cook <kees@kernel.org>
Signed-off-by: Bill Wendling <morbo@google.com>
Link: https://patch.msgid.link/20260116005838.2419118-1-morbo@google.com
Signed-off-by: Kees Cook <kees@kernel.org>
|
||
 Matt Bobrowski
|
ed6bb65cf1 |
bpf: add new BPF_CGROUP_ITER_CHILDREN control option
Currently, the BPF cgroup iterator supports walking descendants in either pre-order (BPF_CGROUP_ITER_DESCENDANTS_PRE) or post-order (BPF_CGROUP_ITER_DESCENDANTS_POST). These modes perform an exhaustive depth-first search (DFS) of the hierarchy. In scenarios where a BPF program may need to inspect only the direct children of a given parent cgroup, a full DFS is unnecessarily expensive. This patch introduces a new BPF cgroup iterator control option, BPF_CGROUP_ITER_CHILDREN. This control option restricts the traversal to the immediate children of a specified parent cgroup, allowing for more targeted and efficient iteration, particularly when exhaustive depth-first search (DFS) traversal is not required. Signed-off-by: Matt Bobrowski <mattbobrowski@google.com> Link: https://lore.kernel.org/r/20260127085112.3608687-1-mattbobrowski@google.com Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
 Menglong Dong
|
adde4f55b7 |
bpf: add fsession support
The fsession is something that similar to kprobe session. It allow to attach a single BPF program to both the entry and the exit of the target functions. Introduce the struct bpf_fsession_link, which allows to add the link to both the fentry and fexit progs_hlist of the trampoline. Signed-off-by: Menglong Dong <dongml2@chinatelecom.cn> Co-developed-by: Leon Hwang <leon.hwang@linux.dev> Signed-off-by: Leon Hwang <leon.hwang@linux.dev> Link: https://lore.kernel.org/r/20260124062008.8657-2-dongml2@chinatelecom.cn Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
 Thomas Gleixner
|
5d02120e10 |
treewide: Update email address
In a vain attempt to consolidate the email zoo switch everything to the kernel.org account. Signed-off-by: Thomas Gleixner <tglx@kernel.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> |
||
|
Leon Hwang
|
a6d7ceaaeb |
bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags
Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags and check them for following APIs: * 'map_lookup_elem()' * 'map_update_elem()' * 'generic_map_lookup_batch()' * 'generic_map_update_batch()' And, get the correct value size for these APIs. Acked-by: Andrii Nakryiko <andrii@kernel.org> Signed-off-by: Leon Hwang <leon.hwang@linux.dev> Link: https://lore.kernel.org/r/20260107022022.12843-2-leon.hwang@linux.dev Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
 Thomas Weißschuh
|
e64e125ef6 |
vfs: use UAPI types for new struct delegation definition
Using libc types and headers from the UAPI headers is problematic as it
introduces a dependency on a full C toolchain.
Use the fixed-width integer types provided by the UAPI headers instead.
Fixes: 1602bad16d7d ("vfs: expose delegation support to userland")
Fixes: 4be9e04ebf75 ("vfs: add needed headers for new struct delegation definition")
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Link: https://patch.msgid.link/20251203-uapi-fcntl-v1-1-490c67bf3425@linutronix.de
Acked-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Jeff Layton <jlayton@kernel.org>
Signed-off-by: Christian Brauner <brauner@kernel.org>
|
||
|
Amery Hung
|
1a41b12b4f |
bpf: Support associating BPF program with struct_ops
Add a new BPF command BPF_PROG_ASSOC_STRUCT_OPS to allow associating a BPF program with a struct_ops map. This command takes a file descriptor of a struct_ops map and a BPF program and set prog->aux->st_ops_assoc to the kdata of the struct_ops map. The command does not accept a struct_ops program nor a non-struct_ops map. Programs of a struct_ops map is automatically associated with the map during map update. If a program is shared between two struct_ops maps, prog->aux->st_ops_assoc will be poisoned to indicate that the associated struct_ops is ambiguous. The pointer, once poisoned, cannot be reset since we have lost track of associated struct_ops. For other program types, the associated struct_ops map, once set, cannot be changed later. This restriction may be lifted in the future if there is a use case. A kernel helper bpf_prog_get_assoc_struct_ops() can be used to retrieve the associated struct_ops pointer. The returned pointer, if not NULL, is guaranteed to be valid and point to a fully updated struct_ops struct. For struct_ops program reused in multiple struct_ops map, the return will be NULL. prog->aux->st_ops_assoc is protected by bumping the refcount for non-struct_ops programs and RCU for struct_ops programs. Since it would be inefficient to track programs associated with a struct_ops map, every non-struct_ops program will bump the refcount of the map to make sure st_ops_assoc stays valid. For a struct_ops program, it is protected by RCU as map_free will wait for an RCU grace period before disassociating the program with the map. The helper must be called in BPF program context or RCU read-side critical section. struct_ops implementers should note that the struct_ops returned may not be initialized nor attached yet. The struct_ops implementer will be responsible for tracking and checking the state of the associated struct_ops map if the use case expects an initialized or attached struct_ops. Signed-off-by: Amery Hung <ameryhung@gmail.com> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Andrii Nakryiko <andrii@kernel.org> Acked-by: Martin KaFai Lau <martin.lau@kernel.org> Link: https://lore.kernel.org/bpf/20251203233748.668365-3-ameryhung@gmail.com |
||
 Asbjørn Sloth Tønnesen
|
7ac4e3a670 |
tools: ynl-gen: add regeneration comment
Add a comment on regeneration to the generated files. The comment is placed after the YNL-GEN line[1], as to not interfere with ynl-regen.sh's detection logic. [1] and after the optional YNL-ARG line. Link: https://lore.kernel.org/r/aR5m174O7pklKrMR@zx2c4.com/ Suggested-by: Jason A. Donenfeld <Jason@zx2c4.com> Signed-off-by: Asbjørn Sloth Tønnesen <ast@fiberby.net> Acked-by: Matthieu Baerts (NGI0) <matttbe@kernel.org> Link: https://patch.msgid.link/20251120174429.390574-3-ast@fiberby.net Signed-off-by: Jakub Kicinski <kuba@kernel.org> |
||
 Samiullah Khawaja
|
cd173d0ea3 |
net: Extend NAPI threaded polling to allow kthread based busy polling
Add a new state NAPI_STATE_THREADED_BUSY_POLL to the NAPI state enum to enable and disable threaded busy polling. When threaded busy polling is enabled for a NAPI, enable NAPI_STATE_THREADED also. When the threaded NAPI is scheduled, set NAPI_STATE_IN_BUSY_POLL to signal napi_complete_done not to rearm interrupts. Whenever NAPI_STATE_THREADED_BUSY_POLL is unset, the NAPI_STATE_IN_BUSY_POLL will be unset, napi_complete_done unsets the NAPI_STATE_SCHED_THREADED bit also, which in turn will make the kthread go to sleep. Signed-off-by: Samiullah Khawaja <skhawaja@google.com> Reviewed-by: Willem de Bruijn <willemb@google.com> Acked-by: Martin Karsten <mkarsten@uwaterloo.ca> Tested-by: Martin Karsten <mkarsten@uwaterloo.ca> Link: https://patch.msgid.link/20251028203007.575686-2-skhawaja@google.com Signed-off-by: Jakub Kicinski <kuba@kernel.org> |
||
|
Kuniyuki Iwashima
|
3fe0a72123 |
bpf: Introduce SK_BPF_BYPASS_PROT_MEM.
If a socket has sk->sk_bypass_prot_mem flagged, the socket opts out
of the global protocol memory accounting.
This is easily controlled by net.core.bypass_prot_mem sysctl, but it
lacks flexibility.
Let's support flagging (and clearing) sk->sk_bypass_prot_mem via
bpf_setsockopt() at the BPF_CGROUP_INET_SOCK_CREATE hook.
int val = 1;
bpf_setsockopt(ctx, SOL_SOCKET, SK_BPF_BYPASS_PROT_MEM,
&val, sizeof(val));
As with net.core.bypass_prot_mem, this is inherited to child sockets,
and BPF always takes precedence over sysctl at socket(2) and accept(2).
SK_BPF_BYPASS_PROT_MEM is only supported at BPF_CGROUP_INET_SOCK_CREATE
and not supported on other hooks for some reasons:
1. UDP charges memory under sk->sk_receive_queue.lock instead
of lock_sock()
2. Modifying the flag after skb is charged to sk requires such
adjustment during bpf_setsockopt() and complicates the logic
unnecessarily
We can support other hooks later if a real use case justifies that.
Most changes are inline and hard to trace, but a microbenchmark on
__sk_mem_raise_allocated() during neper/tcp_stream showed that more
samples completed faster with sk->sk_bypass_prot_mem == 1. This will
be more visible under tcp_mem pressure (but it's not a fair comparison).
# bpftrace -e 'kprobe:__sk_mem_raise_allocated { @start[tid] = nsecs; }
kretprobe:__sk_mem_raise_allocated /@start[tid]/
{ @end[tid] = nsecs - @start[tid]; @times = hist(@end[tid]); delete(@start[tid]); }'
# tcp_stream -6 -F 1000 -N -T 256
Without bpf prog:
[128, 256) 3846 | |
[256, 512) 1505326 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K) 1371006 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[1K, 2K) 198207 |@@@@@@ |
[2K, 4K) 31199 |@ |
With bpf prog in the next patch:
(must be attached before tcp_stream)
# bpftool prog load sk_bypass_prot_mem.bpf.o /sys/fs/bpf/test type cgroup/sock_create
# bpftool cgroup attach /sys/fs/cgroup/test cgroup_inet_sock_create pinned /sys/fs/bpf/test
[128, 256) 6413 | |
[256, 512) 1868425 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[512, 1K) 1101697 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
[1K, 2K) 117031 |@@@@ |
[2K, 4K) 11773 | |
Signed-off-by: Kuniyuki Iwashima <kuniyu@google.com>
Signed-off-by: Martin KaFai Lau <martin.lau@kernel.org>
Acked-by: Roman Gushchin <roman.gushchin@linux.dev>
Link: https://patch.msgid.link/20251014235604.3057003-6-kuniyu@google.com
|