In commit 4794f18bf4 ("sync: Sync .mailmap entries"), we updated the
sync-up script to automatically update libbpf's .mailmap; however, the
script would not take care of committing the changes. Let's address
this.
The code is copied and adapted from the part where we commit changes to
src/bpf_helper_defs.h.
Signed-off-by: Quentin Monnet <qmo@kernel.org>
The kernel repository has a .mailmap file to remap author names and
email addresses to their desired format in Git logs (for details, see
gitmailmap documentation [0]). Alas, this is only visible for author
information when looking at the logs locally, as GitHub does not support
mailmaps at the moment [1].
This commit adds a .mailmap file for libbpf, automatically generated
from the kernel's version. The script to generate the .mailmap is added,
too: it works by grepping email addresses from authors in the
repository, and collecting all lines ending with this address in the
kernel's .mailmap - in other words, all lines where this address is used
as a pattern for a remapping.
To keep the .mailmap up-to-date, add a call to the script to
sync-kernel.sh.
[0] https://git-scm.com/docs/gitmailmap
[1] https://github.com/orgs/community/discussions/22518
Signed-off-by: Quentin Monnet <qmo@kernel.org>
When syncing with the kernel, the script generates a cover letter for
the latest changes using "git format-patch". Unless specified otherwise,
it uses a signature (as in, email footer signature) which defaults to
the Git version in use, and ends up in the commit logs. This doesn't
bring any useful information in there: let's get rid of this version
number.
Signed-off-by: Quentin Monnet <quentin@isovalent.com>
After recent lint changes, commit_signature() function now gets optional
array of paths as multiple arguments, instead of entire array as second
argument. So adjust commit_signature() to handle this correctly.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Now that we enforce Signed-off-by on every commit, make sure that
auto-generatd sync commits also get corrected Signed-off-by tags.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Libbpf relies on F_DUPFD_CLOEXEC constant coming from fcntl.h UAPI
header, so we need to sync it along other UAPI headers. Also update sync
script to keep doing this automatically going forward.
Reported-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Use correct Bash syntax to define these two variables as arrays.
Drop shellcheck opt-out for unquoted use of array.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Don't wrap LIBBPF_PATHS[@] and LIBBPF_VIEW_PATHS[@] in quotes when
passing it to git commands. Not clear how it worked before, but
something recently broke. Either git commands became stricter or
something.
But either way, we do want to pass each element of LIBBPF_PATHS or
LIBBPF_VIEW_PATHS as separate command line arguments, so putting them in
quotes doesn't make sense, as that makes them look like a single
argument to git.
So drop all the quotes around these arrays. The only place where it's
still needed is in commit_signature call, as we do want to pass array as
single arg ($2) and then internally we unfold it into multiple command
line arguments.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
This header is necessary for libbpf-sys to generate perf-related Rust
bindings. It's more convenient to have it available locally with libbpf.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
git apply -3 doesn't always leave conflicted files in the working
directory. Use patch --merge instead, it seems to work better in more
complicated situations.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
`git am -3` will give up frequently even in cases when patch can be
auto-merged with:
```
Applying: libbpf: Unify low-level map creation APIs w/ new bpf_map_create()
error: sha1 information is lacking or useless (src/libbpf.c).
error: could not build fake ancestor
Patch failed at 0001 libbpf: Unify low-level map creation APIs w/ new bpf_map_create()
```
But `git apply -3` in the same situation will succeed with three-way merge just
fine:
```
error: patch failed: src/bpf_gen_internal.h:51
Falling back to three-way merge...
Applied patch to 'src/bpf_gen_internal.h' cleanly.
```
So if git am fails, try git apply and if that succeeds, automatically
`git am --continue`. If not, fallback to user actions.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
The sync-kernel.sh script has two consecutive tests for $BPF_BRANCH
being provided by the user (and so the second one can currently never
fail). Looking at the error message displayed in each case, we want to
keep the second one. Let's remove the first check.
Signed-off-by: Quentin Monnet <quentin@isovalent.com>
The commit_signature() function does not use the hash of the commit,
which typically differs between the kernel repo and the mirrored
version, but the subject for this commit. Fix the comment accordingly.
Signed-off-by: Quentin Monnet <quentin@isovalent.com>
Kernel docs from Documentation/bpf/libbpf go straight to docs/ under libbpf.
Also ignore libbpf-only parts of docs subdir.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
This adds documentation files, including ones for autogenerating API
documentation based on code comments in the source code that's pulled
in via the mirror.
Signed-off-by: Grant Seltzer <grantseltzer@gmail.com>
Let us sync linux/{pkt_cls.h,pkt_sched.h} to libbpf repo.
Otherwise, on ubuntu 16.04, system headers will be picked up
and this will result in compilation error like:
.../netlink.c:416:23: error: ‘TC_H_CLSACT’ undeclared (first use in this function)
*parent = TC_H_MAKE(TC_H_CLSACT,
^
.../netlink.c:418:9: error: ‘TC_H_MIN_INGRESS’ undeclared (first use in this function)
TC_H_MIN_INGRESS : TC_H_MIN_EGRESS);
^
.../netlink.c:418:28: error: ‘TC_H_MIN_EGRESS’ undeclared (first use in this function)
TC_H_MIN_INGRESS : TC_H_MIN_EGRESS);
^
.../netlink.c: In function ‘__get_tc_info’:
.../netlink.c:522:11: error: ‘TCA_BPF_ID’ undeclared (first use in this function)
if (!tbb[TCA_BPF_ID])
^
Signed-off-by: Yonghong Song <yhs@fb.com>
Libbpf doesn't rely on libc_compat.h anymore, so ignore it for the purposes of
syncing libbpf sources into Github.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Few recurring issues are fixed.
1. When there are patches in bpf tree that hasn't been synced yet, but bpf was
already merged into bpf-next, merged patches would be applied twice,
causing failures, requiring manual resolution. Now this is handled smarter
and shouldn't happen.
2. When synced libbpf repo contains fixes from bpf that weren't yet merged
into bpf-next, those bpf tree changes would cause inconsistency against
bpf-next tree state. That's expected and usually is pretty easy for human
to discard during consistency check, but is hard for automation. So instead
of failing at the very end, ask human whether discrepancies look good.
3. If sync script detected no new patches needed syncing, it previously didn't
restore linux repo state back. Fixed.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Allowing --3way leaves conflicts in the local files, which makes manual
conflict resolution so much easier.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Initial versions of sync script couldn't handle non-empty merges. But since
then, script became smarter, more interactive and thus more powerful and can
handle some complicated situations easily on its own, while falling back to
human intervention for even more complicated situations. This non-empty merge
check has outlived its purpose and is just an annoying bump in sync process.
Drop it.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Newer git started emitting warning about dangerousness of filter-branch.
Squelch it with FILTER_BRANCH_SQUELCH_WARNING=1 envvar.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Commit signature, used to determine already synced commits, includes a short
stats per each file relevant. Fix this script to include only files that are
actually synced (i.e., exclude Makefile, Build file, etc).
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Accomodate changes:
- test_libbpf.cpp was renamed to test_libbpf.c;
- bpf_helper_defs.h should be ignored for consistency check at the end,
as it's not checked in on linux side;
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Linux repo doesn't commit bpf_helper_defs.h, as it's re-generated on
build every time. For Github projection, though, it's much nicer to have
this header be pre-generated during sync and commited. This makes
integration story easier for all the users that use libbpf as
a submodule.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Fix how non-empty merge detection is done. Allow to proceed despite
non-empty merges (they will typically will cause conflicts during
applying patches, but if conflicts were handled already, it should be ok
to ignore this problem).
Also ensure that diff's output is in unified diff format.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Add patches sync from both bpf and bpf-next trees at the same time.
Baseline checkpoint commits are tracked independently for both.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Give more control over patching process to allow manual intervention and
fix up, after which process will continue. Also allow an option to
ignore consistency check results (when having both bpf and bpf-next
changes).
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
If cherry-picked commit contains non-libbpf files, chances are high that
this will result in conflict, because we are generally skipping commits
that didn't touch libbpf files, which means that our working copy will
not be up-to-date for non-libbpf files. This change checks if conflicts
are only in non-libbpf files and marks them as resolved. This will work
fine as long as we don't cherry-pick some more non-libbpf changes to
same set of files that happen to conflict with not-so-resolved version
of non-libbpf files. But anyways, this should help in a lot of cases.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
This is sometimes necessary when we did ad-hoc urgent bug fixes, which
are not identical to the ones in kernel.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Extract non-empty merge validation and cherry-picking logic so that it
can be re-used for bpf and bpf-next commits.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Use associative array (requires at least bash 4) to centralize mapping
of paths between kernel's libbpf layout and the one on Github. This
minimizes the chance of all those mappings getting out of sync (which
happened twice before).
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
1. There is one commit ("libbpf: fix btf_dedup equivalence check
handling of different kinds"), which was originally committed into bpf
tree and manually synced into libbpf (it was a critical bug fix). Now
that bpf and bpf-next trees were merged, this causes duplicate commits in
libbpf. To address this, check is added to see if libbpf repo already has
a commit with the same subject and allow to skip it.
2. Just checking by subject is not good enough, as we currently have few
different commits with the same subject "bpf: Sync bpf.h to tools/".
To address this, script now will ask user whether to skip or cherry-pick
commit. This allows for human to double-check the commit and make
informed devision.
3. There is a commit ('bfb35c27c65f bpf: fix whitespace for ENCAP_L2
defines in bpf.h'), which modifies both include/uapi/linux/bpf.h and
tools/include/uapi/linux/bpf.h in the same commit. This break the
sync script, as it doesn't cherry-pick commits that hasn't modified
libbpf files. So at the time that commit bfb35c27c65f is applied, linux
repo is missing some preceding changes to include/uapi/linux/bpf.h. They
would normally be ignored later during repo filtering, but to do
cherry-pick they are necessary. It's hard to fix it automatically, so
script will wait for user to resolve the issue and continue.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Merges are common. `git cherry-pick` can't cherry-pick merge commit and
requires specifying which of the parents should be cherry-picked, so
it's a pain. But merges are also commonly empty, so it's safe to skip them.
This change goes through all merges touching libbpf files, checks that
they don't have any conflict resolution diffs. If there are any, abort,
if not, proceed as usual, ignoring merges.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
With xsk.{o,h} changes we started including Linux UAPI's if_xdp.h
header. We need to sync it along the other UAPI headers.
Also updated README to reflect this.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Previous version of script relied on squashing baseline commit and
rebasing rest of commits on top of it. This doesn't work well with git
histories containing merges. This patch changes approach by cherry-picking
commits that have libbpf-related changes and then rewriting history since
last checkpoint.
This still might fail if there were manually resolved merge conflicts for
libbpf, but it's the best we can get as far as I can tell.
Script now also verifies that state of libbpf in Linux's repository exactly
matches the state of libbpf in github repo.
If everything goes smoothing (including verification step), we clean up after
ourselves and only leave libbpf-sync-XXX branch in github's libbpf repo to
push to remote.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
This script automates the process of applying libbpf-relevant changes
from kernel repository on top of current state of libbpf repository.
It uses CHECKPOINT-COMMIT file to keep track of last commit in kernel
repo up to which libbpf is in sync with. If there are any new libbpf
changes in kernel repository, script extracts them, preserving original
commit metadata. It also creates a "sync commit" using cover letter as
a template, which nicely summarizes changes since last sync with kernel.
Usage: ./scripts/sync-kernel.sh <linux-repo> <libbpf-repo>
If it succeeds, script will create a bunch of local commits in
<libbpf-repo> in separate branch, which can be easily pushed into github
to create a pull request.
Script tries to clean up after itself, except in case of failure. But it
doesn't clean up timestamped branches it creates in both kernel and
libbpf repositories for now. We can add that later.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
