In commit 4794f18bf4 ("sync: Sync .mailmap entries"), we updated the
sync-up script to automatically update libbpf's .mailmap; however, the
script would not take care of committing the changes. Let's address
this.
The code is copied and adapted from the part where we commit changes to
src/bpf_helper_defs.h.
Signed-off-by: Quentin Monnet <qmo@kernel.org>
The kernel repository has a .mailmap file to remap author names and
email addresses to their desired format in Git logs (for details, see
gitmailmap documentation [0]). Alas, this is only visible for author
information when looking at the logs locally, as GitHub does not support
mailmaps at the moment [1].
This commit adds a .mailmap file for libbpf, automatically generated
from the kernel's version. The script to generate the .mailmap is added,
too: it works by grepping email addresses from authors in the
repository, and collecting all lines ending with this address in the
kernel's .mailmap - in other words, all lines where this address is used
as a pattern for a remapping.
To keep the .mailmap up-to-date, add a call to the script to
sync-kernel.sh.
[0] https://git-scm.com/docs/gitmailmap
[1] https://github.com/orgs/community/discussions/22518
Signed-off-by: Quentin Monnet <qmo@kernel.org>
The elfutils project has fixed several issues found by fuzz targets so it
should help to prevent the libbpf fuzz target from running into them.
Signed-off-by: Evgeny Vereshchagin <evvers@ya.ru>
When syncing with the kernel, the script generates a cover letter for
the latest changes using "git format-patch". Unless specified otherwise,
it uses a signature (as in, email footer signature) which defaults to
the Git version in use, and ends up in the commit logs. This doesn't
bring any useful information in there: let's get rid of this version
number.
Signed-off-by: Quentin Monnet <quentin@isovalent.com>
After recent lint changes, commit_signature() function now gets optional
array of paths as multiple arguments, instead of entire array as second
argument. So adjust commit_signature() to handle this correctly.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Now that we enforce Signed-off-by on every commit, make sure that
auto-generatd sync commits also get corrected Signed-off-by tags.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
to make it less likely for the libbpf fuzz target to run into
elfutils bugs that have been fixed upstream since two new fuzz
targets were added there back in April.
Signed-off-by: Evgeny Vereshchagin <evvers@ya.ru>
Libbpf relies on F_DUPFD_CLOEXEC constant coming from fcntl.h UAPI
header, so we need to sync it along other UAPI headers. Also update sync
script to keep doing this automatically going forward.
Reported-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Use correct Bash syntax to define these two variables as arrays.
Drop shellcheck opt-out for unquoted use of array.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Don't wrap LIBBPF_PATHS[@] and LIBBPF_VIEW_PATHS[@] in quotes when
passing it to git commands. Not clear how it worked before, but
something recently broke. Either git commands became stricter or
something.
But either way, we do want to pass each element of LIBBPF_PATHS or
LIBBPF_VIEW_PATHS as separate command line arguments, so putting them in
quotes doesn't make sense, as that makes them look like a single
argument to git.
So drop all the quotes around these arrays. The only place where it's
still needed is in commit_signature call, as we do want to pass array as
single arg ($2) and then internally we unfold it into multiple command
line arguments.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Without it coverage reports can't be built
```
[2022-01-31 00:05:36,094 DEBUG] Generating file view html index file as: "/out/report/linux/file_view_index.html".
Traceback (most recent call last):
File "/opt/code_coverage/coverage_utils.py", line 829, in <module>
sys.exit(Main())
File "/opt/code_coverage/coverage_utils.py", line 823, in Main
return _CmdPostProcess(args)
File "/opt/code_coverage/coverage_utils.py", line 780, in _CmdPostProcess
processor.PrepareHtmlReport()
File "/opt/code_coverage/coverage_utils.py", line 577, in PrepareHtmlReport
self.GenerateFileViewHtmlIndexFile(per_file_coverage_summary,
File "/opt/code_coverage/coverage_utils.py", line 450, in GenerateFileViewHtmlIndexFile
self.GetCoverageHtmlReportPathForFile(file_path),
File "/opt/code_coverage/coverage_utils.py", line 422, in GetCoverageHtmlReportPathForFile
assert os.path.isfile(
AssertionError: "/tmp/tmp.UYax4l19Gh/lib/system.h" is not a file.
```
It's a follow-up to 393a058d06
Signed-off-by: Evgeny Vereshchagin <evvers@ya.ru>
This header is necessary for libbpf-sys to generate perf-related Rust
bindings. It's more convenient to have it available locally with libbpf.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
It should make it easier to start using CFLite or something like that
to fuzz libbpf without getting pointless CVEs :-) More importantly,
now it's possible to build the fuzzer by just cloning the repository,
installing clang and running `./scripts/build-fuzzers.h`:
```
git clone https://github.com/libbpf/libbpf
./scripts/build-fuzzers.h
unzip -d CORPUS fuzz/bpf-object-fuzzer_seed_corpus.zip
./out/bpf-object-fuzzer CORPUS
```
It should make it easier (for me at least) to report some
elfutils bugs because they are much easier to reproduce manually
now.
git apply -3 doesn't always leave conflicted files in the working
directory. Use patch --merge instead, it seems to work better in more
complicated situations.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
`git am -3` will give up frequently even in cases when patch can be
auto-merged with:
```
Applying: libbpf: Unify low-level map creation APIs w/ new bpf_map_create()
error: sha1 information is lacking or useless (src/libbpf.c).
error: could not build fake ancestor
Patch failed at 0001 libbpf: Unify low-level map creation APIs w/ new bpf_map_create()
```
But `git apply -3` in the same situation will succeed with three-way merge just
fine:
```
error: patch failed: src/bpf_gen_internal.h:51
Falling back to three-way merge...
Applied patch to 'src/bpf_gen_internal.h' cleanly.
```
So if git am fails, try git apply and if that succeeds, automatically
`git am --continue`. If not, fallback to user actions.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
The sync-kernel.sh script has two consecutive tests for $BPF_BRANCH
being provided by the user (and so the second one can currently never
fail). Looking at the error message displayed in each case, we want to
keep the second one. Let's remove the first check.
Signed-off-by: Quentin Monnet <quentin@isovalent.com>
The commit_signature() function does not use the hash of the commit,
which typically differs between the kernel repo and the mirrored
version, but the subject for this commit. Fix the comment accordingly.
Signed-off-by: Quentin Monnet <quentin@isovalent.com>
Kernel docs from Documentation/bpf/libbpf go straight to docs/ under libbpf.
Also ignore libbpf-only parts of docs subdir.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
This adds documentation files, including ones for autogenerating API
documentation based on code comments in the source code that's pulled
in via the mirror.
Signed-off-by: Grant Seltzer <grantseltzer@gmail.com>
Let us sync linux/{pkt_cls.h,pkt_sched.h} to libbpf repo.
Otherwise, on ubuntu 16.04, system headers will be picked up
and this will result in compilation error like:
.../netlink.c:416:23: error: ‘TC_H_CLSACT’ undeclared (first use in this function)
*parent = TC_H_MAKE(TC_H_CLSACT,
^
.../netlink.c:418:9: error: ‘TC_H_MIN_INGRESS’ undeclared (first use in this function)
TC_H_MIN_INGRESS : TC_H_MIN_EGRESS);
^
.../netlink.c:418:28: error: ‘TC_H_MIN_EGRESS’ undeclared (first use in this function)
TC_H_MIN_INGRESS : TC_H_MIN_EGRESS);
^
.../netlink.c: In function ‘__get_tc_info’:
.../netlink.c:522:11: error: ‘TCA_BPF_ID’ undeclared (first use in this function)
if (!tbb[TCA_BPF_ID])
^
Signed-off-by: Yonghong Song <yhs@fb.com>
Libbpf doesn't rely on libc_compat.h anymore, so ignore it for the purposes of
syncing libbpf sources into Github.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Few recurring issues are fixed.
1. When there are patches in bpf tree that hasn't been synced yet, but bpf was
already merged into bpf-next, merged patches would be applied twice,
causing failures, requiring manual resolution. Now this is handled smarter
and shouldn't happen.
2. When synced libbpf repo contains fixes from bpf that weren't yet merged
into bpf-next, those bpf tree changes would cause inconsistency against
bpf-next tree state. That's expected and usually is pretty easy for human
to discard during consistency check, but is hard for automation. So instead
of failing at the very end, ask human whether discrepancies look good.
3. If sync script detected no new patches needed syncing, it previously didn't
restore linux repo state back. Fixed.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Allowing --3way leaves conflicts in the local files, which makes manual
conflict resolution so much easier.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Currently we hardcode "gcc", which means we get a bogus result any time
a non-default CC is passed to Make. In fact, it's bogus even when CC is
not explicitly set, since Make's default is "cc", which isn't
necessarily the same as "gcc".
Fix the issue by passing the compiler to use to check-reallocarray.sh.
Signed-off-by: Thomas Hebb <tommyhebb@gmail.com>
Initial versions of sync script couldn't handle non-empty merges. But since
then, script became smarter, more interactive and thus more powerful and can
handle some complicated situations easily on its own, while falling back to
human intervention for even more complicated situations. This non-empty merge
check has outlived its purpose and is just an annoying bump in sync process.
Drop it.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Newer git started emitting warning about dangerousness of filter-branch.
Squelch it with FILTER_BRANCH_SQUELCH_WARNING=1 envvar.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Commit signature, used to determine already synced commits, includes a short
stats per each file relevant. Fix this script to include only files that are
actually synced (i.e., exclude Makefile, Build file, etc).
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Accomodate changes:
- test_libbpf.cpp was renamed to test_libbpf.c;
- bpf_helper_defs.h should be ignored for consistency check at the end,
as it's not checked in on linux side;
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Linux repo doesn't commit bpf_helper_defs.h, as it's re-generated on
build every time. For Github projection, though, it's much nicer to have
this header be pre-generated during sync and commited. This makes
integration story easier for all the users that use libbpf as
a submodule.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Fix how non-empty merge detection is done. Allow to proceed despite
non-empty merges (they will typically will cause conflicts during
applying patches, but if conflicts were handled already, it should be ok
to ignore this problem).
Also ensure that diff's output is in unified diff format.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Add patches sync from both bpf and bpf-next trees at the same time.
Baseline checkpoint commits are tracked independently for both.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Give more control over patching process to allow manual intervention and
fix up, after which process will continue. Also allow an option to
ignore consistency check results (when having both bpf and bpf-next
changes).
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
If cherry-picked commit contains non-libbpf files, chances are high that
this will result in conflict, because we are generally skipping commits
that didn't touch libbpf files, which means that our working copy will
not be up-to-date for non-libbpf files. This change checks if conflicts
are only in non-libbpf files and marks them as resolved. This will work
fine as long as we don't cherry-pick some more non-libbpf changes to
same set of files that happen to conflict with not-so-resolved version
of non-libbpf files. But anyways, this should help in a lot of cases.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
This is sometimes necessary when we did ad-hoc urgent bug fixes, which
are not identical to the ones in kernel.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Extract non-empty merge validation and cherry-picking logic so that it
can be re-used for bpf and bpf-next commits.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
