linux/tools/testing/selftests/bpf
Jakub Kicinski 07cbce2e46 Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next
Daniel Borkmann says:

====================
pull-request: bpf-next 2020-11-14

1) Add BTF generation for kernel modules and extend BTF infra in kernel
   e.g. support for split BTF loading and validation, from Andrii Nakryiko.

2) Support for pointers beyond pkt_end to recognize LLVM generated patterns
   on inlined branch conditions, from Alexei Starovoitov.

3) Implements bpf_local_storage for task_struct for BPF LSM, from KP Singh.

4) Enable FENTRY/FEXIT/RAW_TP tracing program to use the bpf_sk_storage
   infra, from Martin KaFai Lau.

5) Add XDP bulk APIs that introduce a defer/flush mechanism to optimize the
   XDP_REDIRECT path, from Lorenzo Bianconi.

6) Fix a potential (although rather theoretical) deadlock of hashtab in NMI
   context, from Song Liu.

7) Fixes for cross and out-of-tree build of bpftool and runqslower allowing build
   for different target archs on same source tree, from Jean-Philippe Brucker.

8) Fix error path in htab_map_alloc() triggered from syzbot, from Eric Dumazet.

9) Move functionality from test_tcpbpf_user into the test_progs framework so it
   can run in BPF CI, from Alexander Duyck.

10) Lift hashtab key_size limit to be larger than MAX_BPF_STACK, from Florian Lehner.

Note that for the fix from Song we have seen a sparse report on context
imbalance which requires changes in sparse itself for proper annotation
detection where this is currently being discussed on linux-sparse among
developers [0]. Once we have more clarification/guidance after their fix,
Song will follow-up.

  [0] https://lore.kernel.org/linux-sparse/CAHk-=wh4bx8A8dHnX612MsDO13st6uzAz1mJ1PaHHVevJx_ZCw@mail.gmail.com/T/
      https://lore.kernel.org/linux-sparse/20201109221345.uklbp3lzgq6g42zb@ltop.local/T/

* git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (66 commits)
  net: mlx5: Add xdp tx return bulking support
  net: mvpp2: Add xdp tx return bulking support
  net: mvneta: Add xdp tx return bulking support
  net: page_pool: Add bulk support for ptr_ring
  net: xdp: Introduce bulking for xdp tx return path
  bpf: Expose bpf_d_path helper to sleepable LSM hooks
  bpf: Augment the set of sleepable LSM hooks
  bpf: selftest: Use bpf_sk_storage in FENTRY/FEXIT/RAW_TP
  bpf: Allow using bpf_sk_storage in FENTRY/FEXIT/RAW_TP
  bpf: Rename some functions in bpf_sk_storage
  bpf: Folding omem_charge() into sk_storage_charge()
  selftests/bpf: Add asm tests for pkt vs pkt_end comparison.
  selftests/bpf: Add skb_pkt_end test
  bpf: Support for pointers beyond pkt_end.
  tools/bpf: Always run the *-clean recipes
  tools/bpf: Add bootstrap/ to .gitignore
  bpf: Fix NULL dereference in bpf_task_storage
  tools/bpftool: Fix build slowdown
  tools/runqslower: Build bpftool using HOSTCC
  tools/runqslower: Enable out-of-tree build
  ...
====================

Link: https://lore.kernel.org/r/20201114020819.29584-1-daniel@iogearbox.net
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
2020-11-14 09:13:41 -08:00
..
benchs selftests: Remove fmod_ret from test_overhead 2020-09-28 17:20:28 -07:00
gnu
map_tests
prog_tests Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next 2020-11-14 09:13:41 -08:00
progs Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next 2020-11-14 09:13:41 -08:00
verifier selftests/bpf: Add asm tests for pkt vs pkt_end comparison. 2020-11-13 01:42:11 +01:00
.gitignore selftests/bpf: Move test_tcppbf_user into test_progs 2020-11-03 17:52:37 -08:00
bench.c selftests: Remove fmod_ret from test_overhead 2020-09-28 17:20:28 -07:00
bench.h
bpf_legacy.h selftests: bpf: Remove unused bpf_map_def_legacy struct 2020-07-08 01:33:14 +02:00
bpf_rand.h
bpf_rlimit.h
bpf_tcp_helpers.h bpf: selftest: Add test_btf_skc_cls_ingress 2020-09-25 13:58:02 -07:00
bpf_util.h
btf_helpers.c selftests/bpf: Add split BTF dedup selftests 2020-11-05 18:37:31 -08:00
btf_helpers.h selftests/bpf: Add split BTF dedup selftests 2020-11-05 18:37:31 -08:00
cgroup_helpers.c bpf, selftests: Use single cgroup helpers for both test_sockmap/progs 2020-08-01 20:20:59 -07:00
cgroup_helpers.h bpf, selftests: Use single cgroup helpers for both test_sockmap/progs 2020-08-01 20:20:59 -07:00
config
flow_dissector_load.c
flow_dissector_load.h selftests/bpf: Don't use deprecated libbpf APIs 2020-09-03 17:14:40 -07:00
get_cgroup_id_user.c bpf, selftests: Use single cgroup helpers for both test_sockmap/progs 2020-08-01 20:20:59 -07:00
Makefile selftests/bpf: Add checking of raw type dump in BTF writer APIs selftests 2020-11-05 18:37:30 -08:00
netcnt_common.h
network_helpers.c bpf: selftests: Add fastopen_connect to network_helpers 2020-08-24 14:35:00 -07:00
network_helpers.h bpf: selftests: Add fastopen_connect to network_helpers 2020-08-24 14:35:00 -07:00
README.rst selftests/bpf: Add profiler test 2020-10-09 22:03:06 +02:00
settings selftests: bpf: Switch off timeout 2020-08-06 16:57:05 -07:00
test_bpftool.py
test_bpftool.sh
test_bpftool_build.sh selftests, bpftool: Add bpftool (and eBPF helpers) documentation build 2020-09-10 11:02:45 -07:00
test_bpftool_metadata.sh selftests/bpf: Test load and dump metadata with btftool and skel 2020-09-15 18:28:27 -07:00
test_btf.h
test_cgroup_storage.c bpf, selftests: Use single cgroup helpers for both test_sockmap/progs 2020-08-01 20:20:59 -07:00
test_cpp.cpp
test_current_pid_tgid_new_ns.c selftests/bpf: Fix two minor compilation warnings reported by GCC 4.9 2020-08-20 16:45:09 +02:00
test_dev_cgroup.c bpf, selftests: Use single cgroup helpers for both test_sockmap/progs 2020-08-01 20:20:59 -07:00
test_flow_dissector.c
test_flow_dissector.sh
test_ftrace.sh
test_iptunnel_common.h
test_kmod.sh selftests: bpf: test_kmod.sh: Fix running out of srctree 2020-07-21 13:26:24 -07:00
test_lirc_mode2.sh
test_lirc_mode2_user.c
test_lpm_map.c
test_lru_map.c
test_lwt_ip_encap.sh
test_lwt_seg6local.sh selftests/bpf: Fix test_lwt_seg6local.sh hangs 2020-07-21 13:26:26 -07:00
test_maps.c bpf: Lift hashtab key_size limit 2020-11-05 20:04:46 -08:00
test_maps.h
test_netcnt.c bpf, selftests: Use single cgroup helpers for both test_sockmap/progs 2020-08-01 20:20:59 -07:00
test_offload.py selftests/bpf: fix netdevsim trap_flow_action_cookie read 2020-07-30 16:33:07 -07:00
test_progs.c selftests/bpf: Fix test_progs-flavor run getting number of tests 2020-08-24 21:09:24 -07:00
test_progs.h selftests/bpf: Add split BTF basic test 2020-11-05 18:37:30 -08:00
test_select_reuseport_common.h
test_skb_cgroup_id.sh
test_skb_cgroup_id_user.c bpf, selftests: Use single cgroup helpers for both test_sockmap/progs 2020-08-01 20:20:59 -07:00
test_sock.c bpf, selftests: Use single cgroup helpers for both test_sockmap/progs 2020-08-01 20:20:59 -07:00
test_sock_addr.c bpf, selftests: Use single cgroup helpers for both test_sockmap/progs 2020-08-01 20:20:59 -07:00
test_sock_addr.sh
test_socket_cookie.c selftests/bpf: Don't use deprecated libbpf APIs 2020-09-03 17:14:40 -07:00
test_sockmap.c bpf, selftests: Add three new sockmap tests for verdict only programs 2020-10-11 18:09:44 -07:00
test_stub.c
test_sysctl.c bpf, selftests: Use single cgroup helpers for both test_sockmap/progs 2020-08-01 20:20:59 -07:00
test_tag.c
test_tc_edt.sh
test_tc_redirect.sh bpf, selftests: Extend test_tc_redirect to use modified bpf_redirect_neigh() 2020-10-22 01:30:00 +02:00
test_tc_tunnel.sh
test_tcp_check_syncookie.sh
test_tcp_check_syncookie_user.c
test_tcp_hdr_options.h bpf: selftest: Ensure the child sk inherited all bpf_sock_ops_cb_flags 2020-10-02 11:34:48 -07:00
test_tcpbpf.h selftest/bpf: Use global variables instead of maps for test_tcpbpf_kern 2020-11-03 17:52:37 -08:00
test_tcpnotify.h
test_tcpnotify_user.c bpf: Fix compilation warning of selftests 2020-08-06 16:58:42 -07:00
test_tunnel.sh selftest/bpf: Add missed ip6ip6 test back 2020-11-10 15:37:45 -08:00
test_verifier.c selftests/bpf: Add verifier test for d_path helper 2020-08-25 15:41:15 -07:00
test_verifier_log.c
test_xdp_meta.sh
test_xdp_redirect.sh selftests/bpf: Add xdpdrv mode for test_xdp_redirect 2020-07-31 00:43:49 +02:00
test_xdp_veth.sh
test_xdp_vlan.sh
test_xdp_vlan_mode_generic.sh
test_xdp_vlan_mode_native.sh
test_xdping.sh
testing_helpers.c selftests/bpf: Add link detach tests for cgroup, netns, and xdp bpf_links 2020-08-01 20:38:28 -07:00
testing_helpers.h selftests/bpf: Add link detach tests for cgroup, netns, and xdp bpf_links 2020-08-01 20:38:28 -07:00
trace_helpers.c selftests/bpf: Ksyms_btf to test typed ksyms 2020-10-02 14:59:25 -07:00
trace_helpers.h selftests/bpf: Ksyms_btf to test typed ksyms 2020-10-02 14:59:25 -07:00
urandom_read.c
with_addr.sh
with_tunnels.sh
xdping.c
xdping.h

==================
BPF Selftest Notes
==================
General instructions on running selftests can be found in
`Documentation/bpf/bpf_devel_QA.rst`_.

Additional information about selftest failures are
documented here.

profiler[23] test failures with clang/llvm <12.0.0
==================================================

With clang/llvm <12.0.0, the profiler[23] test may fail.
The symptom looks like

.. code-block:: c

  // r9 is a pointer to map_value
  // r7 is a scalar
  17:       bf 96 00 00 00 00 00 00 r6 = r9
  18:       0f 76 00 00 00 00 00 00 r6 += r7
  math between map_value pointer and register with unbounded min value is not allowed

  // the instructions below will not be seen in the verifier log
  19:       a5 07 01 00 01 01 00 00 if r7 < 257 goto +1
  20:       bf 96 00 00 00 00 00 00 r6 = r9
  // r6 is used here

The verifier will reject such code with above error.
At insn 18 the r7 is indeed unbounded. The later insn 19 checks the bounds and
the insn 20 undoes map_value addition. It is currently impossible for the
verifier to understand such speculative pointer arithmetic.
Hence
    https://reviews.llvm.org/D85570
addresses it on the compiler side. It was committed on llvm 12.

The corresponding C code
.. code-block:: c

  for (int i = 0; i < MAX_CGROUPS_PATH_DEPTH; i++) {
          filepart_length = bpf_probe_read_str(payload, ...);
          if (filepart_length <= MAX_PATH) {
                  barrier_var(filepart_length); // workaround
                  payload += filepart_length;
          }
  }

bpf_iter test failures with clang/llvm 10.0.0
=============================================

With clang/llvm 10.0.0, the following two bpf_iter tests failed:
  * ``bpf_iter/ipv6_route``
  * ``bpf_iter/netlink``

The symptom for ``bpf_iter/ipv6_route`` looks like

.. code-block:: c

  2: (79) r8 = *(u64 *)(r1 +8)
  ...
  14: (bf) r2 = r8
  15: (0f) r2 += r1
  ; BPF_SEQ_PRINTF(seq, "%pi6 %02x ", &rt->fib6_dst.addr, rt->fib6_dst.plen);
  16: (7b) *(u64 *)(r8 +64) = r2
  only read is supported

The symptom for ``bpf_iter/netlink`` looks like

.. code-block:: c

  ; struct netlink_sock *nlk = ctx->sk;
  2: (79) r7 = *(u64 *)(r1 +8)
  ...
  15: (bf) r2 = r7
  16: (0f) r2 += r1
  ; BPF_SEQ_PRINTF(seq, "%pK %-3d ", s, s->sk_protocol);
  17: (7b) *(u64 *)(r7 +0) = r2
  only read is supported

This is due to a llvm BPF backend bug. The fix 
  https://reviews.llvm.org/D78466
has been pushed to llvm 10.x release branch and will be
available in 10.0.1. The fix is available in llvm 11.0.0 trunk.

BPF CO-RE-based tests and Clang version
=======================================

A set of selftests use BPF target-specific built-ins, which might require
bleeding-edge Clang versions (Clang 12 nightly at this time).

Few sub-tests of core_reloc test suit (part of test_progs test runner) require
the following built-ins, listed with corresponding Clang diffs introducing
them to Clang/LLVM. These sub-tests are going to be skipped if Clang is too
old to support them, they shouldn't cause build failures or runtime test
failures:

  - __builtin_btf_type_id() ([0], [1], [2]);
  - __builtin_preserve_type_info(), __builtin_preserve_enum_value() ([3], [4]).

  [0] https://reviews.llvm.org/D74572
  [1] https://reviews.llvm.org/D74668
  [2] https://reviews.llvm.org/D85174
  [3] https://reviews.llvm.org/D83878
  [4] https://reviews.llvm.org/D83242