WSL2-Ubuntu18.04编译运行BPF BCC工具集
经过一天半的折腾，总算把hello_world.py运行成功：
1. 主要步骤:
为了得到上面的输出，需要下述步骤。
1.1 更新WSL2内核配置
-
编译WSL2 Kernel，打开必要的BPF相关的开关，参考另外两篇：
当前打开的选项有：
root@Work:~# cat /boot/config-5.10.74.3-wsl2 |grep BPF
CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_LSM=y
CONFIG_BPF_SYSCALL=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_IPV6_SEG6_BPF=y
# CONFIG_NETFILTER_XT_MATCH_BPF is not set
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m
CONFIG_NET_CLS_BPF=y
CONFIG_NET_ACT_BPF=y
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
root@Work:~#
1.2 使用python3
1.3 使用clang-14
使用llvm官网的脚本进行自动安装, 通过update-alternatives设置默认的版本为14。
clang-12及之后的版本，编译后运行会失败
root@Work:~# hello_world.py
: CommandLine Error: Option 'openmp-ir-builder-optimistic-attributes' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
Aborted
编译时需要指定-DENABLE_LLVM_SHARED=1选项，详见：
https://github.com/iovisor/bpftrace/issues/1855
1.4 使用gcc-11
否则报告: undefined symbol: _ZSt28__throw_bad_array_new_lengthv错误，这个是由于libstdc++中没有该符号所致，需要整体升级到g++11，Ubuntu18.04只能安装gcc-11，没有g++11。。。
详见：
https://gcc.gnu.org/pipermail/gcc-help/2021-June.txt
这一步整体编译gcc-11即可。注意需要做符号链接，否则clang++找到的可能是旧版本gcc。
? bcc git:(local-test) sudo ln -sf /usr/local/gcc-11.2/lib/gcc/x86_64-linux-gnu/11.2.0/ /usr/lib/gcc/x86_64-linux-gnu/11.2.0
? bcc git:(local-test) clang++ -v
Ubuntu clang version 14.0.0-++20220223080742+2fe5bf57172c-1~exp1~20220223080752.73
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/10
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11.2.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/8
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11.2.0
Candidate multilib: .;@m64
Selected multilib: .;@m64
? bcc git:(local-test)
1.5. 挂载debugfs
尽管编译内核时打开debug选项，但是WSL2并没有自动挂载它。
root@Work:~# hello_world.py
Traceback (most recent call last):
File "/usr/share/bcc/examples/hello_world.py", line 12, in
BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; }').trace_print()
File "/usr/lib/python3.9/site-packages/bcc/__init__.py", line 1550, in trace_print
line = self.trace_readline(nonblocking=False)
File "/usr/lib/python3.9/site-packages/bcc/__init__.py", line 1526, in trace_readline
trace = self.trace_open(nonblocking)
File "/usr/lib/python3.9/site-packages/bcc/__init__.py", line 1477, in trace_open
self.tracefile = open("%s/trace_pipe" % TRACEFS, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/sys/kernel/debug/tracing/trace_pipe'
open(/sys/kernel/debug/tracing/kprobe_events): No such file or directory
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/bcc/__init__.py", line 1742, in cleanup
self.detach_kprobe_event(k)
File "/usr/lib/python3.9/site-packages/bcc/__init__.py", line 881, in detach_kprobe_event
self.detach_kprobe_event_by_fn(ev_name, fn_name)
File "/usr/lib/python3.9/site-packages/bcc/__init__.py", line 895, in detach_kprobe_event_by_fn
raise Exception("Failed to detach BPF from kprobe")
Exception: Failed to detach BPF from kprobe
执行下述命令进行挂载：
mount -t debugfs none /sys/kernel/debug/
1.6 修改setup.py脚本
使用Python3.9进行编译时，会报告不支持--install-layout deb的错误，导致无法编译python绑定。去掉即可。
diff --git a/src/python/CMakeLists.txt b/src/python/CMakeLists.txt
index f5c40b95..1737c912 100644
--- a/src/python/CMakeLists.txt
+++ b/src/python/CMakeLists.txt
@@ -5,9 +5,9 @@ if(NOT PYTHON_CMD)
set(PYTHON_CMD "python3")
endif()
-if(EXISTS "/etc/debian_version")
- set(PYTHON_FLAGS "${PYTHON_FLAGS} --install-layout deb")
-endif()
+#if(EXISTS "/etc/debian_version")
+# set(PYTHON_FLAGS "${PYTHON_FLAGS} --install-layout deb")
+#endif()
file(GLOB_RECURSE PYTHON_SOURCES RELATIVE ${CMAKE_CURRENT_SOURCE_DIR}
${CMAKE_CURRENT_SOURCE_DIR}/*.py)
2 编译脚本示例
在bcc目录下的编译脚本如下
#!/bin/bash
mkdir build
cd build
cmake -DENABLE_LLVM_SHARED=1 ../
sudo make install -j
cmake -DPYTHON_CMD=python3 ../
pushd src/python
make
sudo make install -j
popd