Building and Installing perf for a Custom Linux Kernel on Ubuntu 22.04 LTS

In my previous blog, I demonstrated how to compile and install the Linux kernel 6.16.1 from upstream sources on Ubuntu 22.04 LTS. At this point, we have a working kernel named 6.16.1-Demo running on our system.

If you already had perf installed for a different kernel and attempted to use it with this new kernel, you may have encountered the following error:

WARNING: perf not found for kernel 6.16.1

  You may need to install the following packages for this specific kernel:
    linux-tools-6.16.1-Demo
    linux-cloud-tools-6.16.1-Demo

  You may also want to install one of the following packages to keep up to date:
    linux-tools-Demo
    linux-cloud-tools-Demo

This happens because perf is tightly coupled to the kernel it runs against. Each kernel version requires its own matching build of perf. For custom kernels, the best approach is to compile perf directly from the kernel’s source tree.

We also added the extra version string Demo to emphasize that this is a custom build. At the time of writing, the latest kernel available in Ubuntu 22.04 LTS repositories is 6.8.0, so relying on Ubuntu packages for kernel 6.16.1 is not possible.

In this tutorial, I’ll show you how to build perf from source for your custom kernel.

Step 1. Obtain the Kernel Source Code

Since we are working with a custom kernel, you’ll need access to its source code from the developer. The method will vary depending on where your kernel came from.

Step 2. Install Required Dependencies

Attempting to compile perf without dependencies will quickly result in errors. On my system, the first error was:

Makefile.config:204: *** Error: pkg-config needed by libtraceevent is missing on this system, please install it.  Stop.

This can be fixed with:

sudo apt update && sudo apt install pkg-config

But don’t celebrate yet—you’ll likely hit another error:

Makefile.config:910: *** ERROR: No python interpreter needed for jevents generation. Install python or build with NO_JEVENTS=1..  Stop.

On Ubuntu 22.04 LTS, resolve this by installing:

sudo apt install python-dev-is-python3

And, of course, another dependency issue may appear:

Makefile.config:1178: *** ERROR: libtraceevent is missing. Please install libtraceevent-dev/libtraceevent-devel and/or set LIBTRACEEVENT_DIR or build with NO_LIBTRACEEVENT=1.  Stop.

To resolve this error:

sudo apt install libtraceevent-dev

At this point, you should have the minimal dependencies needed to build a functional perf.

Step 3. Compile `perf` (Basic Build)

The perf utility lives inside the kernel source tree under tools/perf. It includes its own Makefile. Run:

cd linux-6.16.1/tools/perf
make

Note: You don’t need to specify -j; the Makefile automatically detects and uses the maximum number of available threads:

  BUILD:   Doing 'make -j40' parallel build

This will produce a working perf binary, but with limited functionality. During the build, you’ll see which features were enabled or disabled. For example:

Auto-detecting system features:
...                                   libdw: [ OFF ]
...                                   glibc: [ on  ]
...                                  libelf: [ on  ]
...                                 libnuma: [ OFF ]
...                  numa_num_possible_cpus: [ OFF ]
...                                 libperl: [ OFF ]
...                               libpython: [ on  ]
...                               libcrypto: [ on  ]
...                             libcapstone: [ OFF ]
...                               llvm-perf: [ OFF ]
...                                    zlib: [ on  ]
...                                    lzma: [ OFF ]
...                               get_cpuid: [ on  ]
...                                     bpf: [ on  ]
...                                  libaio: [ on  ]
...                                 libzstd: [ OFF ]

You can inspect the build options afterward with:

./perf version --build-option

Sample output:

perf version 6.16.1-Demo
                   aio: [ on  ]  # HAVE_AIO_SUPPORT
                   bpf: [ on  ]  # HAVE_LIBBPF_SUPPORT
         bpf_skeletons: [ OFF ]  # HAVE_BPF_SKEL
            debuginfod: [ OFF ]  # HAVE_DEBUGINFOD_SUPPORT
                 dwarf: [ OFF ]  # HAVE_LIBDW_SUPPORT
    dwarf_getlocations: [ OFF ]  # HAVE_LIBDW_SUPPORT
          dwarf-unwind: [ OFF ]  # HAVE_DWARF_UNWIND_SUPPORT
              auxtrace: [ on  ]  # HAVE_AUXTRACE_SUPPORT
                libbfd: [ OFF ]  # HAVE_LIBBFD_SUPPORT ( tip: Deprecated, license incompatibility, use BUILD_NONDISTRO=1 and install binutils-dev[el] )
           libcapstone: [ OFF ]  # HAVE_LIBCAPSTONE_SUPPORT
             libcrypto: [ on  ]  # HAVE_LIBCRYPTO_SUPPORT
    libdw-dwarf-unwind: [ OFF ]  # HAVE_LIBDW_SUPPORT
                libelf: [ on  ]  # HAVE_LIBELF_SUPPORT
               libnuma: [ OFF ]  # HAVE_LIBNUMA_SUPPORT
            libopencsd: [ OFF ]  # HAVE_CSTRACE_SUPPORT
               libperl: [ OFF ]  # HAVE_LIBPERL_SUPPORT
               libpfm4: [ OFF ]  # HAVE_LIBPFM
             libpython: [ on  ]  # HAVE_LIBPYTHON_SUPPORT
              libslang: [ OFF ]  # HAVE_SLANG_SUPPORT
         libtraceevent: [ on  ]  # HAVE_LIBTRACEEVENT
             libunwind: [ OFF ]  # HAVE_LIBUNWIND_SUPPORT ( tip: Deprecated, use LIBUNWIND=1 and install libunwind-dev[el] to build with it )
                  lzma: [ OFF ]  # HAVE_LZMA_SUPPORT
numa_num_possible_cpus: [ OFF ]  # HAVE_LIBNUMA_SUPPORT
                  zlib: [ on  ]  # HAVE_ZLIB_SUPPORT
                  zstd: [ OFF ]  # HAVE_ZSTD_SUPPORT

The way to enable these functionalities is also provided during build:

Makefile.config:468: No libdw.h found or old libdw.h found or elfutils is older than 0.157, disables dwarf support. Please install new elfutils-devel/libdw-dev
Makefile.config:564: No elfutils/debuginfod.h found, no debuginfo server support, please install libdebuginfod-dev/elfutils-debuginfod-client-devel or equivalent
Makefile.config:606: No sys/sdt.h found, no SDT events are defined, please install systemtap-sdt-devel or systemtap-sdt-dev
Makefile.config:694: Warning: Disabled BPF skeletons as clang (clang) is missing
Makefile.config:726: Disabling post unwind, no support found.
Makefile.config:789: slang not found, disables TUI support. Please install slang-devel, libslang-dev or libslang2-dev
Makefile.config:836: Missing perl devel files. Disabling perl scripting support, please install perl-ExtUtils-Embed/libperl-dev
Makefile.config:968: No libllvm 13+ found, slower source file resolution, please install llvm-devel/llvm-dev
Makefile.config:1001: No liblzma found, disables xz kernel module decompression, please install xz-devel/liblzma-dev
Makefile.config:1014: No libzstd found, disables trace compression, please install libzstd-dev[el] and/or set LIBZSTD_DIR
Makefile.config:1027: No numa.h found, disables 'perf bench numa mem' benchmark, please install numactl-devel/libnuma-devel/libnuma-dev
Makefile.config:1086: No libbabeltrace found, disables 'perf data' CTF format support, please install libbabeltrace-dev[el]/libbabeltrace-ctf-dev
Makefile.config:1098: No libcapstone found, disables disasm engine support for 'perf script', please install libcapstone-dev/capstone-devel
Makefile.config:1129: No alternatives command found, you need to set JDIR= to point to the root of your Java directory
Makefile.config:1160: libpfm4 not found, disables libpfm4 support. Please install libpfm4-dev
Makefile.config:1192: libtracefs is missing. Please install libtracefs-dev/libtracefs-devel

Note: Package whose name ends with -dev is for Debian-based distros, while -devel is for Red Hat-based distros. Choose the right package name based on your distro.

In the end, if you want to install documentations, you will need to install packages, asciidoc and xmlto.

Step 4. Install Additional Libraries (Full Build)

To build a fully functional perf, you’ll need extra development libraries:

sudo apt update && sudo apt install libdw-dev libdebuginfod-dev systemtap-sdt-dev libslang2-dev libperl-dev llvm-dev liblzma-dev libzstd-dev libnuma-dev libbabeltrace-ctf-dev libcapstone-dev libpfm4-dev libtracefs-dev asciidoc xmlto

This also installs the tools needed to generate man pages.

Step 5. Rebuild `perf` (Full Features)

Clean up the previous build and recompile:

make clean
make

Use ./perf version --build-option to check the supported features again. Sample output:

perf version 6.16.1-Demo
                   aio: [ on  ]  # HAVE_AIO_SUPPORT
                   bpf: [ on  ]  # HAVE_LIBBPF_SUPPORT
         bpf_skeletons: [ OFF ]  # HAVE_BPF_SKEL
            debuginfod: [ on  ]  # HAVE_DEBUGINFOD_SUPPORT
                 dwarf: [ on  ]  # HAVE_LIBDW_SUPPORT
    dwarf_getlocations: [ on  ]  # HAVE_LIBDW_SUPPORT
          dwarf-unwind: [ on  ]  # HAVE_DWARF_UNWIND_SUPPORT
              auxtrace: [ on  ]  # HAVE_AUXTRACE_SUPPORT
                libbfd: [ OFF ]  # HAVE_LIBBFD_SUPPORT ( tip: Deprecated, license incompatibility, use BUILD_NONDISTRO=1 and install binutils-dev[el] )
           libcapstone: [ on  ]  # HAVE_LIBCAPSTONE_SUPPORT
             libcrypto: [ on  ]  # HAVE_LIBCRYPTO_SUPPORT
    libdw-dwarf-unwind: [ on  ]  # HAVE_LIBDW_SUPPORT
                libelf: [ on  ]  # HAVE_LIBELF_SUPPORT
               libnuma: [ on  ]  # HAVE_LIBNUMA_SUPPORT
            libopencsd: [ OFF ]  # HAVE_CSTRACE_SUPPORT
               libperl: [ on  ]  # HAVE_LIBPERL_SUPPORT
               libpfm4: [ on  ]  # HAVE_LIBPFM
             libpython: [ on  ]  # HAVE_LIBPYTHON_SUPPORT
              libslang: [ on  ]  # HAVE_SLANG_SUPPORT
         libtraceevent: [ on  ]  # HAVE_LIBTRACEEVENT
             libunwind: [ OFF ]  # HAVE_LIBUNWIND_SUPPORT ( tip: Deprecated, use LIBUNWIND=1 and install libunwind-dev[el] to build with it )
                  lzma: [ on  ]  # HAVE_LZMA_SUPPORT
numa_num_possible_cpus: [ on  ]  # HAVE_LIBNUMA_SUPPORT
                  zlib: [ on  ]  # HAVE_ZLIB_SUPPORT
                  zstd: [ on  ]  # HAVE_ZSTD_SUPPORT

Now most features should be enabled, including support for libdw, libnuma, libzstd, and others. This build is generally sufficient for production use.

Step 6. Install perf into the System PATH

For everyday use, it’s best to install the compiled perf binary (and its man pages) into the system PATH, so you don’t need to invoke it with a relative or absolute path.

But here’s the catch: you may already have perf installed system-wide for another kernel. Ubuntu’s packaging system solves this by allowing multiple versions of perf to coexist. Let’s dig into how this works.

6.1 Inspecting the Wrapper Script

Let’s verify:

which perf

Sample output:

/usr/bin/perf

Looks like a binary file, right? Don't be fooled:

file $(which perf)

Sample output:

/usr/bin/perf: Bourne-Again shell script, ASCII text executable

So, /usr/bin/perf isn’t a binary—it’s a wrapper script. Let’s take a peek:

#!/bin/bash
full_version=`uname -r`

# First check for a fully qualified version.
this="/usr/lib/linux-tools/$full_version/`basename $0`"
if [ -f "$this" ]; then
	exec "$this" "$@"
fi
[...]

This script dynamically selects the correct perf binary based on your current kernel version (uname -r). That’s how Ubuntu achieves version-specific polymorphism.

6.2 How the Binaries Are Stored

Let’s see what binary it would have chosen for kernel 5.15.0-151-generic:

file /usr/lib/linux-tools/5.15.0-151-generic/perf

Sample output:

/usr/lib/linux-tools/5.15.0-151-generic/perf: symbolic link to ../../linux-tools-5.15.0-151/perf

Follow the symlink:

file /usr/lib/linux-tools-5.15.0-151/perf

Sample output:

/usr/lib/linux-tools-5.15.0-151/perf: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=<redacted>, for GNU/Linux 3.2.0, with debug_info, not stripped

So here’s the structure:

/usr/lib/linux-tools-<kernel release without flavor>/perf → actual binary
/usr/bin/perf → wrapper script
/usr/lib/linux-tools/$(uname -r)/perf → symlink to actual binary

6.3 Package Ownership

Who installs these files? Let’s check:

dpkg -L linux-tools-common

Sample output:

/.
/usr
/usr/bin
/usr/bin/acpidbg
/usr/bin/cpupower
/usr/bin/perf
/usr/bin/turbostat
/usr/bin/usbip
/usr/bin/usbipd
/usr/bin/x86_energy_perf_policy
/usr/sbin
/usr/sbin/bpftool
/usr/share
/usr/share/bash-completion
/usr/share/bash-completion/completions
/usr/share/bash-completion/completions/bpftool
/usr/share/doc
/usr/share/doc/linux-tools-common
/usr/share/doc/linux-tools-common/changelog.Debian.gz
/usr/share/doc/linux-tools-common/copyright
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/cpupower-frequency-info.1.gz
/usr/share/man/man1/cpupower-frequency-set.1.gz
/usr/share/man/man1/cpupower-idle-info.1.gz
/usr/share/man/man1/cpupower-idle-set.1.gz
/usr/share/man/man1/cpupower-info.1.gz
/usr/share/man/man1/cpupower-monitor.1.gz
/usr/share/man/man1/cpupower-set.1.gz
/usr/share/man/man1/cpupower.1.gz
/usr/share/man/man1/perf-annotate.1.gz
/usr/share/man/man1/perf-archive.1.gz
[...]

Among other things, it installs:

/usr/bin/perf (wrapper script)
Documentation and man pages

dpkg -L linux-tools-5.15.0-151-generic

Sample output:

/.
/usr
/usr/lib
/usr/lib/linux-tools
/usr/lib/linux-tools/5.15.0-151-generic
/usr/share
/usr/share/doc
/usr/share/doc/linux-tools-5.15.0-151-generic
/usr/share/doc/linux-tools-5.15.0-151-generic/copyright
/usr/lib/linux-tools/5.15.0-151-generic/acpidbg
/usr/lib/linux-tools/5.15.0-151-generic/bpftool
/usr/lib/linux-tools/5.15.0-151-generic/cpupower
/usr/lib/linux-tools/5.15.0-151-generic/libperf-jvmti.so
/usr/lib/linux-tools/5.15.0-151-generic/perf
/usr/lib/linux-tools/5.15.0-151-generic/turbostat
/usr/lib/linux-tools/5.15.0-151-generic/usbip
/usr/lib/linux-tools/5.15.0-151-generic/usbipd
/usr/lib/linux-tools/5.15.0-151-generic/x86_energy_perf_policy
/usr/share/doc/linux-tools-5.15.0-151-generic/changelog.Debian.gz

It provides:

/usr/lib/linux-tools/5.15.0-151-generic/perf (symlink)

dpkg -L linux-tools-5.15.0-151  # note: this is the dependency of the above

Sample output:

/.
/usr
/usr/lib
/usr/lib/linux-tools-5.15.0-151
/usr/lib/linux-tools-5.15.0-151/acpidbg
/usr/lib/linux-tools-5.15.0-151/bpftool
/usr/lib/linux-tools-5.15.0-151/cpupower
/usr/lib/linux-tools-5.15.0-151/libperf-jvmti.so
/usr/lib/linux-tools-5.15.0-151/perf
/usr/lib/linux-tools-5.15.0-151/turbostat
/usr/lib/linux-tools-5.15.0-151/usbip
/usr/lib/linux-tools-5.15.0-151/usbipd
/usr/lib/linux-tools-5.15.0-151/x86_energy_perf_policy
/usr/share
/usr/share/doc
/usr/share/doc/linux-tools-5.15.0-151
/usr/share/doc/linux-tools-5.15.0-151/changelog.Debian.gz
/usr/share/doc/linux-tools-5.15.0-151/copyright

This package contains:

/usr/lib/linux-tools-5.15.0-151/perf (actual binary)

So the division of responsibility is:

linux-tools-common → wrapper script and docs
linux-tools-<kernel>-generic → symlinks for a specific kernel flavor
linux-tools-<kernel> → actual binaries

6.4 Installing Your Custom `perf`

To integrate your custom perf into this system, mimic the package structure.

# install binaries and auxilary files to /usr/lib/linux-tools-$(uname -r)
PREFIX="/usr/lib/linux-tools"
sudo make prefix="$PREFIX"-"$(uname -r)" install install-doc

# add symlinks to /usr/lib/linux-tools/$(uname -r)
sudo mkdir -p "$PREFIX"/"$(uname -r)"
sudo ln -sr "$PREFIX"-"$(uname -r)"/bin/perf "$PREFIX"/"$(uname -r)"/perf

Now, when you run perf, the wrapper script automatically redirects to the correct binary for your current kernel, just like Ubuntu’s own packaging.

6.5 Reading the perf Man Pages

When installing perf manually into /usr/lib/linux-tools-$(uname -r), the man pages won’t automatically be picked up by the system’s default man path. To read them, you can explicitly point man to the custom location. For example, for our 6.16.1-Demo kernel:

man --manpath=/usr/lib/linux-tools-6.16.1-Demo/share/man perf

Note that the order of arguments is important: --manpath must come before the page name (perf). Otherwise, man will ignore the custom path and fallback to the system default.

Conclusion

By compiling and installing perf from the kernel source, you ensure full compatibility with your custom kernel—even when the version isn’t available in Ubuntu’s repositories. With the proper dependency setup, you can build a fully featured perf, integrate it cleanly into Ubuntu’s version-aware tool system, and even access its dedicated man pages. This approach gives you a reliable, maintainable workflow for performance profiling across custom kernels.