Compile and Install a Real-Time Kernel¶

Introduction¶

Many human-robot interaction applications require a real-time operating system (RTOS). Real-time systems process data and events with critically defined time constraints. The difference between an OS such as Windows or Unix and an RTOS is the response time to external events: an OS typically provides a non-deterministic "soft" real-time response, with no guarantees as to when each task will complete but an attempt to stay responsive; an RTOS typically provides a "hard" real-time response, a fast, highly deterministic reaction to external events. Soft real-time systems operate within a few hundred milliseconds, at the scale of human reaction; hard real-time systems provide responses predictable within tens of milliseconds or less.

When switching between tasks, the RTOS chooses the most appropriate task to load next, using scheduling algorithms such as round-robin, cooperative, or hybrid scheduling. Most RTOSs use a pre-emptive scheduling algorithm for responsiveness, where each task is given an individual priority value (the faster the required response, the higher the priority), and the highest-priority task able to execute is chosen, resulting in a highly responsive system.

Linux PREEMPT-RT Kernel¶

The standard Linux kernel included with most Linux distributions, including Ubuntu, is a soft real-time kernel (check your kernel version with uname -r; it will look something like 5.11.0-38-generic). In the meantime, we compile a Linux kernel with a patch applied that changes some code (task scheduling and memory handling) within the kernel to make it preemptable; this patch is called the PREEMPT-RT patch.

Instructions for compiling and deploying a PREEMPT-RT kernel involve downloading a kernel version source code tarball along with the corresponding patch, unzipping, patching the kernel, compiling it to create an installation package, and deploying it on your machine. These instructions (borrowed from the Franka Emika FCI tutorial, since an RTOS is needed for running FCI, with some edits to help with errors) assume Ubuntu 22.04 LTS (Jammy Jellyfish). Note that NVIDIA binary drivers are not supported on PREEMPT_RT kernels.

Setting Up the Real-Time Kernel¶

Install the necessary dependencies:

sudo apt install build-essential bc curl ca-certificates gnupg2 libssl-dev lsb-release libelf-dev bison flex dwarves zstd libncurses-dev

Decide which kernel version to use (check your current version with uname -r); real-time patches are only available for select kernel versions (see https://www.kernel.org/pub/linux/kernel/projects/rt/). Choose the version closest to your current one. For Ubuntu 22.04 tested with kernel version 6.2:

curl -SLO https://www.kernel.org/pub/linux/kernel/v6.x/linux-6.2.tar.xz
curl -SLO https://www.kernel.org/pub/linux/kernel/v6.x/linux-6.2.tar.sign
curl -SLO https://www.kernel.org/pub/linux/kernel/projects/rt/6.2/patch-6.2-rt3.patch.xz
curl -SLO https://www.kernel.org/pub/linux/kernel/projects/rt/6.2/patch-6.2-rt3.patch.sign

Decompress:

xz -d *.xz

Verifying File Integrity (Optional but Recommended)¶

The .sign files can verify the downloaded files were not corrupted or tampered with (adapted from the Linux Kernel Archive):

gpg2 --verify linux-*.tar.sign
gpg2 --verify patch-*.patch.sign

If the output shows "Can't check signature: No public key", download the signer's public key first, using the key ID shown in the output:

gpg2 --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 647F28654894E3BD457199BE38DBBDC86092693E
gpg2 --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 57892E705233051337F6FDD105641F175712FA5B

Note that keys for other kernel versions may have different IDs; adapt accordingly. A "Good signature" result with a "WARNING: This key is not certified with a trusted signature!" note is expected and does not need attention (see the Linux Kernel Archive for more information).

Compiling the Kernel¶

Extract the source and apply the patch:

tar xf linux-*.tar
cd linux-*/
patch -p1 < ../patch-*.patch

Copy your currently booted kernel configuration as the default config for the new kernel:

cp -v /boot/config-$(uname -r) .config

Configure the build:

make olddefconfig
make menuconfig

In the configuration interface:

Navigate to General Setup > Preemption Model and select Fully Preemptible Kernel (Real-Time).
Navigate to Cryptographic API > Certificates for signature checking > Provide system-wide ring of trusted keys > Additional X.509 keys for default system keyring; remove "debian/canonical-certs.pem" from the prompt and press OK.
Navigate to Cryptographic API > Certificates for signature checking > Provide system-wide ring of revocation certificates > X.509 certificates to be preloaded into the system blacklist keyring; remove "debian/canonical-revoked-certs.pem" from the prompt and press OK.
Save the configuration and exit.

Compile the kernel (set -j to the number of CPU cores, which nproc will populate automatically):

make -j$(nproc) deb-pkg

Install the resulting packages (headers and images, without the dbg suffix):

sudo dpkg -i ../linux-headers-*.deb ../linux-image-*.deb

Verifying the New Kernel¶

Restart your system; the GRUB boot menu should now offer the newly installed kernel. Check which kernel is running with uname -a; it should contain the string PREEMPT RT and the version number you chose. Additionally, /sys/kernel/realtime should exist and contain the number 1.

Allow a User to Set Real-Time Permissions for Its Processes¶

After the PREEMPT_RT kernel is installed and running, add a group named realtime and add the user controlling your robot to it:

sudo addgroup realtime
sudo usermod -a -G realtime $(whoami)

Add the following limits to the realtime group in /etc/security/limits.conf:

@realtime soft rtprio 99
@realtime soft priority 99
@realtime soft memlock 102400
@realtime hard rtprio 99
@realtime hard priority 99
@realtime hard memlock 10240

The limits apply after you log out and in again. You now have an RTOS on your machine (sort of); some tuning is needed to establish how good a real-time system you actually have.

Tuning¶

A PREEMPT-RT kernel isn't exactly a hard-realtime operating system; it's about 95% of the way there, with no hard latency guarantees. You have to establish these latency guarantees yourself through tuning and benchmarking.

CPU Scaling¶

Disable CPU scaling by setting the CPU governor to "performance" using cpufrequtils:

sudo apt install cpufrequtils

Check available governors with cpufreq-info. For real-time applications, we generally want to run the "performance" governor (fast, at the expense of higher energy consumption) on every core:

sudo systemctl enable cpufrequtils
sudo sh -c 'echo "GOVERNOR=performance" > /etc/default/cpufrequtils'
sudo systemctl daemon-reload && sudo systemctl restart cpufrequtils

Re-check with cpufreq-info; the policy for each core should now read "performance."

Enable Performance Mode¶

Your system's UEFI/BIOS usually has an energy profile setting that adjusts available CPU power states; set this to "maximum performance" or equivalent.

Disable Hyperthreading¶

Hyper-threading (HT) / Simultaneous multithreading (SMT) maximizes processor resource usage for low-IPC workloads, but increases contention on processor resources, so it's recommended to turn it off to reduce jitter (this also doubles the effective L1/L2 cache available to a thread, for 2-way SMT). Methods to disable it, in recommended order:

In your system's UEFI/BIOS firmware settings (recommended).
At boot time using the kernel command line parameter nosmt.
At runtime: echo off > /sys/devices/system/cpu/smt/control.
Using CPU hot-plugging to disable one of a pair of sibling threads (identify siblings with lscpu --extended or cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list).

Verify SMT/HT is disabled; the output of cat /sys/devices/system/cpu/smt/active should return 0.

Enable Turbo Boost¶

Intel Turbo Boost / AMD Turbo Core let the processor automatically overclock within a power and thermal envelope. With good cooling (fan speed set to max in BIOS) and unused cores disabled (in BIOS or via CPU hotplug), application cores can run continuously at the higher boost frequency. Check whether Turbo Boost is enabled with cat /sys/devices/system/cpu/intel_pstate/no_turbo (output of 0 means enabled); if not, enable it in the system's UEFI/BIOS.

Disable RT Throttling¶

On systems predating widespread multicore availability, an RT process using up all available CPU time could hang the entire system, since the Linux scheduler would not run a non-RT process while an RT process continuously hogs the CPU. To prevent this kind of lockup (especially on desktop-oriented systems where any process can request RT), the Linux kernel throttles RT processes that use 0.95s out of every 1s of CPU time, pausing the process for the last 0.05s (which may cause deadline misses during those pauses). This can be turned off by writing -1 to /proc/sys/kernel/sched_rt_runtime_us on every system boot.

Benchmarking¶

Benchmarking uses cyclictest, the typical benchmark for detecting hardware and scheduling latency. It roughly implements:

while (true) {
  t1 = now();
  sleep(interval);
  t2 = now();
  latency = t2 - t1 - interval;
  log(latency);
}

If hardware + scheduling latency is zero, t2 - t1 == interval; if greater, the Linux kernel or hardware took up additional CPU time, and the latency is t2 - t1 - interval. Depending on command-line flags, the latency distribution is logged as minimum/average/maximum values or a histogram. cyclictest typically runs as the only userspace RT process on the system, so the kernel schedules and switches to it as soon as possible, ahead of all other normal processes; if configured correctly, it measures the "best-case" hardware + scheduling latency for a given hardware + OS combination. It should be run simultaneously with a stress test of CPU, memory, storage, and network, since an idle system is unlikely to encounter the conditions that trigger significant latency; running it for a long enough period gives a sense of worst-case latency.

Install the required tools:

sudo apt install rt-tests

A simple example that runs forever (use man cyclictest to understand the flags):

sudo cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0

cyclictest spins up a number of threads (equal to the number of processors) and reports minimum, average, maximum, and current latencies. The maximum latency (in microseconds by default) is the crucial number; for our robotics work, we ideally want maximum latency below 1000 (1 ms), since we tend to run 1000 Hz loops.

Run cyclictest under heavy CPU load using stress-ng:

sudo apt install stress-ng
stress-ng --cpu $(nproc) --io 8 --vm 8 --vm-bytes 8M --fork 4 --udp 4 --timeout 10h

A more complete script, run as root, runs cyclictest and produces a histogram afterward (requires gnuplot); source: https://www.osadl.org/Create-a-latency-plot-from-cyclictest-hi.bash-script-for-latency-plot.0.html. Run cyclictest for at least a few hours, up to a few days (a weekend works well), to establish max-latency guarantees over a large duration of time.

Summary¶

These are the basic steps to minimize latency on a PREEMPT-RT system. If latency still exceeds what's desired (for real-time commands on both the KUKA LBR and Franka Emika Panda robots, latency needs to stay below 1 microsecond since the control loop using FRI/FCI runs at 1000 Hz), other techniques are detailed at https://rigtorp.se/low-latency-guide/.

Troubleshooting¶

Display (NVIDIA)¶

If your display goes wonky (low resolution, second monitors blank) after installing the PREEMPT-RT kernel, this may be because you have an NVIDIA graphics card, which generally doesn't play nicely with PREEMPT-RT kernels. To use an NVIDIA card with your new kernel:

Download a driver (.run file) from https://www.nvidia.com/Download/index.aspx, entering the correct Linux x64 and graphics card model details.
Install from the command line:

sudo IGNORE_PREEMPT_RT_PRESENCE=1 bash <*>.run  # Insert downloaded .run file

Follow the prompts to build and install the driver, then reboot.