• Overview
• Embedded Linux
• Linux Kernel and Drivers Development
• Linux Advanced Debugging Tools
• Yocto Project
• Real-Time Linux
• QT for Embedded
• Training Costs

Linux Advanced Debugging Tools

This intensive course introduces you to kernel and userspace debugging techniques and frameworks. An ARMv8 machine and multiple QEMU emulated environments are used as a hardware platforms, on which variety of kernel interfaces, frameworks, and device drivers are explored, traced an analyzed.

The training consists of 70% practical exercises: debugging sample problems.

• Duration: 3 days (24h)

Course agenda

1. Embedded Linux - quick reference

• System architecture.
• BSP and other tools.
• Using Yocto Project based SDK.
• System boot: bootloader, Linux kernel, Device-Tree, userspace.
• Linux kernel source code: working with, configuration, cross-compilation, important settings.
• Cross-compilation, working with SDK, IDE and out-of tree kernel modules.

Exercises

Prepare environment, configure SDK and IDE, compile and load external modules. Observe system behavior. Use serial console and network connection. Get familiar with ARM core and SoC documentation.

To reduce initial setup time and speed-up training, some parts of 'quick reference' are delivered ahead. Participants have time to setup workstations and get familiar with environment and documentation. The package contains ready-to-use virtual machine, as well as a complete detailed instruction. Some additional exercises are designed to improve understanding of advanced training topics.

2. User space debugging

Various advanced debugging tools and techniques for working with userspace applications.

• GDB/gdbserver, debugging with GDB
• Remote debugging (Yocto Project based SDK, Eclipse IDE). Debugging userspace applications utilizing external libraries (QTbase as an example).
• Memory leaks in userspace applications.
• Race conditions and locking issues.
• Post-mortem debugging (core dumps). Debugging userspace segfaults.
• Valgrind - tracing memory, heap, calls.
• Tracing: ldd, strace (system call tracing), ltrace (library calls tracing)
• gcov - code coverage testing.

Exercises

Practical exercises are performed on RaspberryPi hardware running an ARMv8 (64bit) system.

BSP build by Yocto Project is used for remote debugging and tracing. Tasks involve debugging and tracing applications, from simple ones to multi-library, multi-threaded programs.

3. Kernel debugging

This is the main topic of the training.

• Linux Kernel - design and operations. Quick introduction to in-kernel programming and primitives. Overview of the kernel operations (user-mode, interrupt-mode),
  • system calls,
  • memory management and I/O access,
  • interrupts, deferred work,
  • timers,
  • locks and SMP synchronization,
  • block devices,
  • network devices,
  • cryptoAPI
• Debugging and probing techniques:
  • 'print' statements
  • kernel OOPs-es (critical errors) and core dumps
  • querying (procfs, sysfs, debugfs)
  • kprobes, kretprobes
• Using debugger:
  • kdb, kgdb (practical debugging of a network driver, extending kdb commands)
  • Remote kernel debugging - utilizing external gdb and Eclipse IDE to debug kernel.
  • On-chip debugging (J-Tag)
• Profiling methods. Profiling and measuring various parts of running kernel.
  • Linux performance observability tools
  • Linux power management and tools.
• Profiling Linux kernel using LTTng and Eclipse
• Use-cases - practical examples from our work. Debugging PCI, wifi, display, timer, "random hangs", and many more issues using various software techniques and hardware tools.

Exercises

• Various code examples to modify/experiment with (locking, timers, tracing etc.)
• Example network and block device drivers:
  • debug and resolve race conditions
  • debug and resolve memory leaks (memory management issues, using memory after freeing it, 'rouge' memory access)
  • trace functions and variables
  • debug and resolve interrupts (race conditions, context misuse, locking)
  • measure system performance, find driver bottlenecks
• Utilize x86 QEMU to emulate on-chip debugging (debug system boot, typical problems, etc.)
  • Use QEMU setup to investigate
• System tracing methods: perf, LTTng, eBPF

Requirements

Participants should be familiar with:

• C, as this is the language used in Linux kernel.
• shell commands and working in terminal environment.
• Linux device drivers design and organization.

Ideally: you should get our "Linux Device Drivers" training prior to this one.