Real-Time Linux in Industrial Appliances
Booming usage of Linux in embedded environments is creating an on-going demand
to use this general-purpose operating system in industrial control devices and
other fields, previously reserved for classic real-time operating systems. The
course covers key topics, essential to understand the opportunities in using
Linux as a real-time operating system and its fails. You will learn what
Real-Time really means and how to achieve it in Linux.
Exercises done on ARM based development board with example of Real-Time
application (controlling real industrial process), which will be developed
during the course, are to give you a strong technical understanding and
confidence in the subject.
Emphasis is put on developing POSIX-RT based real-time applications using some
modern Linux variants, in order to work out a universal and re-usable framework,
for your future Real-Time Linux projects. In order to learn programming
techniques, the software is verified with partial tests and latency measurements
are done during the exercises.
- Duration
- 4-days (32h)
Course outline
- Introduction to Real-Time, a world about software and hardware design
- terminology (latency, jitter, worst case, safety critical)
- where the latency comes from
- Linux and Real-Time
- RT-PREEMPT – introduction, implementation, usage and tuning, measuring in
real use cases
- Xenomai – utilizing co-kernel design in applications, introduction,
implementation, optimizing and tuning, real use cases, RTDM
(Real-Time-Device-Model), Adeos/I-pipe
- RTAI (LinuxCNC) – alternative co-kernel approach
- Safety-Critical environments (watchdog, memory error recovery).
- POSIX-RT api
- threads
- advanced real-time IPC (signals, shared memory, locking, message queues and
mailboxes).
- AIO – main use of non-real time drivers and peripherals
- Real-Time networking
- RTnet, RTmac
- distributed real-time processing
- best practices of Real-Time Linux
- Talking to the world
- input/output interfaces: Serial, GPIO, SPI, I2C
- analog input
- Industrial I/O
Detailed agenda
Day. 1, Linux and Real-Time – the introduction
- Refreshing the information about Embedded Linux
- Linux as a general-purpose operating system.
- Introduction to Real-Time, a world about software and hardware design
- terminology (latency, jitter, worst case, safety critical)
- where the latency comes from
- latency-related aspects of kernel configuration
- interrupts, delayed interrupt handling
- kernel preemption (voluntary and forced preemption)
- utilizing High Resolution Timers
- Measuring Embedded Linux latency
- external methods (oscilloscope, logic analyzer)
- internal timers, and latency buffer
- kernel profiling
- Tracers (ftrace), function tracing and histograms
- generating load and emulating real usage
- Linux and Real-Time
- Overview of Real-Time solutions in Linux (RT-PREEMPT, co-kernel)
After ‘Day 1’:
The applicable terminology will be defined and problems introduced. Participants will
review their knowledge of Linux Kernel and userspace. Latency and other
problematic variables will be measured. Eventually the general solution will be
introduced (Real-Time patches or co-kernel).
Day 2. How to make Linux Real-Time?
- RT-PREEMPT
- introduction,
- implementation,
- choosing appropriate kernel and patches version (long-time-support kernels, safety critical Linux, external tests)
- usage and tuning,
- testing latency and stabilizing it
- measuring in real use cases (example application controlling industrial process)
- POSIX-RT api
- threads (various aspects of real-time threads handling)
- process to kernel communication (mmap)
- memory locking and prefetching
- advanced real-time IPC - signals, shared memory, locking (semaphores, mutexes, futexes...), message queues and mailboxes.
- AIO – Asynchronous Input-Output - making use of non-real time drivers and peripherals, storing data on disk, using non real-time network.
After ‘Day 2’:
This day is devoted to methods of minimizing and locking worst case latency in
Linux kernel. Applications introduced there can be used on a standard,
non-modified system, but only utilizing PREEMPT-RT will give them determinism.
Parts of Linux, that after applying the patch, became real-time aware are
discussed. Application design essential to achieve full determinism is also
introduced. During hands-on exercises and workshop, the Embedded Linux-based
industrial control application will be created, measured and tuned in order to
achieve real-time determinism.
Day 3. Utilizing Co-kernel approach
- I-pipe (Adeos)
- Nano-kernel design, interrupt domains and pipelines
- Xenomai
- introduction,
- implementation,
- configuring optimizing and tuning
- RTDM – Real Time Device Model
- Xenomai Skins – the way of emulating traditional RTOS-es on top of I-Pipe
- IPC in Xenomai
- Xenomai – Linux communication
- measuring the ability of the system to respond to signal (the lowest possible latency)
- RTAI – alternative co-kernel solution
- introduction
- implementation
- configuring optimizing and tuning
- unique properties of RTAI
- RTAI-Lab – toolchain to develop block diagrams, that can be compiled and executed on the RTAI Linux system.
- Scilab, Scicos, Matlab, Simulink or RTW integration.
After ‘Day 3’
This part is devoted to another Linux real-time approach – utilizing the co-kernel
design. Real-time applications created and discussed here, are being controlled
by real-time co-kernel, that also virtualizes Linux kernel (which is used in
non-critical parts of a process). Various aspects of such system are discussed
here, the problems regarding real-time process to non-real time Linux based GUI
are posed and solved.
Day 4. Real-Time communication
- Real-Time networking
- network stack, protocols, sockets, raw sockets
- RTnet, RTmac
- distributed real-time processing
- Talking to the world
- input/output interfaces: Serial, GPIO, SPI, I2C, CAN
- ... and protocols CAN, Modbus
- analog input
- Industrial I/O
- Comedi Framework
- best practices of Real-Time Linux in Safety-Critical environments
- watchdog
- error recovery (memory ECC, error-safety algorithms).
After ‘Day 4’
Participants will gain knowledge, how to communicate from Linux Real-Time system
to the outer world. Exercises include various standard measuring and control
interfaces (RS232, GPIO, SPI, I2C and so on), as well as some specific ones:
Analog Input-Output, CAN and so on. Industrial IO and Comedi framework for
gathering data is also discussed and tested. In order to build distributed
system, necessary network protocols and Real-Time network (Xenomai based) are
also discussed, implemented, and tested.