Real-Time Linux in Industrial Appliances

Booming usage of Linux in embedded environments is creating an on-going demand to use this general-purpose operating system in industrial control devices and other fields, previously reserved for classic real-time operating systems. The course covers key topics, essential to understand the opportunities in using Linux as a real-time operating system and its fails. You will learn what Real-Time really means and how to achieve it in Linux.

Exercises done on ARM based development board with example of Real-Time application (controlling real industrial process), which will be developed during the course, are to give you a strong technical understanding and confidence in the subject.

Emphasis is put on developing POSIX-RT based real-time applications using some modern Linux variants, in order to work out a universal and re-usable framework, for your future Real-Time Linux projects. In order to learn programming techniques, the software is verified with partial tests and latency measurements are done during the exercises.

Duration

4-days (32h)

Course outline

• Introduction to Real-Time, a world about software and hardware design
• terminology (latency, jitter, worst case, safety critical)
• where the latency comes from
• Linux and Real-Time
• RT-PREEMPT – introduction, implementation, usage and tuning, measuring in real use cases
• Xenomai – utilizing co-kernel design in applications, introduction, implementation, optimizing and tuning, real use cases, RTDM (Real-Time-Device-Model), Adeos/I-pipe
• RTAI (LinuxCNC) – alternative co-kernel approach
• Safety-Critical environments (watchdog, memory error recovery).
• POSIX-RT api
• threads
• advanced real-time IPC (signals, shared memory, locking, message queues and mailboxes).
• AIO – main use of non-real time drivers and peripherals
• Real-Time networking
• RTnet, RTmac
• distributed real-time processing
• best practices of Real-Time Linux
• Talking to the world
• input/output interfaces: Serial, GPIO, SPI, I2C
• analog input
• Industrial I/O

Detailed agenda

Day. 1, Linux and Real-Time – the introduction

• Refreshing the information about Embedded Linux
• Linux as a general-purpose operating system.
• Introduction to Real-Time, a world about software and hardware design
• terminology (latency, jitter, worst case, safety critical)
• where the latency comes from
• latency-related aspects of kernel configuration
• interrupts, delayed interrupt handling
• kernel preemption (voluntary and forced preemption)
• utilizing High Resolution Timers
• Measuring Embedded Linux latency
• external methods (oscilloscope, logic analyzer)
• internal timers, and latency buffer
• kernel profiling
• Tracers (ftrace), function tracing and histograms
• generating load and emulating real usage
• Linux and Real-Time
• Overview of Real-Time solutions in Linux (RT-PREEMPT, co-kernel)

After ‘Day 1’:

The applicable terminology will be defined and problems introduced. Participants will review their knowledge of Linux Kernel and userspace. Latency and other problematic variables will be measured. Eventually the general solution will be introduced (Real-Time patches or co-kernel).

Day 2. How to make Linux Real-Time?

• RT-PREEMPT
• introduction,
• implementation,
• choosing appropriate kernel and patches version (long-time-support kernels, safety critical Linux, external tests)
• usage and tuning,
• testing latency and stabilizing it
• measuring in real use cases (example application controlling industrial process)
• POSIX-RT api
• threads (various aspects of real-time threads handling)
• process to kernel communication (mmap)
• memory locking and prefetching
• advanced real-time IPC - signals, shared memory, locking (semaphores, mutexes, futexes...), message queues and mailboxes.
• AIO – Asynchronous Input-Output - making use of non-real time drivers and peripherals, storing data on disk, using non real-time network.

After ‘Day 2’:

This day is devoted to methods of minimizing and locking worst case latency in Linux kernel. Applications introduced there can be used on a standard, non-modified system, but only utilizing PREEMPT-RT will give them determinism. Parts of Linux, that after applying the patch, became real-time aware are discussed. Application design essential to achieve full determinism is also introduced. During hands-on exercises and workshop, the Embedded Linux-based industrial control application will be created, measured and tuned in order to achieve real-time determinism.

Day 3. Utilizing Co-kernel approach

• I-pipe (Adeos)
• Nano-kernel design, interrupt domains and pipelines
• Xenomai
• introduction,
• implementation,
• configuring optimizing and tuning
• RTDM – Real Time Device Model
• Xenomai Skins – the way of emulating traditional RTOS-es on top of I-Pipe
• IPC in Xenomai
• Xenomai – Linux communication
• measuring the ability of the system to respond to signal (the lowest possible latency)
• RTAI – alternative co-kernel solution
• introduction
• implementation
• configuring optimizing and tuning
• unique properties of RTAI
• RTAI-Lab – toolchain to develop block diagrams, that can be compiled and executed on the RTAI Linux system.
• Scilab, Scicos, Matlab, Simulink or RTW integration.

After ‘Day 3’

This part is devoted to another Linux real-time approach – utilizing the co-kernel design. Real-time applications created and discussed here, are being controlled by real-time co-kernel, that also virtualizes Linux kernel (which is used in non-critical parts of a process). Various aspects of such system are discussed here, the problems regarding real-time process to non-real time Linux based GUI are posed and solved.

Day 4. Real-Time communication

• Real-Time networking
• network stack, protocols, sockets, raw sockets
• RTnet, RTmac
• distributed real-time processing
• Talking to the world
• input/output interfaces: Serial, GPIO, SPI, I2C, CAN
• ... and protocols CAN, Modbus
• analog input
• Industrial I/O
• Comedi Framework
• best practices of Real-Time Linux in Safety-Critical environments
• watchdog
• error recovery (memory ECC, error-safety algorithms).

After ‘Day 4’

Participants will gain knowledge, how to communicate from Linux Real-Time system to the outer world. Exercises include various standard measuring and control interfaces (RS232, GPIO, SPI, I2C and so on), as well as some specific ones: Analog Input-Output, CAN and so on. Industrial IO and Comedi framework for gathering data is also discussed and tested. In order to build distributed system, necessary network protocols and Real-Time network (Xenomai based) are also discussed, implemented, and tested.