Part II: Kernel Architecture

Architecture Overview

QRT is POSIX native Real-Time Kernel which implements a subset of the POSIX.1-2024 standard. In addition, QRT incorporates advanced real-time features such as High-Resolution Scheduling and Memory Protection.

Execution Model

QRT operates in two execution levels: Privileged and Unprivileged. Privileged code has unrestricted access to all system resources, while unprivileged code is restricted from modifying critical kernel or hardware registers.

When Memory Protection is disabled, all code executes in privileged mode and can access the entire memory map.

When Memory Protection is enabled, QRT runs the kernel, idle thread, and all interrupt service routines in privileged mode. User threads may execute in unprivileged mode, isolating them from kernel memory, other thread stacks, and preventing direct access to protected peripherals. User threads may also run in privileged mode if required.

Memory Model

Each thread and the kernel has its own stack memory, isolated from each other.

ARM Cortex-M processors provide two stack pointers, the Process Stack Pointer (PSP) and the Main Stack Pointer (MSP), which the kernel uses to keep thread stacks separate from the stack used by the kernel and interrupt handlers.

Boot & Kernel Bring-Up

QRT is shipped with default BSP for ARM Cortex-M platforms which handles necessary low-level initialization for the kernel operation. BSP must be initialized with bsp_init().

Before booting the kernel, the System Timer must be configured by calling qrt_systimer_configure().

The kernel is initialized and started by calling qrt_kernel_boot(), which takes thread attributes for the startup thread as parameters. After the kernel has started, the scheduler begins managing thread execution and the startup thread begins running. More threads can be created after the kernel has started.

The kernel handles Memory Protection Unit (MPU) configuration if Memory Protection is enabled (QRT_CFG_MPU_ENABLE).

Timers

System Timer

The kernel requires a hardware timer to serve as the primary timekeeping mechanism. The System Timer provides time-based services, ensuring accurate delays, timeouts, and scheduling. The resolution of timing services is set by the frequency of the hardware System Timer.

The System Timer interrupt is configured with the highest priority to maintain accurate timing and ensure no ticks are lost during interrupt handling. To satisfy this requirement, the System Timer interrupt must be the only interrupt assigned the highest priority level.

Configuration

The kernel requires integration with a hardware timer. The user must configure the hardware timer to trigger interrupts on overflow and provide qrt_systimer_configure() with the following parameters:

A function for reloading the hardware timer
A function for reading the hardware timer
The maximum period of the hardware timer in ticks
The frequency of the hardware timer in Hertz (Hz)
The hardware timer’s interrupt request number

The hardware timer’s interrupt handler must invoke qrt_systimer_handler(). The kernel manages the interrupt controller configuration for the hardware timer.

High-Resolution Scheduling

The kernel implements High-Resolution Scheduling: it has no periodic tick. The System Timer is programmed for the next sleep, timeout, timer expiry or round-robin time slice end that falls due, and interrupts when that event is due rather than at a fixed rate. If nothing falls due within the timer’s maximum period, it interrupts at the end of that period so that the kernel can keep system time. Timed services therefore resolve to one count of the hardware System Timer instead of a tick period, and there is no tick rate to configure. clock_getres() reports this resolution in whole nanoseconds.

Comparison with Tick-Resolution Scheduling

A kernel that uses Tick-Resolution Scheduling interrupts at a fixed tick rate, typically every millisecond, whether or not an event is due. Sleeps, timeouts and time slices resolve to whole tick periods, so the tick rate trades timing resolution against interrupt overhead. High-Resolution Scheduling removes that trade-off.

	Tick-Resolution Scheduling	High-Resolution Scheduling
Timer interrupts	Every tick, whether or not an event is due	When a sleep, timeout, timer or time slice falls due; otherwise only for kernel timekeeping, at the latest when the System Timer reaches its maximum period
Resolution of sleeps, timeouts and timers	One tick period	One System Timer count
Time slice length	A whole number of tick periods	Any number of System Timer counts
Tick rate	A compromise between resolution and overhead	None to configure

Tick-Resolution Scheduling diagram — Tick-Resolution Scheduling timing diagram.

System Time in Ticks

For compatibility with code written for tick-based kernels, QRT provides an API for querying the system time in ticks. The system time can be retrieved with qrt_systimer_getmonotonic(), which returns EINVAL until qrt_kernel_boot() has started the kernel. The value it reports is the elapsed system time expressed in ticks of the hardware System Timer, at its configured frequency, not in periods of a fixed-rate tick interrupt.

Scheduler

The scheduler manages thread execution, ensuring fair resource distribution, responsiveness, and prioritized handling of critical threads. The kernel uses preemptive scheduling, allowing it to interrupt a running thread whenever a higher priority thread becomes ready to execute.

Priority-Based Scheduling

The kernel uses preemptive priority-based scheduling, a scheduling algorithm designed to allocate processing time among threads based on their priority level, where a higher value represents higher priority.

To enable Priority-Based scheduling for a thread, pthread attribute must be set with SCHED_FIFO with pthread_attr_setschedpolicy().

A thread preempted by a higher-priority thread resumes before the other ready threads of its priority. A thread whose priority is changed with pthread_setschedparam() goes behind the ready threads of its new priority. pthread_setschedprio() keeps the thread’s place when the priority is unchanged, places a raised thread behind the ready threads of its new priority and a lowered thread ahead of them.

Round-Robin Scheduling

The kernel provides Round-Robin scheduling for threads of equal priority, ensuring fair distribution of CPU time among threads without strict real-time constraints, such as background tasks. Threads execute in rotation, each receiving a fixed execution time slice. When its time slice expires, the thread goes behind the other ready threads of its priority, and the next of them executes. A thread preempted by a higher-priority thread resumes first and runs the rest of its time slice.

To enable Round-Robin scheduling for a thread, pthread attribute must be set with SCHED_RR with pthread_attr_setschedpolicy().

Round-Robin scheduling is a superset of Priority-Based scheduling, meaning that Round-Robin scheduling also considers thread priority levels. Round-Robin scheduling is the default scheduling policy.

Interrupts and Scheduling

Interrupt Service Routines (ISRs) respond to hardware events by temporarily interrupting thread execution.

QRT does not perform scheduling directly within ISR context. Instead, QRT provides deferred system calls qrt_defer_call() for ISRs to request kernel services after the ISR has completed. This approach minimizes interrupt latency and ensures predictable execution.

Yielding

Threads do not need to explicitly yield, but the scheduler automatically preempts lower-priority threads and manages execution order based on priority and scheduling policy. Threads can voluntarily yield the processor using sched_yield(), allowing other threads of equal priority to execute. Yielding is particularly useful when a thread has completed its immediate work and wants to allow other threads to run.

Waiting Threads

Threads waiting on a semaphore, mutex or message queue are woken highest priority first, and in the order they started waiting among threads of equal priority. A waiting thread whose priority changes, through pthread_setschedparam(), pthread_setschedprio() or priority inheritance, is placed after the threads already waiting at its new priority, also when pthread_setschedprio() lowers it; only a ready thread goes ahead of its peers then.

Priority Inversion

Priority inversion occurs when a lower-priority thread indirectly prevents a higher-priority thread from executing, effectively reversing the intended priority order. Priority inversion can be bounded or unbounded:

Bounded priority inversion occurs when a high-priority thread is waiting for a resource held by a low-priority thread. The duration of inversion is bounded because it ends as soon as the low-priority thread releases the resource.
Unbounded priority inversion occurs if the duration of inversion becomes unpredictable due to intermediate-priority threads preempting the low-priority thread holding a critical resource needed by a high-priority thread.

QRT mitigates priority inversion through Priority Inheritance (PI) and Priority Ceiling (PC) protocols when using mutexes. See POSIX Thread Mutex for details on configuring these protocols.

Idle Thread

The kernel creates a idle thread that runs when no other threads are in the ready state. The idle thread always runs at the lowest priority level and System Calls cannot be invoked from it. Idle thread executes in a minimal infinite loop unless power-saving mode is enabled.

If power-saving mode is enabled, the idle thread places the processor into a low-power state when no other threads are ready. Power-saving mode can be enabled with configuration option QRT_CFG_IDLE_DEEP_SLEEP_ENABLE.

An idle thread hook can be registered and unregistered with qrt_kernel_idlehookregister() to execute custom code during idle periods. Care must be taken to ensure the hook function does not compromise the system security as it executes in kernel context.

Floating-Point Support

Some supported processor cores include a FPU (Floating-Point Unit), introducing additional registers that must be managed during context switching. Stacking floating-point registers increases thread stack size, adds interrupt latency, and extends context switching time because the registers must be saved and restored.

The kernel supports stacking floating-point registers and it can be enabled per thread with a call to pthread_setfpcontext_np() passing PTHREAD_FPCONTEXT_ENABLE_NP, and can be disabled by passing PTHREAD_FPCONTEXT_DISABLE_NP.

FPU support can be enabled with configuration option QRT_CFG_FPU_ENABLE, and floating-point stacking can be enabled with QRT_CFG_FPU_STACK_ENABLE.

Console

QRT ships a small, freestanding (libc-free) printf implementation used for boot and diagnostic output. Formatted text is buffered internally and emitted as whole strings through qrt_console_puts(), a weak hook that the platform overrides to target its console (UART, semihosting, ITM).

Configuration

Console output is gated by the QRT_CFG_CONSOLE_ENABLE option in qrt_cfg.h.

Output hook and linkage

qrt_console_puts() is defined weakly inside the kernel as a no-op, so the kernel links standalone with no console dependency. The platform supplies a strong definition with external linkage that routes characters to the actual output device:

/* Platform-side strong override. */
void qrt_console_puts(const char *str)
{
   while (*str != '\0')
   {
      uart_putc(*str);
      str++;
   }
}

Example boot output

With QRT_CFG_CONSOLE_ENABLE set to 1, the kernel prints a banner followed by a trace of each boot stage:

  ___  ____ _____
 / _ \|  _ \_   _|
| | | | |_) || |
| |_| |  _ < | |
 \__\_\_| \_\|_|

    v1.2.1 (yyyy-mm-dd)
    Copyright (c) Preempt Systems Oy

   Booting...
   Initializing CPU core
   Initializing heap allocators
     Kernel heap: 32768 B
     Thread stack pool: 65536 B
   Initializing software timer system
   Initializing threading subsystem
     Initializing scheduler
     Creating idle thread
     Creating startup thread
   Configuring thread stack
   Initializing Thread-Local Storage
   Starting System Timer
     Programming initial timer period of 48000000

   Boot sequence complete
   Dispatching startup thread at 0x000001f4