Iris Kernel Overview

Reason for Writing a New Kernel for Iros

The main question here is why abandon development of the old kernel written in C for a new one. There are 2 primary reasons:

1. The old kernel was unmaintainable, and C++'s ability to add safe abstractions will improve the situtation. For one, RAII and the Result class make handling errors without leaking resources effectively trivial. Smart pointers should make memory safety bugs significantly less likely, and class templates like di::Synchronized should make thread-safety simpler.
2. The old kernel was modelled strictly off of POSIX, which includes an extremely large body of obsolete and duplicate functionality. Writing a kernel with asynchronicity in mind, and leaving userspace to emulate POSIX APIs, many of which are duplicated (think SYS-V semaphores vs. POSIX semaphores), will result in a much cleaner design.

The current issues with the old kernel are mostly around poor handling of error conditions and frequent concurrency issues, which make the system very unstable when trying to do something like run GCC. Due to performance issues in the kernel, running GCC takes at least 10x longer than it would on linux, and that is only considering compiling a single file. Attempting to compile the entire code base would would most likely take hours, but GCC would crash first anyway, even on a single core processor.

There are also several missing features, like support for USB, newer hardware platforms like AHCI, or nvme, IP v6 networking, or even any form of audio playback. The benefits of using a microkernel architecture are that prototyping these sorts of drivers can be done in userspace, which makes common concerns like thread-safety, handling error conditions, and avoiding memory allocations at all cost, much less problematic.

If a new AHCI driver crashes, the system will still be operational and the driver can simply be restarted. Thread-safety is irrelevant if the driver only has 1 thread, and only communicates with the kernel through a pair of SPSC queues. Hypothetically, a userspace program could actually be debugged using gdb, without freezing the entire system. This will make developing this functionality way more managable.

Ultimately, the goal of this new kernel is to acheive good performance while also making development of new features easier.

Design Priorities of the Iris Kernel

1. Asyncrounous
2. Minimal
3. Performant

Aysynchronous

By default, every system call should be non-blocking. Ideally, there will be exactly one mechanism by which processes can block themselves, and no others.

Minimal

For security and robustness, as few components as possible should reside within the kernel. Therefore, the kernel will only handle scheduling, memory management, and direct hardware access.

Performant

The core IPC mechanism needs to be fast and scalable. This minimizes any performance loss caused by using a micro-kernel.

Micro-Kernel vs. Monolithic Kernel

In a "classical" monolithic kernel, all supervisor code, including all drivers, runs in ring 0 in the same singlular address space. In a micro-kernel architecture, as many components as possible are moved into separate processes, most hopefully running in ring 3.

Pros of a Micro-Kernel

• Increased fault tolerance (if the NIC driver crashes, the system can keep running)
• Increased security (it is much harder to exploit components not directly in the kernel)
• Increased modularity (different components can be developed in isolation)

Cons of a Micro-Kernel

• Increased complexity (now, multiple different processes must orchestrate themselves together)
• Relies on Inter-Process Communication (not the much simpler interface, function calls)
• Reduced performance (more context switches, non-blocking APIs, IRQ dispatch, ...)
• Requires complex work-arounds (like bootstrapping a file system so that programs can be loaded)

Pros of a Monolithic Kernel

• Components communicate directly using function calls
• Increased performance (component communication has no overhead)

Cons of a Monolithic Kernel

• Large attack surface
• Failures in less significant components can cause the entire kernel to panic

General Analysis

It is generally held that a micro-kernel architecture is superior, if doing so is actually viable (including achieving comparable performance). This quest has resulted in large amounts of academic research investigating the subject.

Sometimes, the performance loss caused by requiring IPC to communicate with the kernel can be mitigated by having the kernel execute a verifiably safe program provided by said component.

This technique is used by microkernels to handle hardware IRQs without waiting for a response from the underlying device driver. The driver only needs to provide a simple program which convinces the device the interrupt has been received. Then, the kernel arranges for the driver to be scheduled, at which point actual work will be done (e.g., notifying another component the disk read was successful). This 2 phase approach to interrupt handling can be considered to the Linux soft IRQ mechanism. Additionally, hybrid-poll based drivers can disable interrupts when the device is known to actively be doing work, and instead continually poll the device.

This technique is additionally not exclusive to micro-kernels. The eBPF feature of Linux allows userspace code to control program executing in kernel context. This has numerous use cases, including in AF_XDP, where it is used to filter incoming packets before they are handed off to higher layers of the network stack. Using this functionality, the network protocols (e.g. IP v6, TCP, TLS, etc.) can all be implemented directly in userspace.

File System Analysis

Linux also provides the FUSE API for implementing file system drivers in userspace. This is considered valuable for prototyping file system implementations, but file system drivers are usually converted into kernel space versions, in order to achieve supererior performance.

A pure userspace file system implementation faces numerous challenges which make obtaining comparable performance difficult.

The first relevant issue is the so-called "page cache". There must be a centralized cache of memory to minimize memory usage and enable any application to access this cache. Therefore, the kernel must be able to manage the page cache entries with the helper of file system drivers (for example, to flush dirty pages out to disk).

Along with the "page cache", the results of path lookups should also be cached. Using Linux, the dirent cache is consulted along every step of the traversal. For optimal performance, this lookup must be done by a central authority. The result of path lookup is further complicated by the mutable nature of the file system. When a file is deleted, the central authority must be made aware of this deleting to not end up with a stale cache.

Another notable problem is that of frequent context switches. When a userspace program requests to read 2048 bytes from a file, they would likely make a request to some central authority. This request then gets routed to the proper userspace driver (if the requested page is not in the "page cache"). At this point, the device driver must locate the disk sector which contains the relevant data. In the simply case, this can be done without any communication with the disk driver. However, actually reading the data requires calling into the disk driver. Finally, when the disk driver receives a completion event, the chain unravels, passing through the device driver into the cetral authority and then back the requesting application. On a single-processor system, this requires at least 4 context switches.

However, imagine instead this scenario with a 16 core machine that is actively reading an entire 32 MiB file. When doing sufficently large amounts of IO, the effective operation cost be bounded by the speed of the disk itself. To read the 32 MiB file as quickly as possible, it is enough to ensure the disk's queue of read requests is always full. During this scenario, the separate processes will run on separate cores, communicating with each other through lock-less SPSC queues (think Linux io_uring). In this scenario, throughput is maximized, since the file system driver can submit multiple blocks to be read at once.

Notice, this requires the underlying application uses asynchronous IO itself, will allows it to drive more and more requests for data. When using a synchronous IO, the kernel would have to specutively prefetch subsequent sectors to acheive a similar effect.

The conclusion is thus that asynchronous IO can tolerate the increased context switches readily, and although there is higher latency when the "page cache" is missed, throughput should be unaffected.

IPC Wake Up

Another serious issue to consider is the mechanism to wake tasks which have received requests over the IPC mechanism. In Linux io_uring, a system call (io_uring_enter) is used to notify the kernel there is work to do. Alternatively, an application can request the kernel busy-poll its submission queue for new entries. Note this assumes that the IPC is purely asynchronous.

In a multi-processing scenario, the responsible task could already be awake and busy, and thus the calling program needs not wakeup anyone. However, it is equally possible that the task needs to be awoken. To do this, the kernel itself must perform some sort of mediation. This could be implemented by calling into the kernel, which would then attempt to schedule the relevant task, before returning to the caller.

It would be nice if the caller could directly call into the responsible task (using an interrupt vector), but this cannot be considered scalable (given that only 255 vectors are available). More research would be needed to determine the optimal mechanism for this case.