This course will provide a firm background in the kernel services and I/O structure of the FreeBSD kernel. The course will cover basic kernel services, locking, process structure, scheduling, signal handling, jails, capsicum sandboxing, and virtual and physical memory management. The kernel I/O structure will be described showing how I/O is multiplexed, disks are managed, special devices are configured, and system virtualization is done. The presentations will emphasize code organization, data structure navigation, and algorithms. It will not cover the machine specific parts of the system such as the implementation of device drivers.

Morning - Kernel Overview

• * Process structure
• * Locking
• * Communications
• * Process Groups and Sessions
• * Jails
• * Scheduling
• * Signals and timers
• * Virtual memory management

Afternoon - Kernel I/O structure

• I/O data structures
• Disk Management
• Multiplexing I/O
• Autoconfiguration strategy
• Configuration of a device driver

This course provides a broad overview of how the FreeBSD kernel implements its basic services. It will be most useful to those who need to learn how these services are provided. Individuals involved in technical and sales support can learn the capabilities and limitations of the system; applications developers can learn how to effectively and efficiently interface to the system; systems programmers without direct experience with the FreeBSD kernel can learn how to maintain, tune, and interface to such systems. This course is directed to users who have had at least a year of experience using a UNIX-like system. They should have an understanding of fundamental algorithms (searching, sorting, and hashing) and data structures (lists, queues, and arrays).

Prior to taking the course, students are recommended to obtain a copy of the course text: Marshall Kirk McKusick, George Neville-Neil, and Robert N. M. Watson, ‘‘The Design and Implementation of the FreeBSD Operating System’’, Second Edition, Pearson Education, Boston, MA September 2014, ISBN-13: 978-0-321-96897-5, ISBN-10: 0-321-96897-2.

Marshall Kirk McKusick

The OpenBSD Packet Filter (PF) is at the core of the network management toolset available to professionals working with the BSD family of operating systems.

Understanding the networking toolset is essential to building and maintaining a functional envirionment. The present session will teach the principles and hands-on operation of the extensive network tools available on OpenBSD and sister operating systems. Basic to intermediate understanding of TCP/IP networking is expected and required for this session.

Topics covered include

• The basics of and network design and taking it a bit further
• Building rulesets best pracitces, avoiding common pitfalls in firewall rule construction.
• Keeping your configurations readable and maintainable
• Filtering, diversion, redirection, Network Address Translation
• Handling services that require proxying (ftp-proxy and others)
• Address tables and daemons that interact with your setup through them
• The whys and hows of network segmentation, DMZs and other separation techniques
• Tackling noisy attacks and other pattern recognition and learning tricks
• Annoying spammers with spamd
• Basics of and not-so basic traffic shaping
• Monitoring your traffic
• Redundancy of PF firewalls using PF sync + CARP + Ifstated
• Troubleshooting: Discovering and correcting errors and faults
• Your network and its interactions with the Internet at large
• Common mistakes in internetworking and peering
• Keeping the old IPv4 world in touch with the new of IPv6
• Using PF and OpenBGPd together to implement an automated, distributed implementation of PF policies

Time allowing and to the extent necessary, we will cover recent developments in the networking tools and variations between the implementations in the sister BSD operating systems.

Participants should bring a laptop, the format of the session will be compact lectures interspersed with hands-on lab excercises based directly on the theory covered in the lecture parts.

This session is an evolutionary successor to previous sessions. Slides for previous PF tutorial sessions are up at https://home.nuug.no/~peter/pftutorial/, to be updated with the present version when the session opens.

Peter N. M. Hansteen, Senior Technical Specialist at Tietoevry. Author of The Book of PF (https://nostarch.com/pf3), occasional blogger (https://bsdly.blogspot.com) and lecturer on IT security with a strong preference for OpenBSD.

Massimiliano Stucchi, Technical Advisor at The Internet Society, IPv6 enthusiast, frequent lecturer on network security and IPv6 matters.

Tom Smyth, CTO Wireless Connect Ltd. ISP Network operator and security consultant.

This course will provide a firm background in the filesystems and networking capabilities supported by the FreeBSD kernel. The course describes the VFS filesystem interface that supports multiple filesystem types. The course covers the implementation and capabilities of the UFS filesystem and the techniques for maintaining filesystem consistency. The filesystem section ends with a description of the ZFS filesystem capabilities, implementation, and integration into FreeBSD. The course also covers the socket-based network architecture, layering, and implementation. The socket communications primitives and internal layering will be discussed, with emphasis on the interfaces between the layers; the TCP/IP implementation will be used as an example. A discussion of routing issues and the netmap interface will be included. The presentations will emphasize code organization, data structure navigation, and algorithms. It will not cover the machine specific parts of the system such as the implementation of device drivers.

This course provides a broad overview of how the FreeBSD kernel implements its basic services. It will be most useful to those who need to learn how these services are provided. Individuals involved in technical and sales support can learn the capabilities and limitations of the system; applications developers can learn how to effectively and efficiently interface to the system; systems programmers without direct experience with the FreeBSD kernel can learn how to maintain, tune, and interface to such systems. This course is directed to users who have had at least a year of experience using a UNIX-like system. They should have an understanding of fundamental algorithms (searching, sorting, and hashing) and data structures (lists, queues, and arrays).

Prior to taking the course, students are recommended to obtain a copy of the course text: Marshall Kirk McKusick, George Neville-Neil, and Robert N. M. Watson, ‘‘The Design and Implementation of the FreeBSD Operating System’’, Second Edition, Pearson Education, Boston, MA September 2014, ISBN-13: 978-0-321-96897-5, ISBN-10: 0-321-96897-2.

Marshall Kirk McKusick

This tutorial aims at teaching the basics of the BGP Protocol and some of the current best operational practices. The agenda of the tutorial is the following:

• Introduction to routing
• Difference between IGP and EGP
• Brief introduction to OSPF/IS-IS
• Introduction to BGP
• Exercise: Setting up a BGP Session with an upstream
• The BGP State machine and message types
• Exercise: Setting up a BGP Customer
• BGP Attributes
• Traffic engineering using BGP
• Exercise: Configuring localpref and communities on a BGP Session
• Scaling BGP, Route Reflectors
• Exercise: Move your network to use a RR
• Introduction to Routing Security
• Introduction to MANRS
• The Internet Routing Registry
• Exercise: Create your IRR Entries in the local IRR
• Setting up filters based on the IRR
• Exercise: Create filters for your upstream and your customer(s)
• Introduction and demo on PeeringDB
• Introduction to RPKI

Each participants will get a lab environment with routers based on OpenBGPd, FreeBSD + Bird and FreeBSD + FRR.

Target audience are BSD enthusiasts who would like to learn BGP and how to set up their own Autonomouse System with a hands-on course.

Basic knowledge of routing is required. A laptop with a working web browser is required.

Massimiliano Stucchi, Technical Advisor at The Internet Society, IPv6 enthusiast, frequent lecturer on network security and IPv6 matters.

The deduplication feature was added to ZFS version 21 and later incorporated into FreeBSD in early 2011 and first released as part of FreeBSD 8.3. In the more than decade since then, users have been advised to avoid the feature due to very poor performance. We present a series of enhancements to ZFS deduplication to improve performance and make the feature useful for general purpose storage and for high density VM hosting.

The modern home is increasingly becoming a hub of automated efficiency, necessitating the integration of diverse technologies. This paper introduces a FreeBSD-based system that seamlessly integrates Home Assistant, Apache Kafka, and OpenSearch for managing essential home functions such as lighting, shutters, and energy consumption. The system adeptly utilizes a range of data sources, including proprietary protocols like KNX and versatile ReST-APIs, to facilitate real- time analytics and event-triggered responses. This results in a self-learning home automation environment, uniquely prioritizing energy optimization while simultaneously enhancing comfort in contemporary homes.

In November 2023 a silent data corruption bug was discovered in the recently released OpenZFS 2.2.0. The bug was reported, and quickly correlated by the community with a new feature, “block cloning”, which even before its release had gained a reputation for instability and data loss, to the point of being disabled by default in FreeBSD 14. A 2.2.1 release was quickly put together, disabling the feature.

One of OpenZFS’ key selling points is its ability to detect and correct all kinds of data corruption, so news of its apparent failure quickly spread through tech news and social media, aided by its somewhat prickly relationship with some other parts of the open-source software world. This naturally led to a lot of confusion and concern from home and business OpenZFS users alike, worried about the integrity of their data.

With many outside eyes on the bug report, a reproduction test case was developed and it was quickly shown that the bug was present on the previous 2.1 series, and it was theorised that it might have been present right back to the original open-source ZFS release. This naturally caused even more confusion and concern, as now people were concerned about the integrity of their decades-old archive storage.

Over the Thanksgiving long weekend a few OpenZFS developers and many users rallied around the problem. A workaround was identified and documented. The bug was studied and gradually understood, and a fix was developed. People shared ideas and how might identify if their data was corrupt. Others theorised about different workloads that could make the problem easier or harder to hit. Still others took it upon themselves to find ways to run older versions, to find out just how far back the problem went.

This is a story about a 15-year old bug, where it came from, how it works, how it eluded users and developers alike through that time, and how a community pulled together to understand and fix it.

The FreeBSD open source operating system[1] provides a powerful set of features to facilitate the deployment, virtualization, and serving of Microsoft Windows environments ranging from small research labs to enterprise deployments. Its exemplary integration with the OpenZFS file system and volume manager, its bhyve hypervisor, and its overall unity play key roles in FreeBSD’s ability to perform these three roles and combined with ported third party software, it can facilitate deployment, virtualization, and serving. This paper will describe each of these facilities with examples of their usage.

On OpenZFS, `fsync()` cannot fail - it will wait until the application’s changes are on disk before it returns. If there is a problem, such as a hardware failure, that causes the pool to suspend, then it may wait forever. This feels strange, but is acceptable according to the API contract: `fsync()` never returned success, so the application has no reason to believe its data on disk.

However, OpenZFS pools can recover if the fault is repaired, and so `fsync()` can still return. As it turns out though, its possible in rare situations for the pool to return to service but not have actually put the data on disk. `fsync()` returns success, because it cannot fail, and the application has been lied to.

In this paper I describe the path taken from the `fsync()` call, through the ZFS Intent Log, the transaction machinery, the pool failure system and the IO scheduler to understand what happens to IO when disks fail and return and why OpenZFS believed that writes had succeeded when they had not. I then describe the changes I made to make OpenZFS understand that something had gone wrong, and how I threaded that response was threaded back up the stack such that `fsync()` could finally return failure - and what it means when it does.

This paper presents a pioneering initiative to in- tegrate OpenStack, an open-source cloud computing platform, with FreeBSD, a robust Unix-like operating system. Traditionally, OpenStack has been closely associated with Linux-based environ- ments, leveraging specific Linux features and technologies. This integration aims to expand OpenStack’s applicability by har- nessing FreeBSD’s advanced networking, security, and efficient resource management capabilities.

The project scope involves adapting OpenStack’s key compo- nents to function seamlessly within FreeBSD’s system architec- ture, focusing on virtualization with bhyve and FreeBSD’s unique networking stack. A Proof of Concept (PoC) has been successfully developed, demonstrating the viability of this integration and laying a foundation for further development.

We address several critical challenges in this integration process, including adapting libvirt for bhyve, managing VLANs, modifying Open vSwitch for FreeBSD, ensuring efficient DHCP services, and aligning FreeBSD’s network namespace and firewall functionalities with OpenStack’s requirements. The project also tackles the adaptation of OpenStack’s oslo.privsep library to FreeBSD’s privilege model and addresses the complexities of nested virtualization and VM console access within FreeBSD.

Future work involves expanding the integration to more Open- Stack components, enhancing system performance, and fostering collaboration within the FreeBSD and OpenStack communities. This integration represents a significant advancement in cloud computing, offering a versatile platform that combines the strengths of both OpenStack and FreeBSD. The project invites collaboration and contribution from the community to overcome the challenges and fully realize the potential of this innovative integration.

OpenZFS, a well-known and modern file system, evolved from the original ZFS developed for OpenSolaris and is now predominantly developed for FreeBSD and Linux. Its copy-on-write model enables powerful features such as an always-consistent on-disk state, along with rapid file system snapshots and clones. A frequently requested feature by users has been the ability to clone individual files, a need highlighted by the introduction of the copy_file_range(2) system call, which could leverage such functionality.

This presentation will delve into the implementation of block cloning in OpenZFS, outlining the challenges encountered and the solutions devised to overcome them, thus enhancing the system's capabilities and efficiency.

Container virtualization technologies such as Docker [1] are now widely used as lightweight application execution environments in cloud services. They are based on OS pro- cesses, which are lightweight compared with traditional VMs, resulting in faster startup and a lower memory footprint. Many cloud services [2], [3] take advantage of the characteristics of containers for rapid scale-out, scale-in, and more efficient resource usage.

We aim to achieve a container sandbox with Container Transplantation in which Linux containers run compatibly in another OS and apply another OS-specific security mechanism to resolve the trade-off between robust container isolation and container characteristics and application performance. Since many cloud services utilize Linux as the environment for hosting containers, vulnerabilities in the Linux kernel and its functions are directly linked to container vulnerabilities. Unfortunately, attacks that exploit those vulnerabilities often have fatal adverse effects that make it impossible for cloud services to run correctly, such as unauthorized access to hosts and other containers through privilege escalation and the theft of sensitive information. We have proposed that executing Linux containers in another OS can avoid these various attacks that exploit vulnerabilities related to the Linux kernel and its functionality. Moreover, it is possible to apply unique security models and functions implemented in various OSes to Linux containers to achieve more finely-grained access control and isolation between containers resilient to attacks targeting specific OSes.

This paper investigates the possibility of resisting attacks using Linux kernel vulnerabilities through the Linux jails. Among the vulnerabilities in Linux Kernel 4.4, emulated by Linux jails on FreeBSD13.2, we have chosen those that the Linuxulator can emulate. Then, we categorized the Linux kernel vulnerabilities that the secure container must be able to avoid, tried the attacks on both Linux and the Linuxulator, and summarized the success or failure of the attacks. Linux- ulator, and summarized the success or failure of the attacks. Our experimental results illustrate that the Linuxulator can efficiently repel exploits against Linux kernel vulnerabilities.

The use of virtual memory practically eliminated the need for the contiguity of physical mem- ory allocation as physically discontiguous memory can be contiguous in the virtual address space. Unfortu- nately, avoiding performance degradation for memory- intensive workloads on modern CPUs requires a steady supply of contiguous physical memory, making external physical memory fragmentation in modern operating systems a serious issue once again.

This paper presents the design and implementation of several several anti-fragmentation mechanisms for the FreeBSD kernel. Parts of this work were sponsored by the Google Summer of Code ’23 program.

This presentation delves into the innovative integration of vector databases with ChatGPT, enriching the capabilities of the FreeBSD Expert System. The synergy of these technologies opens new avenues for intelligent, adaptive system administration tools.

Design Driven Development using the Spin Verifier.

The Problem: Currently, the NetBSD sources are inherited from historic research, and maintained by a small, but committed group of volunteers. Often, the role of the NetBSD developer is to implement or port the design and implementation of a research project, and this involves buildup of design understanding and interpretation within the memory of a few “experts” within the community. This poses a few problems, especially for newcomers to the project:

1. Documentation and reading code alone, is insufficient to capture design nuances.
2. The “experts” can become a point of gated access or failure.
3. Design discussions with new developers often occur informally in the form of socratic questioning, or Q&A with an “expert” oracle.
4. Design and implementation may drift, simply due to the lack of a canon for design, and more importantly a way to mechanically check it for consistency.

In this talk, I demonstrate DDD, compare it with industry’s “Model Based System Development” approaches, and show how the tech- nique may be used to help architects and developers (who may often be the same person), to separate concerns.

For architects, it takes off the burden of having to rely on their memory, random tests and bespoke models to keep state. For developers, it provides a clear ”source of truth” and a process based mechanism to verify their implementation based on this source. And for project managers, there’s much better visibility over the entire Q&A, analogous to but much more powerful than a simple TDD style development process.

Although UNIX’s descendants or derivatives are not hard real-time operating systems, some have support for soft real-time through allowing to assign to userland processes higher priorities normally reserved to the system, sometimes coupled with preemption of kernel tasks, making them suitable as a fundation of soft real-time system. POSIX standardized its first real-time extensions in 1993 in a document usually referred to as POSIX.1b. At that time, however, some operating systems already had support for soft real-time in the form of ad-hoc APIs, such as System V Release 4 (SVR4) with its priocntl(2) system call and HP/UX with its rtprio(2) one. FreeBSD first adopted its own rtprio(2) system call in 1994, largely based on HP/UX’s with extensions such as idle processes. POSIX.1b extensions concerning processes were implemented later, in 1998, and some preliminary thread support added the next year. Since then, these APIs have been present in the system for applications to use.

In this paper, we provide a thorough description of both FreeBSD’s rtprio(2) and POSIX.1b’s scheduling interfaces and embark on a journey around FreeBSD’s implementation of scheduling priorities. It started with a desire to fix a few apparently simple bugs of rtprio(2) and to add some reasonable features and, one thing leading to another, became an almost complete rewrite of this system call and the POSIX.1b’s interfaces’ implementations. We will expose the many problems that the current implementation has, in terms of POSIX compliance, security and consistency and how we are fixing them. As of this writing, this project is still a work in progress, and we will report about its status during the conference presentation.

DTrace and eBPF are today’s most potent observ- ability tools available on general-purpose operating systems. They empower users to ask random questions and receive complex answers about any part of the system. Their performance is unmatched by any traditional observability tools.

Unfortunately, their performance characteristics are not well- researched, partially because designing a benchmark to measure the measurement tool is challenging.

In this paper, we learn about the basics of art benchmarking and the importance and difficulties of operating system instru- mentation. We review three generations of observability tools to understand their design and implementation limitations. Finally, we design, implement, and conduct a microbenchmark and an application benchmark to peak into the enigmatic domain of DTrace and eBPF performance overhead.

This paper presents the high-level ideas behind kinst, a new DTrace provider available in FreeBSD, which enables tracing of arbitrary instructions and inline functions in the kernel.

This paper introduces the low level debugger (LLDB) kernel module debug facility for the FreeBSD kernel. The current functional status of LLDB within the FreeBSD kernel is attributed to contributions from [1] and the collaborative efforts of the community. Key functionalities include core dump parsing and memory context building for the coredump, specifically integrated into the process plugin within LLDB for the FreeBSD kernel. This enhancement equips LLDB for effective post-mortem debugging on the FreeBSD kernel. While the implementation of the process plugin has been successfully completed, the paper emphasizes the imperative need to implement the Dynami- cLoader plugin for the kernel loader. This plugin plays a critical role in loading the symbol file of the kernel module, ensuring comprehensive parsing of symbols for loadable kernel modules. Additionally, given the potential existence of the kernel module as either a relocatable file (for x86) or a shared object (for ARM), the implementation should confirms the usability of both types of ELF format kernel modules.