The deduplication feature was added to ZFS version 21 and later incorporated into FreeBSD in early 2011 and first released as part of FreeBSD 8.3. In the more than decade since then, users have been advised to avoid the feature due to very poor performance. We present a series of enhancements to ZFS deduplication to improve performance and make the feature useful for general purpose storage and for high density VM hosting.
The modern home is increasingly becoming a hub of automated efficiency, necessitating the integration of diverse technologies. This paper introduces a FreeBSD-based system that seamlessly integrates Home Assistant, Apache Kafka, and OpenSearch for managing essential home functions such as lighting, shutters, and energy consumption. The system adeptly utilizes a range of data sources, including proprietary protocols like KNX and versatile ReST-APIs, to facilitate real- time analytics and event-triggered responses. This results in a self-learning home automation environment, uniquely prioritizing energy optimization while simultaneously enhancing comfort in contemporary homes.
In November 2023 a silent data corruption bug was discovered in the recently released OpenZFS 2.2.0. The bug was reported, and quickly correlated by the community with a new feature, “block cloning”, which even before its release had gained a reputation for instability and data loss, to the point of being disabled by default in FreeBSD 14. A 2.2.1 release was quickly put together, disabling the feature.
One of OpenZFS’ key selling points is its ability to detect and correct all kinds of data corruption, so news of its apparent failure quickly spread through tech news and social media, aided by its somewhat prickly relationship with some other parts of the open-source software world. This naturally led to a lot of confusion and concern from home and business OpenZFS users alike, worried about the integrity of their data.
With many outside eyes on the bug report, a reproduction test case was developed and it was quickly shown that the bug was present on the previous 2.1 series, and it was theorised that it might have been present right back to the original open-source ZFS release. This naturally caused even more confusion and concern, as now people were concerned about the integrity of their decades-old archive storage.
Over the Thanksgiving long weekend a few OpenZFS developers and many users rallied around the problem. A workaround was identified and documented. The bug was studied and gradually understood, and a fix was developed. People shared ideas and how might identify if their data was corrupt. Others theorised about different workloads that could make the problem easier or harder to hit. Still others took it upon themselves to find ways to run older versions, to find out just how far back the problem went.
This is a story about a 15-year old bug, where it came from, how it works, how it eluded users and developers alike through that time, and how a community pulled together to understand and fix it.
The FreeBSD open source operating system[1] provides a powerful set of features to facilitate the deployment, virtualization, and serving of Microsoft Windows environments ranging from small research labs to enterprise deployments. Its exemplary integration with the OpenZFS file system and volume manager, its bhyve hypervisor, and its overall unity play key roles in FreeBSD’s ability to perform these three roles and combined with ported third party software, it can facilitate deployment, virtualization, and serving. This paper will describe each of these facilities with examples of their usage.
On OpenZFS, `fsync()` cannot fail - it will wait until the application’s changes are on disk before it returns. If there is a problem, such as a hardware failure, that causes the pool to suspend, then it may wait forever. This feels strange, but is acceptable according to the API contract: `fsync()` never returned success, so the application has no reason to believe its data on disk.
However, OpenZFS pools can recover if the fault is repaired, and so `fsync()` can still return. As it turns out though, its possible in rare situations for the pool to return to service but not have actually put the data on disk. `fsync()` returns success, because it cannot fail, and the application has been lied to.
In this paper I describe the path taken from the `fsync()` call, through the ZFS Intent Log, the transaction machinery, the pool failure system and the IO scheduler to understand what happens to IO when disks fail and return and why OpenZFS believed that writes had succeeded when they had not. I then describe the changes I made to make OpenZFS understand that something had gone wrong, and how I threaded that response was threaded back up the stack such that `fsync()` could finally return failure - and what it means when it does.
This paper presents a pioneering initiative to in- tegrate OpenStack, an open-source cloud computing platform, with FreeBSD, a robust Unix-like operating system. Traditionally, OpenStack has been closely associated with Linux-based environ- ments, leveraging specific Linux features and technologies. This integration aims to expand OpenStack’s applicability by har- nessing FreeBSD’s advanced networking, security, and efficient resource management capabilities.
The project scope involves adapting OpenStack’s key compo- nents to function seamlessly within FreeBSD’s system architec- ture, focusing on virtualization with bhyve and FreeBSD’s unique networking stack. A Proof of Concept (PoC) has been successfully developed, demonstrating the viability of this integration and laying a foundation for further development.
We address several critical challenges in this integration process, including adapting libvirt for bhyve, managing VLANs, modifying Open vSwitch for FreeBSD, ensuring efficient DHCP services, and aligning FreeBSD’s network namespace and firewall functionalities with OpenStack’s requirements. The project also tackles the adaptation of OpenStack’s oslo.privsep library to FreeBSD’s privilege model and addresses the complexities of nested virtualization and VM console access within FreeBSD.
Future work involves expanding the integration to more Open- Stack components, enhancing system performance, and fostering collaboration within the FreeBSD and OpenStack communities. This integration represents a significant advancement in cloud computing, offering a versatile platform that combines the strengths of both OpenStack and FreeBSD. The project invites collaboration and contribution from the community to overcome the challenges and fully realize the potential of this innovative integration.
OpenZFS, a well-known and modern file system, evolved from the original ZFS developed for OpenSolaris and is now predominantly developed for FreeBSD and Linux. Its copy-on-write model enables powerful features such as an always-consistent on-disk state, along with rapid file system snapshots and clones. A frequently requested feature by users has been the ability to clone individual files, a need highlighted by the introduction of the copy_file_range(2) system call, which could leverage such functionality.
This presentation will delve into the implementation of block cloning in OpenZFS, outlining the challenges encountered and the solutions devised to overcome them, thus enhancing the system's capabilities and efficiency.
Container virtualization technologies such as Docker [1] are now widely used as lightweight application execution environments in cloud services. They are based on OS pro- cesses, which are lightweight compared with traditional VMs, resulting in faster startup and a lower memory footprint. Many cloud services [2], [3] take advantage of the characteristics of containers for rapid scale-out, scale-in, and more efficient resource usage.
We aim to achieve a container sandbox with Container Transplantation in which Linux containers run compatibly in another OS and apply another OS-specific security mechanism to resolve the trade-off between robust container isolation and container characteristics and application performance. Since many cloud services utilize Linux as the environment for hosting containers, vulnerabilities in the Linux kernel and its functions are directly linked to container vulnerabilities. Unfortunately, attacks that exploit those vulnerabilities often have fatal adverse effects that make it impossible for cloud services to run correctly, such as unauthorized access to hosts and other containers through privilege escalation and the theft of sensitive information. We have proposed that executing Linux containers in another OS can avoid these various attacks that exploit vulnerabilities related to the Linux kernel and its functionality. Moreover, it is possible to apply unique security models and functions implemented in various OSes to Linux containers to achieve more finely-grained access control and isolation between containers resilient to attacks targeting specific OSes.
This paper investigates the possibility of resisting attacks using Linux kernel vulnerabilities through the Linux jails. Among the vulnerabilities in Linux Kernel 4.4, emulated by Linux jails on FreeBSD13.2, we have chosen those that the Linuxulator can emulate. Then, we categorized the Linux kernel vulnerabilities that the secure container must be able to avoid, tried the attacks on both Linux and the Linuxulator, and summarized the success or failure of the attacks. Linux- ulator, and summarized the success or failure of the attacks. Our experimental results illustrate that the Linuxulator can efficiently repel exploits against Linux kernel vulnerabilities.