Embedded OpenBSD

A while ago at work, I experienced some problems with NTP server reliability. We had some Soekris 4501 boxes that were laying around in the way and gathering dust, because they hadn't worked out for a project attempted prior to my being hired. Rather than having them thrown out or otherwise wasted, I decided they would make great dedicated NTP servers. A bit of research revealed that OpenBSD and FreeBSD supported the hardware rather well. Because we were already running OpenBSD on our routers, I decided to go with that. I wanted to be able to run the OS from a ramdisk-based filesystem, but I wasn't particularily pleased with any of the pre-existing embedded setups I found online. Knowing that the OpenBSD install used a ramdisk filesystem, I decided to look into what it would take to create a custom system.

It turns out that OpenBSD's ramdisk root filesystem is compiled into the kernel. Having created custom-compiled Linux kernels in the past, my first thought was: No problem, I can do this! But further investigation revealed that the OpenBSD world strongly frowns upon custom-compiled kernels. As a matter of fact, the FAQ on the official OpenBSD web site goes so far as to strongly say that compiling a custom kernel is NOT supported ( http://www.openbsd.org/faq/faq5.html#Why):

Actually, you probably don't.

A custom kernel is a kernel built with a configuration file other than the provided GENERIC configuration file. A custom kernel can be based on -release, -stable or -current code, just as a GENERIC kernel can be. While compiling your own GENERIC kernel is supported by the OpenBSD team, compiling your own custom kernel is not.

...

Some reasons why you should not build a custom kernel:

...

Again, developers will usually ignore bug reports dealing with custom kernels, unless the problem can be reproduced in a GENERIC kernel as well. You have been warned.

So in light of this information, how could I create a ramdisk-based filesytem without compiling a custom kernel? Previous research had uncovered some interesting information about OpenBSD's MFS filesystem (Memory File System). One of the features that intrigued me was that it's mount command included a -P option, which would populate the filesystem with a specified directory. This started me thinking about how to go from a booting a root filesystem from a Compact Flash card, to mounting and populating the "usual suspects": /bin, /etc, /dev, /usr, /var and /tmp/. This would solve the need to run from a ramdisk, without having to use a custom-compiled kernel!

Another (self-imposed) requirement for the project was the ability to easily install and configure this custom, embedded OpenBSD system. Looking into how the OpenBSD install system works, I discovered that the native system could be used to install custom versions of the base and etc gzipped tarballs, as long the MD5 file is updated accordingly (keep in mind that we were using version 4.4 at the time, and that with current versions you'd need to modify the SHA256 file). Also, the index.txt file could be modified to only include the install files needed for the custom system.

After some thought, I decided that with a few easy steps, I could create a re-usable build system:

  1. Set up a local mini-mirror of the OpenBSD install repository ( http://ftp.openbsd.org/pub/OpenBSD/4.4/), which contains the entire original i386/ sub-directory, and the packages/i386/ sub-directory, containing any package files needed for the project (e.g. ntp-4.2.0ap2.tgz).
  2. Create a "fork" of the standard i386/ directory, called i386custom/. This directory contains a sub-directory for each custom, embedded server system (e.g. NTPserver/, DHCPserver/, Test/, etc.). The sub-directory contains the following:
    • 00README.txt - A README describing the build system found in 00build
    • 00build/ - The directory containing the build system
    • INSTALL.custom - Install instructions for the custom system
    • INSTALL.i386 - The standard install instructions
    • MD5 - The MD5 checksum file
    • base44.tgz - The customised system base gzipped tarball
    • bsd - The stock BSD kernel
    • etc44.tgz - The customised system etc gzipped tarball
    • index.txt - The index file listing the installable files
    Originally, the base44.tgz and the etc44.tgz are copies of the stock files, but after the build system is first used, these files become customised.
  3. Create two build scripts in the 00build/ directory:
    1. makerootfs - This script uses the two .tgz files in the directory in the directory above to create a root filesystem containing the system to be installed. This root filesystem is contained in a subdirectory called 'rootfs'. Once this script has been run and rootfs created, the files therein modified as needed. Once the modifications finished, the second build script must be run.
    2. makesetfiles - This script uses the rootfs directory to create two gzipped tarballs and an MD5 file. Once this script has been run, the newly-created .tgz and MD5 files must be copied to the directory above.

Once the build system was in place, I removed all the files therein, with the exception of what I thought would be needed for the barest minimum of a system. Specifically, I only wanted what was needed to run a custom /etc/rc that would initialise my custom, ramdisk-based system with files found on the Compact Flash card, which would be mounted read-only, except for occasional system configuration changes. The idea was that the standard Unix directories such as /bin, /etc, etc., would be created as writeable MFS filesystems (ramdisks) mounted over the flash card's read-only versions, and would be populated from a special directory containing a mirror of the files meant to be available on the final, fully-initialised system. Once up and running, the system would periodically check for any configuration changes made to the ramdisks, and if so, would temporarily remount the flash card as writable, save the changes, then do another remount as read-only again. For obvious reasons, I decided that "/Nonvolatile" would be the perfect name for the special directory. :)

As I began to build and test my new minimalist system, I discovered a problem: some the files in /etc were no longer accessible for configuration after the ramdisk-based /etc/ was mounted. The resolution I came up with was to create a post-install script that I could run from the command-line after the stock OpenBSD install was completed. This script would create an /etc_premount directory, and therein created hard links to the following configuration files in /etc:

boot.conf fstab login.conf rc ttys

This gave me the ability to edit these files using the hard links in /etc_premount, and the ramdisk-based version of /etc had symlinks of these files pointing to ../etc_premount/[filename].

The other problem I ran into was that the MFS mount program's -P option used an external executable (/bin/pax) to do the populating process, which made mounting the ramdisk-based version of /bin fail to be populated. This meant that if I still wanted a ramdisk /bin, I had to find another way to populate it. My first attempt involved ignoring /etc/fstab and hard-coding its info into the rc script. It worked, but it was an awkward, counter-intuitive kludge, and it really annoyed me. It meant that /etc/fstab was really a lie, and that any changes to the mount information involved directly editing the rc script, which was the source of my annoyance. In the process of "scratching the itch" by trying to create a way to use the fstab, I found that some of my fstab processing was going awry, leaving me with an unfinished filesystem that wasn't able to get to a login prompt so I could even investigate. So I came up with a two-part solution: my rc first did an attempt to do the mounts using fstab, then it would verify the mounts. If the verification failed, it would then fall back to the hard-coded version. This then gave me a system I could login to for debugging the problem. After fixing the problem with my fstab-based mounting, I decided to leave the two-step process in place, so as to have a "best-effort" solution.

Another goal for which I came up with an interesting solution, was how to efficiently save config changes. I didn't really want to have to include the 'find' command, but I needed something like it that could recurse through the filesystem for config changes and only make updates for changed files (if any). I decided to add the functionality to the rc script, then wrote a shell wrapper called /usr/sbin/saveconfigs. It included a -t option to allow for a manual test mode.

All in all, I managed to put together a unique embedded OpenBSD solution that is as easy to install as the native installer (because it in fact uses the native installer :). Additionally, I created a build system that allows sysadmins an easy way to modify my custom solution to meet their own specific needs. I hope you'll find it as fun and as useful as I have. You can download my OpenBSD mini-repository and quick-and-dirty embedded build system here: http://www.jrwz.net/pub/OpenBSD/4.4/i386custom/

If you have comments, suggests or requests, I'd be glad to hear from you, you'll find various means for getting in touch on my contact page.

Since doing the original NTP server, I've customised the system for my own home network to include DHCP and internal DNS service. I'd like to make other improvements, but I'd rather do them using a newer version of OpenBSD. For the moment, one specific change comes to mind: I want to move the environment variables for enabling/disabling daemons out of the rc script and into a separate /etc/rc.conf file, to limit the need for editing /etc/rc. And I should probably create some better documentation for all this. For the time being, though, it's "Use the Source, Luke!".

Various Links Associated with Embedded OpenBSD Systems