Installing FreeBSD 7.0 via serial console and PXE

— by Jeremy Chadwick <koitsu@freebsd.org>

Sections

  1. Preface
  2. Requirements
  3. Common Paths/Terms Used
  4. Step 1: Configuring DHCP
  5. Step 2: Configuring TFTP
  6. Step 3: Configuring NFS
  7. Step 4: Copying the FreeBSD CD to an exported NFS mount
  8. Step 5: Rebuilding pxeboot(8)
  9. Step 6: Configuring loader.conf
  10. Step 7: Workaround for a bug in mfs_root
  11. Step 8: Attempting your first PXE boot
  12. Step 9: Choosing an installation medium

Preface

Before I get started, I want to take a moment to remind readers of an obvious fact which, apparently, some have forgotten:

The year is 2008, not 1989.

FreeBSD was originally considered the "die-hard server operating system", which means it was intended for servers. It was used as such back when the 2.2.x series was available. Datacenters today are usually massive (read: have a very large number of servers), and require the need of 3 simple things, all of which are well-established and common:

  1. Installation of an entire OS without physical access to the box
  2. Heavy reliance on the network for installation (e.g. network install)
  3. Use of high speed serial console (9600+bps) from start to finish to assist or automate said installation.

For sake of comparison, both Linux and Solaris (sparc and i386) have the above needs addressed, but FreeBSD trails behind. Accomplishing the above on FreeBSD requires the knowledge which I think many administrators lack — and that's not their fault. Things like:

  1. Intricate knowledge of how the FreeBSD multi-stage (boot0/boot1/boot2/loader) bootstrap behaves, and what pxeboot(8) really is
  2. Rebuilding above bootstraps with custom options during compile
  3. Implementation of said bootstraps into PXE-based installation environments
  4. Knowledge of how TFTP and DHCP work, and how to debug them if they break,
  5. Intricate knowledge of configuring a DHCP server (common question: "what's the 'next-server' and 'option root-path' stuff? Is it needed? Why?")
  6. Familiarity with NFS (configuration and security awareness)
  7. Caveats/bugs with all of the above, and more. Examples: how out-of-the-box FreeBSD sets the maximum serial port speed to 9600bps, not being able to do a complete 100% TFTP-based (e.g. no NFS) install, security issues surrounding use of rpcbind/mountd/nfsd (daemons binding to INADDR_ANY despite use of -h, using arbitrary port numbers which make it difficult to firewall off, etc.) since NFS is then mandatory, undocumented pxeboot(8) options, and a confirmed gzip'd mfs_root loader bug.

If you're not "acceptably" familiar with all of the above, you'll end up smashing your head against your monitor for days before posting to a mailing list — only to hear leaves rustle, or possibly someone saying "me too!" Most administrators do not know how to program, and even if they did (like myself), lack knowledge of the inner-workings of the above. Therefore, we can't realistically expect administrators of systems to provide patches or enhancements to make this process easier (for sake of comparison, it took me 7 hours just to get it all up and working) — we can only expect them to report the problems and hope someone more technical steps up to the plate.

Rant over.

I hope the below documentation helps anyone/everyone who is looking to do a remote installation of present-day FreeBSD 7.0 via PXE using serial console rather than VGA.

Requirements

These are the things you are absolutely going to need — no questions asked. If you don't have any of these, you should stop here.

Also, I only cover how to accomplish this on i386. I'm willing to bet that on amd64 it's identical, but I can't promise that. I did see some -stable posts about how certain pieces of PXE booting weren't working on amd64, but I don't know if those were specific to the OP's system, or actual problems. :-(

Common Paths/Terms Used

TermDescription
192.168.1.1 The IP of the DHCP, TFTP, and NFS server.
192.168.1.100 The IP of the box which we're doing the PXE boot on.
newbox.home.lan FQDN of 192.168.1.100
/usr/local/freebsd7 Contains contents of the FreeBSD 7.0 "disc1" CD; you will be modifying some files in this directory, so it cannot be a mounted CD (e.g. read-only).
boot0
boot1
boot2
loader
All of these are the different stages of the FreeBSD bootstraps. All are used when physical media is used for installation (e.g. floppies, CDROM, hard disk, USB thumb drive, etc.), but aren't used when PXE booting. loader(8) is technically used, but it's actually called pxeboot(8). See below for details on that.
pxeboot
pxeldr
This is the sole binary that gets fetched via TFTP after the Intel PXE boot ROM gets an IP via DHCP. The binary itself is actually a modified version of loader(8), and includes some extra (undocumented) features.
mfsroot This is a UFS filesystem in a file. It contains a "bare bones" root filesystem, including such things as init(8) and other necessities. It's loaded by loader(8) (a.k.a. pxeboot(8)) into memory, and then referenced as a UFS disk known as /dev/md0c. mfsroot is usually found gzip'd.

Step 1: Configuring DHCP

I'm familiar with ISC's DHCP server, available as ports/net/isc-dhcp3-server. The important pieces you need to add to your /usr/local/etc/dhcpd.conf are here:

host newbox.home.lan {
	  hardware ethernet 00:11:22:33:44:55;
	  fixed-address newbox.home.lan;
	  next-server 192.168.1.1;
	  filename "freebsd7/boot/pxeboot";
	  option root-path "/usr/local/freebsd7";
}

Since not a single other document on the Web has described what the important options do, I'll describe their purpose:

next-server 192.168.1.1;

This line isn't necessary if the NFS server and the DHCP server are on the same machine. Including it doesn't hurt.

filename "freebsd7/boot/pxeboot";

When a DHCP request is made from the Intel PXE boot ROM on the NIC, there are additional parameters you can tell it besides just an IP number. This argument tells the PXE boot ROM what filename to do a TFTP request for now that it has an IP address. It's the first piece of code which gets run after the initial DHCP IP negotiation is completed.

option root-path "/usr/local/freebsd7";

This is for FreeBSD. It tells pxeboot(8) where the root filesystem should be mounted from, NFS-wise. Thus, the kernel will do the equivalent of the following, in an attempt to get a working root filesystem, and then start init(8):

mount -t nfs 192.168.1.1:/usr/local/freebsd7 /

Step 2: Configuring TFTP

Uncomment the following line in /etc/inetd.conf, and modify it so that the -s flag points to /usr/local (that's where tftpd will chroot() before serving content):

tftp    dgram   udp     wait    root    /usr/libexec/tftpd      tftpd -l -s /usr/local

Add the following to /etc/rc.conf (assuming it's not already there):

inetd_enable="yes"

Now start the inetd service (thus starting tftpd):

/etc/rc.d/inetd start

If you already had inetd running prior to modifying /etc/inetd.conf, all you need to do is send a SIGHUP to it: killall -HUP inetd

Step 3: Configuring NFS

Add the following to /etc/rc.conf, again assuming they're not already there:

rpcbind_enable="yes"
mountd_enable="yes"
nfs_server_enable="yes"

Now we need to add an NFS export. Edit /etc/exports and add the following lines to it:

/usr/local/freebsd7	-network 192.168.1 -mask 255.255.255.0

This example assumes your network is 192.168.1.0/24; if it's something else, you'll need to make appropriate changes.

Don't forget to make the exported directory:

# mkdir /usr/local/freebsd7
# chmod 755 /usr/local/freebsd7

Now start all of the above services:

/etc/rc.d/rpcbind start
/etc/rc.d/mountd start
/etc/rc.d/nfsd start

If these were running beforehand and you simply edited /etc/exports, all you need to do is send a SIGHUP to mountd(8) to get it to re-read the exports: killall -HUP mountd.

You should verify the NFS mount which was added is being exported:

# showmount -e
Exports list on localhost:
/usr/local/freebsd7		192.168.1.0

Step 4: Copying the FreeBSD CD to an exported NFS mount

We're going to need to copy all the contents off the FreeBSD 7.0 "disc1" CD to /usr/local/freebsd7 before continuing. We can't mount the CD and use it directly because we're going to modify some of the contents after the copy. If you have the physical CD, you can mount it in the CDROM drive of the NFS server. You should know how to do this. :P Otherwise, if you only have the ISO image, you can do the following:

# mdconfig -a -t vnode -u 3 -f /path/to/7.0-RC1-i386-disc1.iso
# mount_cd9660 /dev/md3 /mnt

See the mdconfig(8) man-page for details of what the flags are. Yes, all the flags are important, and it's VERY important that the number you pass -u matches the /dev/mdX number.

Now for the data copy. Use one of the following methods; I recommend the rsync method. Also, for the rsync method, note that the trailing slashes are needed.

Step 5: Rebuilding pxeboot(8)

FreeBSD out-of-the-box has a hard-set serial port speed of 9600bps built in to the boot2 bootstrap. "So I can just pick a higher speed, right?" Nope — if you try to do so, it'll stay at 9600bps. If you tell boot2 to set the speed to 115200 (e.g. comconsole_speed="115200"), it won't work — you'll still get 9600bps. The best solution for systems that boot off of a hard disk is to tell the boot2 bootstrap what to set the serial port speed to. You end up putting -S115200 in /boot.config and reboot — no need to rebuild the boot blocks. Nothing else is needed.

But when PXE booting, there's only one piece of the bootstrap used: pxeboot(8). This means the only solution is to rebuild the boot blocks with a serial port speed that has the speed you want — in this case, 115200bps. But first, let's clear something up: pxeboot(8) is actually nothing more than loader(8) with some extra code (see src/sys/boot/i386/pxeldr.S) in it. loader(8) uses a library called libi386, which is a library used by the bootstraps only — it's it's not something you'll find in /usr/lib or /usr/libexec. libi386 is what actually controls what the serial port speed is set to, thus, anything using libi386 should have support for higher serial console speeds. The BOOT_COMCONSOLE_SPEED and BOOT_COMCONSOLE_PORT variables are used in libi386.

Anyway, on the NFS server, do the following:

# cd /sys/boot
# make clean
# make BOOT_COMCONSOLE_SPEED=115200

IMPORTANT: DO NOT type "make install" after this step! We don't want to modify the bootstrap on the NFS server itself — we just want a 115200bps-capable pxeboot(8) binary!

There's also a two additional (undocumented) options which are specific to pxeboot(8) and aren't part of lib386:

BOOT_PXELDR_PROBE_KEYBOARD=1
BOOT_PXELDR_ALWAYS_SERIAL=1

BOOT_PXELDR_PROBE_KEYBOARD is similar to the -P flag documented in boot(8). It causes pxeboot(8) to look for an attached PS/2 or AT keyboard — and if there is one, use VGA/keyboard, otherwise assume serial console. BOOT_PXELDR_ALWAYS_SERIAL tells pxeboot(8) to always use the serial port, and never do anything with a locally-attached VGA console or keyboard.

Now that we have a 115200bps-capable pxeboot(8) binary, so let's copy it over what was originally on the CD image:

# cd /sys/boot/i386
# cp pxeldr/pxeboot /usr/local/freebsd7/boot

At this point, a PXE booting client should now have the capability to set the serial port speed to 115200bps. And don't forget to clean up the /sys/boot directory too! We don't want cruft laying around in there.

# cd /sys/boot
# make clean

Step 6: Configuring loader.conf

Now we have to update /usr/local/freebsd7/boot/loader.conf to tell it to utilise serial console, and more importantly, point to where the root filesystem (for booting) will be. The lines in bold are the ones you want to add; the others shown should already be there (don't mess with them!):

mfsroot_load="YES"
mfsroot_type="mfs_root"
mfsroot_name="/boot/mfsroot"
comconsole_speed="115200"
console="comconsole"
vfs.root.mountfrom="ufs:/dev/md0c"

The serial console lines should be obvious.

vfs.root.mountfrom tells the kernel that after it's done booting, to try to load the root filesystem from ufs:/dev/md0c before starting init(8). Without this line, the system will attempt to mount the root filesystem via NFS, which is not what we want. If you want to know more about this part of the booting procedure, you should look at the related Forth files, as well as loader.rc in the same directory.

Also, some clarification (since other documentation I've read seems to indicate that you want /dev/md0 and not /dev/md0c): /dev/md0c is what points to the beginning of the filesystem (block 0). The FreeBSD Handbook goes over such filesystem lettering semantics. So trust me, it's /dev/md0c. :-)

The mfs_root loading mechanism is also programmed to support gzip decompression automatically, so it will also look for the /boot/mfsroot.gz file — which is what comes on the FreeBSD 7.0 "disc1" CD. However.....

Step 7: Workaround for a bug in mfs_root

I've confirmed that there is in fact a bug in the mfs_root loading mechanism. It's likely been there for a very long time, since a couple other guides state that after rebuilding their mfsroot, if they re-gzip'd the mfsroot image, their machines would reboot.

On our servers, something bizarre happens: the kernel appears to get reloaded, all of the environment variables are lost (which means serial console is lost), and then there's some nastiness on the console about how it can't find "kernel", and some weird Device ID 0xffffffff error. I'd have to take a photo of the monitor to provide additional details, but regardless, this problem is easily reproducible. I filed a PR for this problem, kern/120127, but as of this writing not a single developer has bothered to look into it. I'm willing to bet this bug has bitten the EtherBoot folks as well.

The workaround is simple: remove the gzip compression on the mfsroot.gz image, and everything should work:

# gzip -d /usr/local/freebsd7/boot/mfsroot

Step 8: Attempting your first PXE boot

Alright, it's time to give it a try and see if it breaks! Be patient, since most of the TFTP loading is slow (it's UDP-based, and uses small UDP packets). If all goes well, you should see something like this appear:

/boot/kernel/kernel text=0x71b90c data=0xadb60+0x5ac20 syms=[0x4+0x70fc0+0x4+0x90e69]

 ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿
 ³                                         ³
 ³                                         ³      ______
 ³                                         ³     |  ____| __ ___  ___
 ³          Welcome to FreeBSD!            ³     | |__ | '__/ _ \/ _ \
 ³                                         ³     |  __|| | |  __/  __/
 ³                                         ³     | |   | | |    |    |
 ³  1. Boot FreeBSD [default]              ³     |_|   |_|  \___|\___|
 ³  2. Boot FreeBSD with ACPI disabled     ³      ____   _____ _____
 ³  3. Boot FreeBSD in Safe Mode           ³     |  _ \ / ____|  __ \
 ³  4. Boot FreeBSD in single user mode    ³     | |_) | (___ | |  | |
 ³  5. Boot FreeBSD with verbose logging   ³     |  _ < \___ \| |  | |
 ³  6. Escape to loader prompt             ³     | |_) |____) | |__| |
 ³  7. Reboot                              ³     |     |      |      |
 ³                                         ³     |____/|_____/|_____/
 ³                                         ³
 ³                                         ³
 ³                                         ³
 ³  Select option, [Enter] for default     ³
 ³  or [Space] to pause timer  10          ³
 ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ

Hit Enter here, and let things boot normally. If all goes well, you should begin to see the acpi.ko module load, and then standard kernel output. Near the end, you should be shown something similar to the following:

md0: Preloaded image </boot/mfsroot> 4423680 bytes at 0xc0d26ed4
Trying to mount root from ufs:/dev/md0c
/stand/sysinstall running as init on serial console

These are the predefined terminal types available to
sysinstall when running stand-alone.  Please choose the
closest match for your particular terminal.

1 ...................... Standard ANSI terminal.
2 ...................... VT100 or compatible terminal.
3 ...................... FreeBSD system console (color).
4 ...................... FreeBSD system console (monochrome).

5 ...................... xterm terminal emulator.

Pick a terminal type, and the rest should be obvious: the standard sysinstall(8) selection menu, and all that jazz.

Step 9: Choosing an installation medium

When prompted what installation medium you want to install from, you should pick Install over NFS and enter the name of the share path manually, e.g.: 192.168.1.1:/usr/local/freebsd7

From there, everything should "just work"!