Before I get started, I want to take a moment to remind readers of an obvious fact which, apparently, some have forgotten:
The year is 2008, not 1989.
FreeBSD was originally considered the "die-hard server operating system", which means it was intended for servers. It was used as such back when the 2.2.x series was available. Datacenters today are usually massive (read: have a very large number of servers), and require the need of 3 simple things, all of which are well-established and common:
For sake of comparison, both Linux and Solaris (sparc and i386) have the above needs addressed, but FreeBSD trails behind. Accomplishing the above on FreeBSD requires the knowledge which I think many administrators lack — and that's not their fault. Things like:
pxeboot(8) really isIf you're not "acceptably" familiar with all of the above, you'll end up smashing your head against your monitor for days before posting to a mailing list — only to hear leaves rustle, or possibly someone saying "me too!" Most administrators do not know how to program, and even if they did (like myself), lack knowledge of the inner-workings of the above. Therefore, we can't realistically expect administrators of systems to provide patches or enhancements to make this process easier (for sake of comparison, it took me 7 hours just to get it all up and working) — we can only expect them to report the problems and hope someone more technical steps up to the plate.
Rant over.
I hope the below documentation helps anyone/everyone who is looking to do a remote installation of present-day FreeBSD 7.0 via PXE using serial console rather than VGA.
These are the things you are absolutely going to need — no questions asked. If you don't have any of these, you should stop here.
cu -l ttyd0 -s 115200
and make sure it works.Also, I only cover how to accomplish this on i386. I'm willing to bet that on amd64 it's identical, but I can't promise that. I did see some -stable posts about how certain pieces of PXE booting weren't working on amd64, but I don't know if those were specific to the OP's system, or actual problems. :-(
| Term | Description |
|---|---|
| 192.168.1.1 | The IP of the DHCP, TFTP, and NFS server. |
| 192.168.1.100 | The IP of the box which we're doing the PXE boot on. |
| newbox.home.lan | FQDN of 192.168.1.100 |
| /usr/local/freebsd7 | Contains contents of the FreeBSD 7.0 "disc1" CD; you will be modifying some files in this directory, so it cannot be a mounted CD (e.g. read-only). |
| boot0 boot1 boot2 loader |
All of these are the different stages of the FreeBSD bootstraps. All are
used when physical media is used for installation (e.g. floppies, CDROM, hard
disk, USB thumb drive, etc.), but aren't used when PXE booting. loader(8)
is technically used, but it's actually called pxeboot(8). See below
for details on that. |
| pxeboot pxeldr |
This is the sole binary that gets fetched via TFTP after the Intel PXE boot ROM gets
an IP via DHCP. The binary itself is actually a modified version of loader(8),
and includes some extra (undocumented) features. |
| mfsroot | This is a UFS filesystem in a file. It contains a "bare bones" root filesystem,
including such things as init(8) and other necessities. It's loaded by
loader(8) (a.k.a. pxeboot(8)) into memory, and then referenced
as a UFS disk known as /dev/md0c. mfsroot is usually found gzip'd. |
I'm familiar with ISC's DHCP server, available as ports/net/isc-dhcp3-server.
The important pieces you need to add to your /usr/local/etc/dhcpd.conf are here:
host newbox.home.lan {
hardware ethernet 00:11:22:33:44:55;
fixed-address newbox.home.lan;
next-server 192.168.1.1;
filename "freebsd7/boot/pxeboot";
option root-path "/usr/local/freebsd7";
}
Since not a single other document on the Web has described what the important options do, I'll describe their purpose:
next-server 192.168.1.1;
This line isn't necessary if the NFS server and the DHCP server are on the same machine. Including it doesn't hurt.
filename "freebsd7/boot/pxeboot";
When a DHCP request is made from the Intel PXE boot ROM on the NIC, there are additional parameters you can tell it besides just an IP number. This argument tells the PXE boot ROM what filename to do a TFTP request for now that it has an IP address. It's the first piece of code which gets run after the initial DHCP IP negotiation is completed.
option root-path "/usr/local/freebsd7";
This is for FreeBSD. It tells pxeboot(8) where the root filesystem
should be mounted from, NFS-wise. Thus, the kernel will do the equivalent of the
following, in an attempt to get a working root filesystem, and then start
init(8):
mount -t nfs 192.168.1.1:/usr/local/freebsd7 /
Uncomment the following line in /etc/inetd.conf, and modify it so that
the -s flag points to /usr/local (that's where tftpd will chroot() before
serving content):
tftp dgram udp wait root /usr/libexec/tftpd tftpd -l -s /usr/local
Add the following to /etc/rc.conf (assuming it's not already there):
inetd_enable="yes"
Now start the inetd service (thus starting tftpd):
/etc/rc.d/inetd start
If you already had inetd running prior to modifying /etc/inetd.conf, all you
need to do is send a SIGHUP to it: killall -HUP inetd
Add the following to /etc/rc.conf, again assuming they're not already there:
rpcbind_enable="yes" mountd_enable="yes" nfs_server_enable="yes"
Now we need to add an NFS export. Edit /etc/exports and add the following
lines to it:
/usr/local/freebsd7 -network 192.168.1 -mask 255.255.255.0
This example assumes your network is 192.168.1.0/24; if it's something else, you'll need to make appropriate changes.
Don't forget to make the exported directory:
# mkdir /usr/local/freebsd7 # chmod 755 /usr/local/freebsd7
Now start all of the above services:
/etc/rc.d/rpcbind start /etc/rc.d/mountd start /etc/rc.d/nfsd start
If these were running beforehand and you simply edited /etc/exports,
all you need to do is send a SIGHUP to mountd(8) to get it to re-read
the exports: killall -HUP mountd.
You should verify the NFS mount which was added is being exported:
# showmount -e Exports list on localhost: /usr/local/freebsd7 192.168.1.0
We're going to need to copy all the contents off the FreeBSD 7.0 "disc1" CD
to /usr/local/freebsd7 before continuing. We can't mount the CD
and use it directly because we're going to modify some of the contents after
the copy. If you have the physical CD, you can mount it in the CDROM drive of
the NFS server. You should know how to do this. :P Otherwise, if you only
have the ISO image, you can do the following:
# mdconfig -a -t vnode -u 3 -f /path/to/7.0-RC1-i386-disc1.iso # mount_cd9660 /dev/md3 /mnt
See the mdconfig(8) man-page for details of what the flags are. Yes, all
the flags are important, and it's VERY important that the number you pass -u
matches the /dev/mdX number.
Now for the data copy. Use one of the following methods; I recommend the rsync method. Also, for the rsync method, note that the trailing slashes are needed.
rsync -avH /mnt/ /usr/local/freebsd7/cp -pR /mnt/* /usr/local/freebsd7tar -C /mnt -pcf - . | tar -C /usr/local/freebsd7 -pxvf -FreeBSD out-of-the-box has a hard-set serial port speed of 9600bps built in
to the boot2 bootstrap. "So I can just pick a higher speed, right?" Nope — if
you try to do so, it'll stay at 9600bps. If you tell boot2 to set the speed to
115200 (e.g. comconsole_speed="115200"), it won't work — you'll
still get 9600bps. The best solution for systems that boot off of a hard
disk is to tell the boot2 bootstrap what to set the serial port speed to.
You end up putting -S115200 in /boot.config and
reboot — no need to rebuild the boot blocks. Nothing else is needed.
But when PXE booting, there's only one piece of the bootstrap used:
pxeboot(8). This means the only solution is to rebuild the boot
blocks with a serial port speed that has the speed you want — in this case,
115200bps. But first, let's clear something up: pxeboot(8) is
actually nothing more than loader(8) with some extra code (see
src/sys/boot/i386/pxeldr.S) in it. loader(8) uses a
library called libi386, which is a library used by the bootstraps only — it's
it's not something you'll find in /usr/lib or /usr/libexec. libi386 is what
actually controls what the serial port speed is set to, thus, anything using
libi386 should have support for higher serial console speeds. The
BOOT_COMCONSOLE_SPEED and BOOT_COMCONSOLE_PORT
variables are used in libi386.
Anyway, on the NFS server, do the following:
# cd /sys/boot # make clean # make BOOT_COMCONSOLE_SPEED=115200
IMPORTANT: DO NOT type "make install" after this step! We don't want to modify the bootstrap on the NFS server itself — we just want a 115200bps-capable pxeboot(8) binary!
There's also a two additional (undocumented) options which are specific to
pxeboot(8) and aren't part of lib386:
BOOT_PXELDR_PROBE_KEYBOARD=1 BOOT_PXELDR_ALWAYS_SERIAL=1
BOOT_PXELDR_PROBE_KEYBOARD is similar to the -P flag documented
in boot(8). It causes pxeboot(8) to look for an
attached PS/2 or AT keyboard — and if there is one, use VGA/keyboard,
otherwise assume serial console. BOOT_PXELDR_ALWAYS_SERIAL tells
pxeboot(8) to always use the serial port, and never do anything
with a locally-attached VGA console or keyboard.
Now that we have a 115200bps-capable pxeboot(8) binary, so let's copy it over what was originally on the CD image:
# cd /sys/boot/i386 # cp pxeldr/pxeboot /usr/local/freebsd7/boot
At this point, a PXE booting client should now have the capability to set the serial port speed to 115200bps. And don't forget to clean up the /sys/boot directory too! We don't want cruft laying around in there.
# cd /sys/boot # make clean
Now we have to update /usr/local/freebsd7/boot/loader.conf to
tell it to utilise serial console, and more importantly, point to where the
root filesystem (for booting) will be. The lines in bold are the ones
you want to add; the others shown should already be there (don't mess with
them!):
mfsroot_load="YES" mfsroot_type="mfs_root" mfsroot_name="/boot/mfsroot" comconsole_speed="115200" console="comconsole" vfs.root.mountfrom="ufs:/dev/md0c"
The serial console lines should be obvious.
vfs.root.mountfrom tells the kernel that after it's done booting, to try to
load the root filesystem from ufs:/dev/md0c before starting
init(8). Without this line, the system will attempt to mount the
root filesystem via NFS, which is not what we want. If you want to know more
about this part of the booting procedure, you should look at the related Forth
files, as well as loader.rc in the same directory.
Also, some clarification (since other documentation I've read seems to indicate that you want /dev/md0 and not /dev/md0c): /dev/md0c is what points to the beginning of the filesystem (block 0). The FreeBSD Handbook goes over such filesystem lettering semantics. So trust me, it's /dev/md0c. :-)
The mfs_root loading mechanism is also programmed to support gzip decompression automatically, so it will also look for the /boot/mfsroot.gz file — which is what comes on the FreeBSD 7.0 "disc1" CD. However.....
I've confirmed that there is in fact a bug in the mfs_root loading mechanism. It's likely been there for a very long time, since a couple other guides state that after rebuilding their mfsroot, if they re-gzip'd the mfsroot image, their machines would reboot.
On our servers, something bizarre happens: the kernel appears to get reloaded, all of the environment variables are lost (which means serial console is lost), and then there's some nastiness on the console about how it can't find "kernel", and some weird Device ID 0xffffffff error. I'd have to take a photo of the monitor to provide additional details, but regardless, this problem is easily reproducible. I filed a PR for this problem, kern/120127, but as of this writing not a single developer has bothered to look into it. I'm willing to bet this bug has bitten the EtherBoot folks as well.
The workaround is simple: remove the gzip compression on the mfsroot.gz image, and everything should work:
# gzip -d /usr/local/freebsd7/boot/mfsroot
Alright, it's time to give it a try and see if it breaks! Be patient, since most of the TFTP loading is slow (it's UDP-based, and uses small UDP packets). If all goes well, you should see something like this appear:
/boot/kernel/kernel text=0x71b90c data=0xadb60+0x5ac20 syms=[0x4+0x70fc0+0x4+0x90e69] ÚÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿ ³ ³ ³ ³ ______ ³ ³ | ____| __ ___ ___ ³ Welcome to FreeBSD! ³ | |__ | '__/ _ \/ _ \ ³ ³ | __|| | | __/ __/ ³ ³ | | | | | | | ³ 1. Boot FreeBSD [default] ³ |_| |_| \___|\___| ³ 2. Boot FreeBSD with ACPI disabled ³ ____ _____ _____ ³ 3. Boot FreeBSD in Safe Mode ³ | _ \ / ____| __ \ ³ 4. Boot FreeBSD in single user mode ³ | |_) | (___ | | | | ³ 5. Boot FreeBSD with verbose logging ³ | _ < \___ \| | | | ³ 6. Escape to loader prompt ³ | |_) |____) | |__| | ³ 7. Reboot ³ | | | | ³ ³ |____/|_____/|_____/ ³ ³ ³ ³ ³ ³ ³ Select option, [Enter] for default ³ ³ or [Space] to pause timer 10 ³ ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ
Hit Enter here, and let things boot normally. If all goes well, you should begin to see the acpi.ko module load, and then standard kernel output. Near the end, you should be shown something similar to the following:
md0: Preloaded image </boot/mfsroot> 4423680 bytes at 0xc0d26ed4 Trying to mount root from ufs:/dev/md0c /stand/sysinstall running as init on serial console These are the predefined terminal types available to sysinstall when running stand-alone. Please choose the closest match for your particular terminal. 1 ...................... Standard ANSI terminal. 2 ...................... VT100 or compatible terminal. 3 ...................... FreeBSD system console (color). 4 ...................... FreeBSD system console (monochrome). 5 ...................... xterm terminal emulator.
Pick a terminal type, and the rest should be obvious: the standard sysinstall(8) selection menu, and all that jazz.
When prompted what installation medium you want to install from, you
should pick Install over NFS and enter the name of the share
path manually, e.g.: 192.168.1.1:/usr/local/freebsd7
From there, everything should "just work"!