[[TOC]] = Welcome to 'building the !WirelessLeiden NanoBSD image' = As an introduction you may wish to check out [http://www.freebsd.org/doc/en/articles/nanobsd/index.html FreeBSD default method] for installing [http://www.freebsd.org FreeBSD] on embedded devices, like [http://www.soekris.com Soekris], [http://www.pcengines.ch/alix.htm Alix] and others. The manual described below is (loosely) based on the [http://svn.wirelessleiden.nl/svn/projects/iris/doc/afstudeerverslag/AfstudeerVerslag.pdf basic concept plan of Tim Baas], but also consists of many hints by [http://www.freebsd.org/doc/en/articles/nanobsd/index.html official 'Introduction to NanoBSD' manual] and other stuff found on the net ;-). The advantages of using Nanobsd have been nicely summarized recently on the [http://blog.pfsense.org/?tag=nanobsd pfsense website]. = 'Have to remember' links = * [http://people.freebsd.org/~phk/nanobsd/soekris_4x26/make.soekris_4x26.conf] * Active work in progress node images at [http://webfolder.wirelessleiden.nl/rick/nodefactory/] * [http://forums.freebsd.org/showthread.php?t=2198 Getting Xircom pcmcia network adapter to work] * [http://lists.freebsd.org/pipermail/freebsd-usb/2009-March/006503.html How to add a USB vendor] * [https://projects.math.leidenuniv.nl/trac/howto/wiki/FreeBSD_install LIACS wiki] = Procedure = Please note that you can find the supporting files in the [http://svn.wirelessleiden.nl/svn/code svn/code subdirectory]. There are different versions available for node- and gateway (proxy) configurations and also 'trunk' (operational) and 'branches' (development) versions. Also keep a close look at the ''INFO'' ''WARNING'' ''ERR'' directives as they guide you through some common pitfalls. This guide was originally written for FreeBSD 7.2, this page is still available [wiki:FreeBSD7.2_Guide]. As there were a lot of (positive) changes in the wireless stack, we moved to FreeBSD 8.2 which was implemented on all nodes in the course of 2011 and to 9.0 beginning of 2012. We now use exclusively a '''FreeBSD 9 version'''. Code can be found in svn: http://svn.wirelessleiden.nl/svn/code/hybrid/branches/releng-9/ . ''Note'': there is an issue with usb-lan adapters in 9.1-release (9.0-release is OK). == Changelog == For every new release of FreeBSD we also tend to release a new version a few weeks after it, containing all our latest fixes. But bear in mind that we also do bugfixes in the meanwhile and that the changelog is not always kept super up2date, so keep an eye on ``svn log'' ;-). 11.2-RELEASE: * see [wiki:FreeBSD11_Guide] 10.2-RELEASE (obsolete): * see [wiki:FreeBSD10_Guide] 9.1-RELEASE: * Merged Proxy and Node configurations in one configuration, such that we need fewer boards. * Using ISC-DHCPD instead of DNSMASQ, because we have more memory available now. * Using BIND instead of MARADNS, maradns support for slaves was hard to maintain. * Made captive portal more robust and user friendly for mobile devices. * Implemented incoming RDR support, to allow accessing some WL devices from the internet. 9.0-RELEASE: * use of pf instead of ipfw - because of rich feature set. * Better design of captive portal, faster response. * More monitoring support, also monitoring of nanostations * Better maintenance tools and overviews (dynamic motd). * gformat support for genesis nodes database. * Overhaul of the DNS infrastructure master/slave on the nodes with named. * Better support for PEN (more than three proxies) and better sorting. * Ensure that the fastest nameservers are automagically ranked first in /etc/resolv.conf to shorten DNS response time. The procedure to setup and configure a 9.1 nodefactory is similar to the 9.0 and 8.2 procedure described below. == A. Setup a FreeBSD host == ---- Warning: 1) Make sure ''/usr'' is '''at least 5GB''' in size, as building images requires quite some space. 2) Make sure you install the 32bit '''i386''' release of FreeBSD also when your system does support amd64, as cross compiling can give some nasty surprises. Tip: use a separate hard disk, mounted on /usr/obj to speed up the compilation process. ---- Get yourself a fresh i386 freebsd host with ports and subversion installed as follows: === A.1. Run the basic CD installer === The procedure below has been tested with 9.0-RELEASE (standard developer install - no ports -, e.g. with default partitioning will fit our needs). Installing FreeBSD is outside the scope of this document, take a look into [http://www.freebsd.org/doc/en/books/handbook/install.html the FreeBSD handbook Chapter 2 Installing FreeBSD] if you do not know the details. Please do mind that all commands below need to be executed as root, because of the many mounts and unmounts done in various phases. It maybe convenient to permit ssh root login (''modify /etc/ssh/sshd_config'' accordingly). Internet connection is required. Set correct date/time, e.g: build# ntpdate -s 0.nl.pool.ntp.org (You may also wish to add ''ntpd_enable="YES"'' to /etc/rc.conf) === A.2 fetch ports === build# portsnap fetch extract === A.3 set some useful variables === Alter the shell configuration file:/root/.cshrc: Ensure ftp is set to passive mode, to avoid potential firewall issues: build# echo 'setenv FTP_PASSIVE_MODE YES' >> /root/.cshrc Set a default password for the images that you will produce build# echo 'setenv CFG_ROOT_PASSWORD DefaultPassword12!' >> /root/.cshrc Define your nanobsd (svn) working directory: NOTE: All commands at later stages will refer to this so you better get it right! build# echo 'setenv R /root/nanobsd' >> /root/.cshrc Next load your file (or login again): build# source /root/.cshrc === A.4. install subversion and ocaml === Note: Install all packages via ports to prevent issues later on. build# make -C /usr/ports/devel/subversion install clean BATCH=yes to compile lvrouted you need ocaml support, without X11, without TOOLBOX: build# make -C /usr/ports/devel/ocaml-findlib install clean WITHOUT="TOOLBOX X11 TK" BATCH=YES build# make -C /usr/ports/ports-mgmt/pkg install clean BATCH=yes build# make -C /usr/ports/dns/bind-tools/ install clean BATCH=yes #(required for nagios-plugins in FreeBSD10) === A.5. OPTIONAL, every developer has his own preferences, these are mine === build# make -C /usr/ports/editors/vim-lite install clean BATCH=yes build# make -C /usr/ports/security/sudo install clean BATCH=yes build# make -C /usr/ports/sysutils/screen install clean BATCH=yes === A.6. get latest sources === build# svn co svn://svn.freebsd.org/base/releng/9.0 /usr/src or build# svn co svn://svn.freebsd.org/base/releng/10.0 /usr/src or build# svn co svn://svn.freebsd.org/base/stable/10.0 /usr/src or use mirror svn0.eu.FreeBSD.org == B. Build environment == === B.1 Download the environment from the Wireless Leiden svn repository === {{{ $ svn checkout \ https://svn.wirelessleiden.nl/svn/code/hybrid/branches/releng-9/nanobsd $R (or https://svn.wirelessleiden.nl/svn/code/hybrid/branches/releng-10/nanobsd/ $R ) (what about https://svn.wirelessleiden.nl/svn/code/hybrid/branches/releng-9.2/nanobsd $R) $ cd $R }}} Note: if svn is not found: svn is in /usr/local/bin, alternatively log out and in, or use {{{rehash}}} in a {{{csh}}} shell to make it available. === B.2. Compile all required packages using === {{{ $ $R/tools/package-build.sh }}} === B.3. Set your favorite root password to be used in the image === Note: you can skip this step if you are satisfied with the default password set in step A.3 above. {{{ $ setenv CFG_ROOT_PASSWORD `dd if=/dev/random bs=10k count=10 |\ tr -cd '[a-zA-Z0-9]' | cut -c -15` $ echo $CFG_ROOT_PASSWORD }}} If you like a simple password, substitute the {{{ `dd if=/dev/random bs=10k count=10 | tr -cd '[a-zA-Z0-9]' | cut -c -15` }}} with your password. === B.4 Build nanobsd (make sure to prepare some coffee;-) ) === {{{ $ sh /usr/src/tools/tools/nanobsd/nanobsd.sh -c $R/cfg/nanobsd.wleiden }}} Note 1: Take a coffee of go for a hike, this normally takes 2 - 8 hours depending on the machine configuration. If you like to save some power use the script provided by Rick ([http://rickvanderzwet.nl/svn/personal/misc/power-saver]). This script is mainly used on a home server for building FreeBSD world and kernels. As soon it is done it can shutdown if not being used anymore. The system has Wake-On-Lan support and can thus be activated again from remote. Note 2: even this little script got flags, check the output of {{{sh /usr/src/tools/tools/nanobsd/nanobsd.sh -h}}} {{{ -b suppress builds (both kernel and world) -k suppress buildkernel -w suppress buildworld -c specify config file ... }}} Note 3: The geometry of the cf card is defined in the nanobsd configuration file $R/cfg/nanobsd.wleiden The 'default' values are for a Peak 1 GB card. They also work on an Alix2D3 board with a 1 GB PCEngines 'blanc' cf card and 1 or 2 GB Transcend cf cards although those geometries are different. There are issues with Soekris boards, depending on BIOS version. == C. Fetch node configuration onto image, write to CF disk or remotely update == C.1. Fetch configuration {{{ $ $R/tools/image config for }}} The script is connecting to the [http://svn.wirelessleiden.nl/svn/node-config/genesis/nodes/ Wireless Leiden 'genesis' database]. First make sure that the configuration file is up to date by clicking the 'update' button on [http://sunfire.wirelessleiden.nl/wleiden/config/ http://sunfire.wirelessleiden.nl/wleiden/config/] Pick your situation! Note: proxy configurations are [http://sunfire.wirelessleiden.nl/wleiden/config/ here]. You can inspect the image bij mounting as memory disk: {{{ mdconfig -a -t vnode -f /usr/obj/nanobsd.wleiden-hybrid/_.disk.full mount /dev/md0s1a /mnt ls /mnt umount /mnt mount /dev/md0s3 /mnt ls /mnt umount /mnt mdconfig -d -u 0 }}} Editing can also be done using the image script: {{{ $R/tools/image edit }}} C.2. Write the correct image to CF (media based on SLC and not MLC flash seem to perform much better). ---- ''NOTE:'' '''''_.disk.full''''' is required for '''new CF cards''' as it contains two base systems and one configuration. [[BR]] '''''_.disk.image''''' on the other hand can be used to update an '''existing CF card''' ---- a. '''New image''': Put full image on compact flash disk (attach a card reader/writer with a CF disk of minimum 1 GB) Use the script {{{ $R/tools/write-image.sh }}} or the command line: {{{ $ dd bs=64k if=/usr/obj/nanobsd.wleiden-hybrid/_.disk.full of=/dev/da0 }}} (assuming {{{/dev/da0}}} is your compact flash entry; this takes about 15 minutes, check progress by typing Ctrl-T; you may wish to check afterwards with fdisk whether there are three partitions on the disk, you can also mount /dev/da0s1a and /dev/da0s3 and check the configuration) a. '''Existing image''': Put partial image on slice (slice 2) (attach a card reader/writer with the CF disk containing the existing image) {{{ $ dd bs=64k if=/usr/obj/nanobsd.wleiden-hybrid/_.disk.image of=/dev/da0s2 }}} a. '''Existing image remote update''' (slice 2) (network connection to machine 172.x.y.z required) For remotely updating an existing configuration use: the image script {{{ $R/tools/image deploy on [and reboot] }}} or manually: {{{ dd if=/usr/obj/nanobsd.wleiden-hybrid/_.disk.image bs=10k | ssh root@172.x.y.z /tools/updatep2 }}} You may adjust the block size to make the transfer more efficient (additionally ssh -C is possible) and you can use Ctrl-T to check progress. '''WARNING''': in case of a remote update from 7.2 to 8.0 you must scp the new rc.conf.local to the node before reboot because the interface configuration statements are quite different. If you do not adjust the configuration before reboot the interfaces will not be configured after reboot and you will not be able to reach the node!(see F.4 below) == D. Check the cf card and apply last minute changes == D.1. Check cf card While the cf card is still in your flash card reader you can check whether the image has been written correctly. Check whether you can mount the partitions, e.g. {{{ mount /dev/da0s1a /mnt ls /mnt umount /mnt }}} and the configuration partition: {{{ mount /dev/da0s3 /mnt ls /mnt }}} If you cannot mount the partitions, take a fresh cf card and start again with writing the image. D.2 adjust lvrouted conf If the node has a /28 subnet that should be routable modify the /conf/base/etc/rc.conf file: {{{ lvrouted_flags="...... -m 28 " }}} (for a /24 subnet use -m 24) Note. For an 'iLeiden'-type node you have to add also the -z flag (pointing to iLeiden internet gateway ip-addresses, separated by comma's and no spaces) here. This is now automagically done in the config step. D.3 adjust memory disk sizes For non-standard hard ware you may change the memory disk sizes by modifying the rc.conf.local configuration file. /dev/md0 /etc /dev/md1 /var /dev/md2 /tmp For instance a Soekris4521 has only 64 MB RAM, add for instance the following values {{{ varsize="2m" tmpsize="2m" }}} D.4 *** now obsolete **** Customized captive portal page can be added as /cfg/local/captive/include.htm The default captive portal page is defined at the end of /usr/local/www/wlportal/index.cgi == E. All done! Load the machine == Place the new CF disk in the machine and boot it up (existing nodes: just reboot), you should be good to go! Especially first time booting can take a long time, depending also on hardware. In a Soekris4521 this takes about ten minutes. You can follow the boot process using e.g. minicom or tip, via a serial connection (communication parameters: 9600 8N1). Note 1. For subsequent nodes you can skip the build and go directly for step C, or take a look at development if you have to rebuild the image (after small changes). Note 2. Check your bios version in case of booting problems (note: default communication setting for alix is 38400 8N1, bios should be 0.99h) == F. Applying Updates == For small changes there is a shortcut in the build (to save you some coffee ;-)). Pick the situation applicable 1. First lets pump its source to the latest version {{{ $ svn up /root/nanobsd }}} 2. Set your favorite root password to be used in the image - if different from the password defined in your setup file (see A.4 above): {{{ $ setenv CFG_ROOT_PASSWORD `dd if=/dev/random bs=10k count=10 | & tr -cd '[a-zA-Z0-9]' | cut -c -15`; echo $CFG_ROOT_PASSWORD }}} If you like a simple password, substitute the {{{ `dd if=/dev/random bs=10k count=10 | & tr -cd '[a-zA-Z0-9]' | cut -c -15` }}} with your password. 3. Next step depends on the kind of changes: a. Only changes in kernel options - `kernel.wleiden` altered - ETA 0:45 {{{ $ sh /usr/src/tools/tools/nanobsd/nanobsd.sh -w -c $R/cfg/nanobsd.wleiden }}} a. Only changes in world options - `CONF_{BUILD,COMMON,INSTALL}` in `nanobsd.wleiden` altered - ETA 2:00 - 6:00 {{{ $ sh /usr/src/tools/tools/nanobsd/nanobsd.sh -k -c $R/cfg/nanobsd.wleiden }}} a. _NO_ changes in build options - any other file altered - ETA 0:10 {{{ $ sh /usr/src/tools/tools/nanobsd/nanobsd.sh -b -c $R/cfg/nanobsd.wleiden }}} a. ONLY different configuration {{{ $ $R/tools/image config [for node] }}} 4. Upgrading from 7.2 to 8.2 is tricky because the interface configuration statements have changed. You have to update the config file on slice 3 as well in this case. Mount the image as memory disk: {{{ mdconfig -a -t vnode -f /usr/obj/nanobsd.wleiden/_.disk.full mount /dev/md0s1a /mnt }}} and apply changes to lvrouted flags in rc.conf if needed, see D.2 above {{{ umount /mnt }}} Mount the configuration slice: {{{ mount /dev/md0s3 /mnt }}} scp the rc.conf.local file to the /cfg directory on the node running the old 7.2 configuration {{{ umount /mnt mdconfig -d -u 0 }}} 5. Partition switch bug in FreeBSD8.0 There appears to be a bug in FreeBSD8.0 upgrade procedure. A work around for the partition switch problem in the upgrade procedure that works for Wireless Leiden: * reboot ( unclear why this is needed, but otherwise it doesnot work, i.e. no switch of the active partition) * updatep2 (or p1) * log in, check whether fdisk has activated p2 (or p1). * If positive go to runlevel 6 (init 6), log out. * After a couple of minutes you can log in, welcome on the new slice! Note we have recently modified the /tools/update scripts to use gpart instead of fdisk and boot0cfg, see the /tools/updatep scripts for details. This solved the problem. = G, Development = == Handy tools == Some handy tools are available in the /tools directory, such as * /tools/wl-version : to get information on the configuration that is running * /tools/image-build.sh for building images (replaces long command lines and prevents unintentional, time-consuming rebuilds; do not forget to '''setenv the root password'''!) == Further research == * stumber `/usr/src/tools/tools/net80211/stumber` == Background == * Packages needed are to be found in the nanobsd/tools/package-build.sh script i.e. {{{ $PORTSDIR/dns/dnsmasq $PORTSDIR/www/py-cherrypy $PORTSDIR/www/tinyproxy $PORTSDIR/editors/vim-lite $PORTSDIR/net-mgmt/net-snmp $PORTSDIR/net-mgmt/nrpe2 $PORTSDIR/benchmarks/iperf $PORTSDIR/net/pen $PORTSDIR/net/rsync # Wireless Leiden ports at $WL_PORTSDIR $WL_PORTSDIR/net/lvrouted $WL_PORTSDIR/net/transproxy }}} == Testing Images == * [wiki:KnownErrorMessages] * [wiki:TestingViaNFS] * [wiki:TestMatrix] == Hardware == * Soekris 4521, 4801, 5501, bios version 1.23, 1.31b, 1.32, 1.33 * Alix 2D3, only use latest bios: 0.99h Note on updating Soekris bios (CTRL+P to enter): no luck with minicom, used cu on linux (Ubuntu: sudo apt-get install lrzsz cu): {{{ chown uucp /dev/ttyS0 cu -l /dev/ttyS0 -s 9600 > download - Start sending file using XMODEM/CRC protocol. ~+sx -X b4501_133.bin Sending b4501_133.bin, 608 blocks: Give your local XMODEM receive command now. Bytes Sent: 77824 BPS:892 Transfer complete File downloaded succesfully, size 608 Blocks. }}} flashupdate reboot == What changed for FreeBSD 8.0 ? == All modifications required for 8.0 are incorporated in the nodefactory version that can be found in svn: http://svn.wirelessleiden.nl/svn/code/iris/trunk/nanobsd/ Some modifications for freebsd-8.0 RELEASE going from 7.2 to 8.0: * use of rc.conf.local in stead of rc.node.local to be more in-line with standard FreeBSD config * apply only patch for usbdevs and if_udav.c to make Davicom usb-lan adapter work; * no cardbus patch * modify kernel.wleiden configuration file, see http://svn.wirelessleiden.nl/svn/projects/proxy/2.0/nanobsd/nanobsd/cfg/kernel.wleiden {{{ # options CLK_USE_I8254_CALIBRATION # options ADAPTIVE_GIANT # Giant mutex is adaptive. # device wlan_scan_ap # 802.11 AP mode scanning. # device wlan_scan_sta # 802.11 STA mode scanning. # device ppp # Kernel PPP # device ugen # Generic # device sio }}} * modify ttys-nanobsd.patch file * wireless interface configuration is quite different (configuration script was changed) modify rc.conf.local somewhat like this: {{{ wlans_ath0="wlan0" create_args_wlan0="wlanmode hostap mode 11b ssid ap-omni.huub.wleiden.net channel 7" ifconfig_wlan0="inet 172.17.16.1/26" }}} note that interface has to be 'down' to create_args_wlan0! * usb-lan convertor interface for Digitus adapter is now ue (not aue) * '''known issues''' (see freebsd-embedded mailinglist): geometry mismatch between cf-card and disklable; boot-sector/upgrade-procedure. = H. Hints = * You could always decide to install your FreeBSD base instance into a [http://www.freebsd.org/doc/en/books/handbook/virtualization-guest.html virtual environment] * {{{cvsup.nl.freebsd.org}}} is the local dutch cvsup mirror, replace `nl` with your proper country code. An alternative way to update sources is svn, possibly with a symlink for use of different releases, e.g.: {{{ $ svn co http://svn.freebsd.org/base/release/8.2.0/ /usr/src-release-8.2.0 $ ln -s /usr/src-release-8.2.0 /usr/src }}} * updating ports: {{{ # cd hybrid-releng-9.0-nanobsd # svn up . # pkg_delete /var/db/pkg/* # rm -R /usr/local/* /var/db/ports/* /var/db/pkg/* # ./tools/image ports update }}} check list of installed packages: {{{ # echo pkg/All/* | wc -w }}} * It seems tempting to follow {{{/usr/share/examples/cvsup/stable-supfile}}} instead of {{{/usr/share/examples/cvsup/standard-supfile}}} please __DO NOT__ unless your like big trouble as stable-supfile is the stable __DEVELOPMENT__ branch e.g. upcoming stable. * Default username/password = root/, so please do mind, _when no password is set ssh login is disabled_. * Always use `cu` or `putty` for serial communication minicom, screen, putty all have issues of some kind. * After writing an image to cf-card using a usb-card-reader/writer you may encounter a problem dd-ing another image. Rebooting will help. Alternatively you can apply a power-cycle to the usb device: {{{ usbconfig -u 1 -a 2 power_off usbconfig -u 1 -a 2 power_on }}} (unit and address may vary, see usbconfig list). = J. Comments, questions or remarks? = Feel free to edit this page or [mailto:techniek@lijst.wirelessleiden.nl?subject=NanoBSD%20odefactory%improvement send an email] to our [http://lijst.wirelessleiden.nl/mailman/listinfo/techniek techniek mailinglist]