Nodefactory based on FreeBSD 11-RELEASE

---

Scope

We intend to implement a number of changes/improvements.

Done:

• implement new package management (pkgng)
• cleanup packages that are no longer required: nrpe, vim, bash-static, pftop, nmap, python-Jinja2, sixxs-aiccu, openvpn
• implement latest version of lvrouted (auto update default route)
• add ​net/ladvd
• add inet check in snmpd.conf
• update scripts to use 'drill' instead of 'dig'
• remove the "rcvar='set_rcvar'" statement from startup scripts in /etc/rc.d/ and /usr/local/etc/rc.d
• based on FreeBSD-11.1-RELEASE
• ​unbound replaces bind; recursive dns server on standard-node; authorative server for wleiden.net domain on two special servers in the network; ​unbound tutorial. Unbound optimizes forwarders automatically, so nameservershuffle is no longer needed. Use unbound from ports, not local_unbound.
• replaced thttpd by ​apache-2.4
• no longer soft-updates
• update captive portal to increase speed: use of static html landing page
• merged APU specifics into generic configuration (we do not use X64, MIPS or ARM)
• added NSD, may be used as authorative nameserver
• correction of dhcpd.conf for aliased interfaces (no 'shared-networks')
• use Poudriere for package building

Pending:

• use ipfw instead of pf in order to make usb-lan adapter work, see ticket #206 for problem description. Functionality:

    Requirements:
      direct new user to the Captive Portal page (rdr, blocking acl) on node.
      only allow http/https traffic to internet (gateway setup).
      NAT on gateways.
      Incoming RDR/NAT voor (webcam) services on gateways. 
    Nice-to-have features:
      Bandwidth limiting (FUP).
      Bad traffic detection and blocking. 
  

• add 'welcome back' page to captive portal (activated when connection interrupted)
• bsnmp may replace net-snmp (low priority)
• wl-web page redesign for local users / maintainance
• add ssh-guard or no password-login, only keys
• watchdog for critical daemons?
• evaluate use of nsd for wleiden.net local domain dns service
• correct /tools/wl-config external fetch url (wirelessleiden.nl/config/)
• gformat should include ssh-redirect rule in pf.hybrid.conf.local for the gateways.

  # Redirect SSH on internetport 1022 to 22 because of ssh spam, block external port 22: 
  rdr on $ext_if inet proto tcp from any to $ext_if port 1022 tag SRV -> 172.17.87.1 port 22
  

• captive portal: allow some (specified) external IP's (such as the sponsor website or to allow browsing the WirelessLeiden website e.g. for support information online). Also the captive portal page could provide node specific information that would be maintained through a website (CMS).
• captive portal (enhancement): For user support it might be helpful to provide technical user-related information on the bottom of the captive portal; maybe a comment field and button to allow the user to send input to support@….

Will keep

• ucspi-tcp-0.88_2 for redirect captive portal
• python
• mtr, curl, screen, sudo
• dnsmasq (for Soekris hardware and possibly dhcpd service)
• pen
• tinyproxy

Gradually we are implementing the various changes, starting from the present software configuration (9.0-RELEASE) on FreeBSD10.2-RELEASE with the next generation package management system (pkgng). The procedure to build this 'work-in-progress' nodefactory-host is described below.

A ​test-node-configuration is available in svn. This wleiden.yaml file can be used to test the image by installing on an Alix-board with connection to the local network. We are also evaluating the new APU-board, see the wiki-page.

A. Setup a FreeBSD host

Setup is tested in a high performance ​VirtualBox environment, running 4 x Core i7 CPU, 4096MB RAM and SSD disk (20GB). Timing estimates are based on this system:

• Make sure /usr is at least 12 GB but better is 20 GB in size, as building images requires quite some space.
• Make sure you install the 32bit i386 release of FreeBSD also when your system does support amd64, as cross compiling can give some nasty surprises.
• Internet connection is required for initial building phase. Ones completed succesfully no internet connection is required anymore.

Tip: Use an SSD disk. If not available use a separate hard disk, mounted on /usr/obj to speed up the compilation process.

Get yourself a fresh i386 FreeBSD host installed as follows:

A.1. Run the basic CD installer

The procedure below has been tested with 11.1 (standard developer install - no ports -, e.g. with default partitioning will fit our needs). Installing FreeBSD is outside the scope of this document, take a look into ​the FreeBSD handbook Chapter 2 Installing FreeBSD if you do not know the details.

A.2. After first boot

Warning: Because of the many mounts and unmounts done in various phases running as root is required. Do NOT use a production machine as WL build environment, because it will fail at some point in time!

Allow SSH root login, since all commands below need to be executed as root: PermitRootLogin yes in file:/etc/ssh/sshd_config

Set correct date/time and make sure it is kept up2date: build# ntpdate 0.nl.pool.ntp.org build# echo 'ntpd_enable="YES"' >> /etc/rc.conf

A.3 Set some useful variables

Alter the shell configuration file:/root/.cshrc:

Ensure ftp is set to passive mode, to avoid potential firewall issues: build# echo 'setenv FTP_PASSIVE_MODE YES' >> /root/.cshrc

Set a default password for the images that you will produce: build# echo 'setenv CFG_ROOT_PASSWORD DefaultPassword12!' >> /root/.cshrc

Define your nanobsd (svn) working directory: build# echo 'setenv R /root/nanobsd' >> /root/.cshrc

Next load your file (or login again): build# source /root/.cshrc

A.4. Development tools

Bootstrap development packages: build# env ALWAYS_ASSUME_YES pkg install vim-console sudo screen ca_root_nss subversion

Tip: screen can be a handy tool if you are working on a remote host.

With 'screen' you can open a virtual terminal, in which you can do everything like in the normal terminal. 
But you can detach it if you want to do other things in the main terminal and reattach it later. It even works after you quit your main terminal.

Common screen commands:
1. Start a new terminal:
build# screen
2. Type CTRL+A and D to detach this new terminal
3. Reattach it:
build# screen -R

A.5. get latest sources

build# svn co -q ​https://svn.FreeBSD.org/base/releng/11.1 /usr/src

A.6 set up Poudriere

Setting up of Poudriere is taken care off by the package-build.sh script see section B.2 below. For details refer to separate page Poudriere

B. Build environment

B.1 Download the environment from the Wireless Leiden svn repository

build# svn checkout ​https://svn.wirelessleiden.nl/svn/code/hybrid/branches/releng-11/nanobsd $R build# cd $R

B.2. Compile all required packages

First time use requires the required packages are installed, ports are updated, port trees are merged and the Poudriere jail is setup and loaded:

Set up Poudriere environment by running [15 min]:

build# screen $R/tools/package-build.sh initenv

Now build all packages, this will take quite some time [90 min]: build# $R/tools/package-build.sh build

This will generate the following listing of packages: build# ls $R/pkg/All

GeoIP-1.6.6.txz			isc-dhcp43-server-4.3.3.txz	pkg-1.6.1_2.txz
apache24-2.4.17.txz		ladvd-1.0.4_2.txz		py27-setuptools27-17.0.txz
apr-1.5.2.1.5.4.txz		ldns-1.6.17_5.txz		py27-yaml-3.11.txz
ca_root_nss-3.20.txz		libevent2-2.0.22_1.txz		python-2.7_2,2.txz
curl-7.45.0.txz			libffi-3.2.1.txz		python2-2_3.txz
db5-5.3.28_2.txz		libidn-1.31.txz			python27-2.7.10_1.txz
dnsmasq-2.75,1.txz		lvrouted-13473.txz		screen-4.3.1_2.txz
expat-2.1.0_3.txz		mtr-0.86.txz			sudo-1.8.15.txz
gdbm-1.11_2.txz			net-snmp-5.7.3_11.txz		tinyproxy-1.8.3_1,1.txz
gettext-runtime-0.19.6.txz	nettle-2.7.1.txz		ucspi-tcp-0.88_2.txz
gmp-5.1.3_2.txz			nsd-4.1.5.txz			unbound-1.5.5.txz
iftop-1.0.p4.txz		pcre-8.37_4.txz			vim-lite-7.4.900.txz
indexinfo-0.2.4.txz		pen-0.31.1.txz
iperf-2.0.5.txz			perl5-5.20.3_8.txz


(list updated nov 6, 2015)

B.4 Build nanobsd aka make sure to prepare some coffee ;-)

Start the build process [2 hours]: build# screen $R/tools/image build

Tip: Save some power and use an ​power-saver script to shut-down the machine ones not being using any-more. Tip: The geometry of a default image is based on a 1GB CF card, changes file:$R/cfg/nanobsd.wleiden

C. Fetch node configuration onto image, write to CF disk or remotely update

C.1. Fetch node-configuration

Fetch the configuration from wirelessleiden.nl/config/ which is generated from the Wireless Leiden 'genesis' database: build# $R/tools/image config for <Nodename>

You can inspect the image bij mounting as memory disk: build# $R/tools/image edit

C.2. Write the correct image to CF (media based on SLC and not MLC flash seem to perform much better)

---

NOTE: _.disk.full is required for new CF cards as it contains two base system-partitions and one configuration partition.

_.disk.image can be used to update one system-partitioin on an existing CF card

---

1. New image to local, fresh CF disk: Put full image on a fresh compact flash disk using a card reader/writer attached to your buildbot PC. Minimum size of the CF disk is 1 GB. Use the script if you are using a usb connected cf-disk reader/writer:

build# $R/tools/write-image.sh

or the command line:

build# dd bs=64k if=/usr/obj/nanobsd.wleiden-hybrid/_.disk.full of=/dev/da0

(assuming /dev/da0 is your compact flash entry; this takes about 15 minutes, check progress by typing Ctrl-T; you may wish to check afterwards with fdisk whether there are three partitions on the disk. You can also mount /dev/da0s1a and /dev/da0s3 and check the configuration)

2. CF disk with existing image: Put partial new image on slice (slice 2) of CF disk using a card reader/writer attached to the buildbot PC with the CF disk containing the existing image.

build# dd bs=64k if=/usr/obj/nanobsd.wleiden-hybrid/_.disk.image of=/dev/da0s2

D. Check the cf card and apply last minute changes

D.1. Check cf card

While the cf card is still in your flash card reader you can check whether the image has been written correctly. Check whether you can mount the partitions, e.g.

build# mount /dev/da0s1a /mnt build# ls /mnt build# umount /mnt

and the configuration partition:

build# mount /dev/da0s3 /mnt build# ls /mnt

If you cannot mount the partitions, take a fresh cf card and start again with writing the image.

D.2 Adjust captive portal

We now use a static landing page (in previous versions of the nodefactory the landing page was generated on the fly by a python script, this caused a considerable delay). You can find this page in the /usr/local/www/wlportal directory.

(to be completed)

F. Notes

F.2 update lvrouted port (ONLY) if necessary

Warning: Skip the following steps unless you want to update the current lvrouted version (12879) and know what you are doing:

Checkout latest version of lvrouted (i.e. beyond 13473): build# svn checkout ​http://svn.wirelessleiden.nl/svn/node-config/other/lvrouted/trunk /tmp/lvrouted

Run the release.sh script to produce the tar 'release' file file:lvrouted-.......tar.gz: build# cd /tmp/lvrouted build# ./tools/release.sh

Upload the file:lvrouted-13473.tar.gz to the [webfolder.wirelessleiden.nl/lvrouted/ directory.

Update PORTVERSION to match current release: build# vim $R/ports/net/lvrouted/Makefile

Update the hash values: build# make makesum -C $R/ports/net/lvrouted

commit changes: build# svn ci $R/ports/net/lvrouted

F.3 Existing image on node, remote update (slice 2) (network connection to machine 172.x.y.z required)

For remotely updating an existing node software configuration use the image script: build# $R/tools/image deploy on <node> [and reboot]

Tip: You may adjust the block size to make the transfer more efficient (additionally ssh -C is possible) and you can use Ctrl-T to check progress.