Setting up a local DCS network

Introduction

The DCS boards are designed to be connected to a server--or central node--through a network. This server will most importantly be running the InterComLayer. However, the network will also allow remote logon to the boards using SSH, as well as providing them with access to remote NFS-shares. The network is preferably set up and managed with central configuration tools, such as DHCP and DNS. Configuring each board locally will in most cases be very inconvenient, particuarly as all boards are supposed to be descriminated solely by their hardware board numbber--not by local configuration files. This page will outline how such a network is set up.

DHCP configuration

To reach the outside world, the DCS boards will need an IP-address. This they will obtain from the DHCP server. The first thing they do when they are woken up and want to set up their network interface, is to send a querry about it. The DHCP server can identify clients asking for IP-addresses on their hardware--or MAC--address. For networks based on Ethernet, this is always a unique 48 bit identifier. For the special case of the boards, the first 24 bits are 40:EA:54. The last 24 is the board number. In the setup we propose here, the board number will also be used as part of the IP-address and hostname. We will be using a private class A network, 10.0.0.0, allowing for 24 bits of client address space. 16 bits in the range 10.1.0.0 to 10.1.255.255 are reserved for the DCS boards. Each of them will have an etry in /etc/dhcpd.conf: host dcs0000 { hardware ethernet 40:EA:54:00:00:00; fixed-address 10.1.0.0; } Further, we reserve the range 10.0.1.0 to 10.0.1.255 for registered devices (servers, workstations, laptops, etc.), and 10.0.2.0 to 10.0.2.255 for unregistered devices. The former will be registered in dehcpd.conf, hence given fixed IP-addresses, while the later are not, and will have to suffice with random addresses in the mentioned range. Any but the most temporary devices are encouraged to be registered.

The DHCP-server in this example will also act as a gateway to the outside world. We will discuss this in greater detail later. Here we will only cover the implications for the configuration of DHCP. The sever will thus have two network interfaces--one for the private net, and one for the external. Addresses for the external will be provided by another server with which we are only concerned as a mere client. Our DHCP-server can, of cource, in principle hand out addresses on both interfaces--definitely not what we want. Thus, the important issue here is to prevent this from ever happening. As we are a normal client on the external network, we do in principle not even know anything of the parameters of the external network until we have received this information from the DHCP-server of the external network. However, it is in most cases safe to assume the subnet and netmask will reamin the same as long as we connect to the same physical network. This information is all we need. We define this network in dhcpd.conf, but without any pool of addresses to hand out, effectively dissableing our server from answering dhcp-requests on this interface. subnet 128.141.0.0 netmask 255.255.0.0 { not authoritative; } In our case the IP-address of gateway and DHCP-, DNS- and time-server is all 10.10.10.10. One may choose to spread these services to several machines, but for a small test-network like this, performance and up-time are not a great concern. In a final setup, it might be a consideration, though. We have defined these servies as options in the begining if the file. The essential sections now become like this:

DHCP set-up for DCS private network

ddns-update-style none; subnet 10.0.0.0 netmask 255.0.0.0 { authoritative; allow booting; allow bootp; default-lease-time 3600; max-lease-time 7200; ddns-update-style none;

Get hostname from host entries
use-host-decl-names on;
Get hostnames from DNS

get-lease-hostnames on; option domain-name "feenet"; option routers 10.10.10.10; option domain-name-servers 10.10.10.10; option time-servers 10.10.10.10; option ntp-servers 10.10.10.10; option nntp-server 10.10.10.10; option broadcast-address 10.255.255.255; option subnet-mask 255.0.0.0; pool {

Reserved for registered devices - not DCS boards!

range 10.0.1.0 10.0.1.255; deny unknown-clients; boot-unknown-clients off; } pool {

Reserved for _not_ registered devices - not DCS boards!

range 10.0.2.0 10.0.2.255; allow unknown-clients; boot-unknown-clients on; } pool {

Reserved for DCS boards - must be registered!

range 10.1.0.0 10.1.255.255; deny unknown-clients; boot-unknown-clients off; } } subnet 128.141.0.0 netmask 255.255.0.0 { not authoritative; } group { host dcs0000 { hardware ethernet 40:EA:54:00:00:00; fixed-address 10.1.0.0; } host dcs0001 { hardware ethernet 40:EA:54:00:00:01; fixed-address 10.1.0.1; } } The entries for most of the DCS boards have been excluded in a effort to save space. A complete, working file (tested on SLC3) can be found on http://www.ift.uib.no/~dagtl/dhcpd.conf. For details concerning configuring the DHCP-server, please refer to "man 5 dhcpd.conf", "man dhcpd" and "man dhcp-options".

DNS configuration

As we know, all devics on a IP-network is identified by its IP-address. This is a numeric reference. While computers generally prefer this naming scheme, the same can not be said on the account of human beings, we prefer names that make "sense". The purpose of the DNS server is to map such "sane names" to IP-addresses--and vice versa. As the name imply, the DNS is centered around domains. They are organized into a hierachy. For our small network, we will only explore a small fraction of the capabilities of the DNS. We will only have one domain, to be called feenet. The main file to configure is /etc/named.conf. We will have to add two more sections to it: zone "feenet" { type master; file "feenet.zone"; }; zone "10.in-addr.arpa" { type master; file "10.in-addr.arpa.zone"; }; Further, the two files feenet.zone and 10.in-addr.arpa.zone have to be created. They shall be located at the rather awkward directory /var/named/chroot/var/named/. The first will handle the forward address resolution, i.e. names to numbers, while the last will take care of the reverse address resolution--numbers to names. It is important that these two files are kept consistent; so that names and numbers have a one-to-one relation! Without forther delay we will present them in order: $TTL 86400 @ IN SOA feenet. root.master.feenet. ( 16 ; serial 28800 ; refresh 14400 ; retry 3600000 ; expire 86400 ; ttl ) IN NS master.feenet. master IN A 10.10.10.10 dimdns IN CNAME master.feenet. tpc-fee_0_02_1 IN A 10.1.0.0 dcs0000 IN CNAME tpc-fee_0_02_1.feenet. dcs0001 IN A 10.1.0.1 and: $TTL 86400 @ IN SOA feenet. root.master.feenet. ( 12 ; serial 28800 ; refresh 14400 ; retry 3600000 ; expire 86400 ; ttl ) IN NS master.feenet. 10.10.10 IN PTR master.feenet. 0.0.1 IN PTR dcs0000.feenet. 1.0.1 IN PTR dcs0001.feenet. Once again, we have only included the essential head section, including the first few clients. The mayority of the clients are left out, as they are a mere repetition of the first. The complete files are available from http://www.ift.uib.no/~dagtl/feenet.zone and http://www.ift.uib.no/~dagtl/10.in-addr.arpa.zone. Links to these two files has to be present in /var/named.

If we take a closer look at the dhcpd.conf file, we notice it has been set up to obtain the host names from the DNS server. Alternatively, it can use the name provided for each separate host. However, this last approach will put us in the trouble of manually keeping the the DNS and DHCP host name configurations synchronized, whereas the initially suggested method will only require updates to be made to the DNS.

We can also provide the DCS boards with aliases. For example, DCS borad number 0000, with IP address 10.1.0.0, has been given the name tpc-fee_0_02_1.feenet as its "real" name, and dcs0000.feenet as an alias. The first name is based on its geographical location in the TPC detector, wheras the second is derived from the board number. We can choose to address it using the name we at any given time find most convenient. It is important to point out that only the real name will be returned on reverse name lookups.

The name of the FeeServer running on the DCS board will be the same as the board's geographical name, or more precicely, the geographical name is the FeeServer name. The purpose of using this as the main name, is to allow the name of the FeeServer to be set from the host name. We want the DCS boards to be completely configured from a server. This is an easy way to transfer the name to the DCS board.

Further information on configuring DNS can be found in "man 5 named.conf" and "man dhcpd".

DNS entry generation

In order to simplify the DNS configuration the entries can be generated by the script names-feenet.sh. The script can be downloaded from the download page.

Usage: names-feenet.sh [OPTION] [configuration file]

The script was written for the TPC setup, so all the default values concern the TPC. All values can be changed be command line parameters in order to customize the output. The routing can either be created from a configuration file (see below for format) or by looping over the geographical address range or board number range.

General Options

--configure-named configure the name deamon, root priviledges required --add-configuration add the configuration to the existing, by default all all DCS board entries are deleted from the existing configuration. --print-names print hostname table to stdout --print-backrouting print back routing table to stdout --enumerate simply enumerate the boards without building hostnames with geographical address --file=<file path> read configuration from file

Detector layout options

The hostnames are built from 3-dimensional geographical addresses, e.g. tpc-fee_0_04_3. The number of boards in each direction can be specified by the --nof[X,Y,Z] option

--nofX=<x> TPC default 2 --nofY=<y> TPC default 18 --nofZ=<z> TPC default 6

The number of digits for each coordinate. Missing digits are filled with preceeding 0's.

--digitsX=<d> TPC default 1 --digitsY=<d> TPC default 2 --digitsZ=<d> TPC default 1

Range of board numbers: specify the first and/or the last board.

--min-boardno default 1 --max-boardno default unlimited

--basename=<name> TPC default: tpc-fee --defaultname=<name> default: dcs

Board enumeration skips the generation of the geographical addresses and generates just names with the default name --enumerate

Network settings

--netaddr=<address> network base address, default 10.0.0.0 --netmask=<mask> network mask, default 255.0.0.0

Format of the configuration file

The configuration file is a simple text file, each line contains board number and the 3 coordinates. Comments can be included as usual by marking with '#'.

sample configuration for the TPC
boardno side sector partition

11 1 6 4 1 1 0 5 10 0 0 0 12 0 1 5 13 1 7 3 105 0 9 5

Examples

[tpcfee01] ~ > names-feenet.sh --print-names --nofX=1 --nofY=1 tpc-fee_0_00_0 IN A 10.1.0.1 tpc-fee_0_00_1 IN A 10.1.0.2 tpc-fee_0_00_2 IN A 10.1.0.3 tpc-fee_0_00_3 IN A 10.1.0.4 tpc-fee_0_00_4 IN A 10.1.0.5 tpc-fee_0_00_5 IN A 10.1.0.6 dcs0001 IN CNAME tpc-fee_0_00_0 dcs0002 IN CNAME tpc-fee_0_00_1 dcs0003 IN CNAME tpc-fee_0_00_2 dcs0004 IN CNAME tpc-fee_0_00_3 dcs0005 IN CNAME tpc-fee_0_00_4 dcs0006 IN CNAME tpc-fee_0_00_5

[tpcfee01] ~ > names-feenet.sh --print-names --enumerate --min-boardno=65 --max-boardno=70 dcs0065 IN A 10.1.0.65 dcs0066 IN A 10.1.0.66 dcs0067 IN A 10.1.0.67 dcs0068 IN A 10.1.0.68 dcs0069 IN A 10.1.0.69 dcs0070 IN A 10.1.0.70

[tpcfee01] ~ > names-feenet.sh --print-names --basename=phos-fee --nofX=1 --nofY=1 --nofZ=4 --digitsY=1 --min-boardno=128 phos-fee_0_0_0 IN A 10.1.0.128 phos-fee_0_0_1 IN A 10.1.0.129 phos-fee_0_0_2 IN A 10.1.0.130 phos-fee_0_0_3 IN A 10.1.0.131 dcs0128 IN CNAME phos-fee_0_0_0 dcs0129 IN CNAME phos-fee_0_0_1 dcs0130 IN CNAME phos-fee_0_0_2 dcs0131 IN CNAME phos-fee_0_0_3

[tpcfee01] ~ > names-feenet.sh --print-names /tmp/configuration.txt tpc-fee_0_00_0 IN A 10.1.0.10 tpc-fee_0_01_5 IN A 10.1.0.12 tpc-fee_0_09_5 IN A 10.1.0.105 tpc-fee_1_00_5 IN A 10.1.0.1 tpc-fee_1_06_4 IN A 10.1.0.11 tpc-fee_1_07_3 IN A 10.1.0.13 dcs0001 IN CNAME tpc-fee_1_00_5 dcs0010 IN CNAME tpc-fee_0_00_0 dcs0011 IN CNAME tpc-fee_1_06_4 dcs0012 IN CNAME tpc-fee_0_01_5 dcs0013 IN CNAME tpc-fee_1_07_3 dcs0105 IN CNAME tpc-fee_0_09_5

NFS configuration

We have to provide the DCS boards with disk space shared over the network. The purpose of this is twofold. Firstly, the flash memory occupied by the Linux systems on the DCS boards has a very limited capacity--only 8 megabytes. This will allow for the core system, but is marginal for storing FeeServer and log files. Secondly, flash memory is weakened for each write cycle. It is therefore desireable to reduce write operations to a minimum. This is particuarly true for the log files, which are constantly updated.

Two directores should be provided, one with read-write (rw) access, and one read-only (ro). The FeeServer and other binaries/files the DCS bords are supposed not to write to should go in the ro-directory. Log files and similar files should be placed in the rw directory. As the ro directory we use /nfs_export/fee-net/, and as rw we use /dcs-share. A separate directory for the log files is provided as a subdirectory: /dcs-share/fee-log. The totally different locations of the ro and rw directories might seem odd. This is partly due to historical reasons (the rw directory was early introduced for developent, the ro was introduced when "order" was needed), and partly practical (the rw is much more frequently accessed during debug/development than the ro). Everybody has to be given all rights to the /dcs-share and /dcs-share/fee-log directories.

The configuration of the NFS shares are made in the /etc/exports file: /dcs-share 10.0.0.0/255.0.0.0(rw,sync,no_wdelay,all_squash) /nfs_export/fee-net 10.0.0.0/255.0.0.0(ro,sync,no_wdelay,all_squash) The file is also available from http://www.ift.uib.no/~dagtl/exports.

During init, the DCS boards will try to run a startup file for the FeeServer from the /nfs_exports/fee-net directory. It is thus important that the directory is populated properly with these files. They can be downloded from http://www.ift.uib.no/~dagtl/fee.tgz.

NAT configuration

NAT and IP forwarding will make the external network (Internet) accessible from the DCS network. The DCS netwok is however not accessible from the Internet. The DCS borads and other computers on the DCS network is "hiding" behind the gateway, from the outside it appears as if all requests originate from the gateway. For the computers on the inside, at the other hand, will not notice the precence of the gateway. NAT and IP forwarding is configured using IPTABLES. A simple script will set ip up appropriately, assuming eth0 is Internet, and eth1 DCS network:

!/bin/bash

iptables --flush iptables --table nat --flush iptables --delete-chain iptables --table nat --delete-chain iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE iptables --append FORWARD --in-interface eth1 -j ACCEPT echo 1 > /proc/sys/net/ipv4/ip_forward The script can also be downloaded from http://www.ift.uib.no/~dagtl/masq. It is wise to load this script file on startup, preferably from /etc/rc.d/rc.local.