View source for Setting up a local DCS network

== Introduction ==
The DCS boards are designed to be connected to a server--or central node--through a network. This server will most importantly be running the InterComLayer. However, the network will also allow remote logon to the boards using SSH, as well as providing them with access to remote NFS-shares. The network is preferably set up and managed with central configuration tools, such as DHCP and DNS. Configuring each board locally will in most cases be very inconvenient, particuarly as all boards are supposed to be descriminated solely by their hardware board numbber--not by local configuration files. This page will outline how such a network is set up.
== DHCP configuration ==
To reach the outside world, the DCS boards will need an IP-address. This they will obtain from the DHCP server. The first thing they do when they are woken up and want to  set up their network interface, is to send a querry about it. The DHCP server can identify clients asking for IP-addresses on their hardware--or MAC--address. For networks based on Ethernet, this is always a unique 48 bit identifier. For the special case of the boards, the first 24 bits are 40:EA:54. The last 24 is the board number. In the setup we propose here, the board number will also be used as part of the IP-address and hostname.
We will be using a private class A network, 10.0.0.0, allowing for 24 bits of client address space. 16 bits in the range 10.1.0.0 to 10.1.255.255 are reserved for the DCS boards. Each of them will have an etry in /etc/dhcpd.conf:
host dcs0000 {
hardware ethernet 40:EA:54:00:00:00;
fixed-address 10.1.0.0;
}
Further, we reserve the range 10.0.1.0 to 10.0.1.255 for registered devices (servers, workstations, laptops, etc.), and 10.0.2.0 to 10.0.2.255 for unregistered devices. The former will be registered in dehcpd.conf, hence given fixed IP-addresses, while the later are not, and will have to suffice with random addresses in the mentioned range. Any but the most temporary devices are encouraged to be registered.

The DHCP-server in this example will also act as a gateway to the outside world. We will discuss this in greater detail later. Here we will only cover the implications for the configuration of DHCP. The sever will thus have two network interfaces--one for the private net, and one for the external. Addresses for the external will be provided by another server with which we are only concerned as a mere client. Our DHCP-server can, of cource, in principle hand out addresses on both interfaces--definitely not what we want. Thus, the important issue here is to prevent this from ever happening. As we are a normal client on the external network, we do in principle not even know anything of the parameters of the external network until we have received this information from the DHCP-server of the external network. However, it is in most cases safe to assume the subnet and netmask will reamin the same as long as we connect to the same physical network. This information is all we need. We define this network in dhcpd.conf, but without any pool of addresses to hand out, effectively dissableing our server from answering dhcp-requests on this interface.
subnet 128.141.0.0 netmask 255.255.0.0 {
not authoritative;
}
In our case the IP-address of gateway and DHCP-, DNS- and time-server is all 10.10.10.10. One may choose to spread these services to several machines, but for a small test-network like this, performance and up-time are not a great concern. In a final setup, it might be a consideration, though. We have defined these servies as options in the begining if the file. The essential sections now become like this:
# DHCP set-up for DCS private network
ddns-update-style none;
subnet 10.0.0.0 netmask 255.0.0.0 {
authoritative;
allow booting;
allow bootp;
default-lease-time              3600;
max-lease-time                  7200;
ddns-update-style               none;
#Get hostname from host entries
# use-host-decl-names             on;
#Get hostnames from DNS
get-lease-hostnames             on;
option domain-name              "feenet";
option routers                  10.10.10.10;
option domain-name-servers      10.10.10.10;
option time-servers             10.10.10.10;
option ntp-servers              10.10.10.10;
option nntp-server              10.10.10.10;
option broadcast-address        10.255.255.255;
option subnet-mask              255.0.0.0;
pool {
#Reserved for registered devices - not DCS boards!
range 10.0.1.0 10.0.1.255;
deny unknown-clients;
boot-unknown-clients off;
}
pool {
#Reserved for _not_ registered devices - not DCS boards!
range 10.0.2.0 10.0.2.255;
allow unknown-clients;
boot-unknown-clients on;
}
pool {
#Reserved for DCS boards - must be registered!
range 10.1.0.0 10.1.255.255;
deny unknown-clients;
boot-unknown-clients off;
}
}
subnet 128.141.0.0 netmask 255.255.0.0 {
not authoritative;
}
group {
host dcs0000 {
hardware ethernet 40:EA:54:00:00:00;
fixed-address 10.1.0.0;
}
host dcs0001 {
hardware ethernet 40:EA:54:00:00:01;
fixed-address 10.1.0.1;
}
}
The entries for most of the DCS boards have been excluded in a effort to save space. A complete, working file (tested on SLC3) can be found on [http://www.ift.uib.no/~dagtl/dhcpd.conf http://www.ift.uib.no/~dagtl/dhcpd.conf]. For details concerning configuring the DHCP-server, please refer to "man 5 dhcpd.conf", "man dhcpd" and "man dhcp-options".
== DNS configuration ==
As we know, all devics on a IP-network is identified by its IP-address. This is a numeric reference. While computers generally prefer this naming scheme, the same can not be said on the account of human beings, we prefer names that make "sense". The purpose of the DNS server is to map such "sane names" to IP-addresses--and vice versa. As the name imply, the DNS is centered around ''domains''. They are organized into a hierachy. For our small network, we will only explore a small fraction of the capabilities of the DNS. We will only have one domain, to be called ''feenet''. The main file to configure is /etc/named.conf. We will have to add two more sections to it:
zone "feenet" {
type master;
file "feenet.zone";
};
zone "10.in-addr.arpa" {
type master;
file "10.in-addr.arpa.zone";
};
Further, the two files feenet.zone and 10.in-addr.arpa.zone have to be created. They shall be located at the rather awkward directory /var/named/chroot/var/named/. The first will handle the forward address resolution, i.e. names to numbers, while the last will take care of the reverse address resolution--numbers to names. It is important that these two files are kept consistent; so that names and numbers have a one-to-one relation! Without forther delay we will present them in order:
$TTL 86400
@       IN      SOA     feenet.    root.master.feenet.     (
16 ; serial
28800 ; refresh
14400 ; retry
3600000 ; expire
86400 ; ttl
)
IN      NS      master.feenet.
master          IN      A       10.10.10.10
dimdns          IN      CNAME   master.feenet.
tpc-fee_0_02_1  IN      A       10.1.0.0
dcs0000         IN      CNAME   tpc-fee_0_02_1.feenet.
dcs0001         IN      A       10.1.0.1
and:
$TTL 86400
@       IN      SOA     feenet. root.master.feenet.     (
12 ; serial
28800 ; refresh
14400 ; retry
3600000 ; expire
86400 ; ttl
)
IN      NS      master.feenet.
10.10.10        IN      PTR     master.feenet.
0.0.1           IN      PTR     dcs0000.feenet.
1.0.1           IN      PTR     dcs0001.feenet.
Once again, we have only included the essential head section, including the first few clients. The mayority of the clients are left out, as they are a mere repetition of the first. The complete files are available from [http://www.ift.uib.no/~dagtl/feenet.zone http://www.ift.uib.no/~dagtl/feenet.zone] and [http://www.ift.uib.no/~dagtl/10.in-addr.arpa.zone http://www.ift.uib.no/~dagtl/10.in-addr.arpa.zone]. Links to these two files has to be present in /var/named.

If we take a closer look at the dhcpd.conf file, we notice it has been set up to obtain the host names from the DNS server. Alternatively, it can use the name provided for each separate host. However, this last approach will put us in the trouble of manually keeping the the DNS and DHCP host name configurations synchronized, whereas the initially suggested method will only require updates to be made to the DNS.

We can also provide the DCS boards with aliases. For example, DCS borad number 0000, with IP address 10.1.0.0, has been given the name tpc-fee_0_02_1.feenet as its "real" name, and dcs0000.feenet as an alias. The first name is based on its geographical location in the TPC detector, wheras the second is derived from the board number. We can choose to address it using the name we at any given time find most convenient. It is important to point out that only the real name will be returned on reverse name lookups.

The name of the FeeServer running on the DCS board will be the same as the board's geographical name, or more precicely, the geographical name ''is'' the FeeServer name. The purpose of using this as the main name, is to allow the name of the FeeServer to be set from the host name. We want the DCS boards to be completely configured from a server. This is an easy way to ''transfer'' the name to the DCS board.

Further information on configuring DNS can be found in "man 5 named.conf" and "man dhcpd".

=== DNS entry generation ===
In order to simplify the DNS configuration the entries can be generated by the script ''names-feenet.sh''. The script can be downloaded from the [[DCS Download#Network setup Tools | download page]].

Usage: <tt>names-feenet.sh [OPTION] [configuration file]</tt>

The script was written for the TPC setup, so all the default values concern the TPC. All values can be changed be command line parameters in order to customize the  output.
The routing can either be created from a configuration file (see below for format) or by looping over
the geographical address range or board number range.

==== General Options ====

--configure-named      configure the name deamon, root priviledges required
--add-configuration    add the configuration to the existing, by default all
all DCS board entries are deleted from the existing
configuration.
--print-names          print hostname table to stdout
--print-backrouting    print back routing table to stdout
--enumerate            simply enumerate the boards without building hostnames
with geographical address
--file=<file path>     read configuration from file

==== Detector layout options ====

The hostnames are built from 3-dimensional geographical addresses, e.g. <tt>tpc-fee_0_04_3</tt>. The number
of boards in each direction can be specified by the --nof[X,Y,Z] option

--nofX=<x>       TPC default 2
--nofY=<y>       TPC default 18
--nofZ=<z>       TPC default 6

The number of digits for each coordinate. Missing digits are filled with preceeding 0's.

--digitsX=<d>    TPC default 1
--digitsY=<d>    TPC default 2
--digitsZ=<d>    TPC default 1

Range of board numbers: specify the first and/or the last board.

--min-boardno    default 1
--max-boardno    default unlimited

--basename=<name>     TPC default: tpc-fee
--defaultname=<name>  default:     dcs

Board enumeration skips the generation of the geographical addresses and generates just names with the default name
--enumerate

Network settings

--netaddr=<address>  network base address, default 10.0.0.0
--netmask=<mask>     network mask, default 255.0.0.0

==== Format of the configuration file ====
The configuration file is a simple text file, each line contains board number
and the 3 coordinates. Comments can be included as usual by marking with '#'.

# sample configuration for the TPC
# boardno side sector partition
11    1     6       4
1    1     0       5
10    0     0       0
12    0     1       5
13    1     7       3
105    0     9       5


==== Examples ====

[tpcfee01] ~ > names-feenet.sh --print-names --nofX=1 --nofY=1
tpc-fee_0_00_0  IN      A       10.1.0.1
tpc-fee_0_00_1  IN      A       10.1.0.2
tpc-fee_0_00_2  IN      A       10.1.0.3
tpc-fee_0_00_3  IN      A       10.1.0.4
tpc-fee_0_00_4  IN      A       10.1.0.5
tpc-fee_0_00_5  IN      A       10.1.0.6
dcs0001         IN      CNAME   tpc-fee_0_00_0
dcs0002         IN      CNAME   tpc-fee_0_00_1
dcs0003         IN      CNAME   tpc-fee_0_00_2
dcs0004         IN      CNAME   tpc-fee_0_00_3
dcs0005         IN      CNAME   tpc-fee_0_00_4
dcs0006         IN      CNAME   tpc-fee_0_00_5

[tpcfee01] ~ > names-feenet.sh --print-names --enumerate --min-boardno=65 --max-boardno=70
dcs0065 IN      A       10.1.0.65
dcs0066 IN      A       10.1.0.66
dcs0067 IN      A       10.1.0.67
dcs0068 IN      A       10.1.0.68
dcs0069 IN      A       10.1.0.69
dcs0070 IN      A       10.1.0.70

[tpcfee01] ~ > names-feenet.sh --print-names --basename=phos-fee --nofX=1 --nofY=1 --nofZ=4 --digitsY=1 --min-boardno=128
phos-fee_0_0_0  IN      A       10.1.0.128
phos-fee_0_0_1  IN      A       10.1.0.129
phos-fee_0_0_2  IN      A       10.1.0.130
phos-fee_0_0_3  IN      A       10.1.0.131
dcs0128         IN      CNAME   phos-fee_0_0_0
dcs0129         IN      CNAME   phos-fee_0_0_1
dcs0130         IN      CNAME   phos-fee_0_0_2
dcs0131         IN      CNAME   phos-fee_0_0_3

[tpcfee01] ~ > names-feenet.sh --print-names /tmp/configuration.txt
tpc-fee_0_00_0  IN      A       10.1.0.10
tpc-fee_0_01_5  IN      A       10.1.0.12
tpc-fee_0_09_5  IN      A       10.1.0.105
tpc-fee_1_00_5  IN      A       10.1.0.1
tpc-fee_1_06_4  IN      A       10.1.0.11
tpc-fee_1_07_3  IN      A       10.1.0.13
dcs0001         IN      CNAME   tpc-fee_1_00_5
dcs0010         IN      CNAME   tpc-fee_0_00_0
dcs0011         IN      CNAME   tpc-fee_1_06_4
dcs0012         IN      CNAME   tpc-fee_0_01_5
dcs0013         IN      CNAME   tpc-fee_1_07_3
dcs0105         IN      CNAME   tpc-fee_0_09_5

== NFS configuration ==
We have to provide the DCS boards with disk space shared over the network. The purpose of this is twofold. Firstly, the flash memory occupied by the Linux systems on the DCS boards has a very limited capacity--only 8 megabytes. This will allow for the core system, but is marginal for storing FeeServer and log files. Secondly, flash memory is weakened for each write cycle. It is therefore desireable to reduce write operations to a minimum. This is particuarly true for the log files, which are constantly updated.

Two directores should be provided, one with read-write (rw) access, and one read-only (ro). The FeeServer and other binaries/files the DCS bords are supposed ''not'' to write to should go in the ro-directory. Log files and similar files should be placed in the rw directory. As the ro directory we use /nfs_export/fee-net/, and as rw we use /dcs-share. A separate directory for the log files is provided as a subdirectory: /dcs-share/fee-log. The totally different locations of the ro and rw directories might seem odd. This is partly due to historical reasons (the rw directory was early introduced for developent, the ro was introduced when "order" was needed), and partly practical (the rw is much more frequently accessed during debug/development than the ro). Everybody has to be given all rights to the /dcs-share and /dcs-share/fee-log directories.

The configuration of the NFS shares are made in the /etc/exports file:
/dcs-share                10.0.0.0/255.0.0.0(rw,sync,no_wdelay,all_squash)
/nfs_export/fee-net       10.0.0.0/255.0.0.0(ro,sync,no_wdelay,all_squash)
The file is also available from [http://www.ift.uib.no/~dagtl/exports http://www.ift.uib.no/~dagtl/exports].

During init, the DCS boards will try to run a startup file for the FeeServer from the /nfs_exports/fee-net directory. It is thus important that the directory is populated properly with these files. They can be downloded from [http://www.ift.uib.no/~dagtl/fee.tgz http://www.ift.uib.no/~dagtl/fee.tgz].
== NAT configuration ==
NAT and IP forwarding will make the external network (Internet) accessible from the DCS network. The DCS netwok is however not accessible from the Internet. The DCS borads and other computers on the DCS network is "hiding" behind the gateway, from the outside it appears as if all requests originate from the gateway. For the computers on the inside, at the other hand, will not notice the precence of the gateway. NAT and IP forwarding is configured using IPTABLES. A simple script will set ip up appropriately, assuming eth0 is Internet, and eth1 DCS network:
#!/bin/bash
iptables --flush
iptables --table nat --flush
iptables --delete-chain
iptables --table nat --delete-chain
iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE
iptables --append FORWARD --in-interface eth1 -j ACCEPT
echo 1 > /proc/sys/net/ipv4/ip_forward
The script can also be downloaded from [http://www.ift.uib.no/~dagtl/masq http://www.ift.uib.no/~dagtl/masq]. It is wise to load this script file on startup, preferably from /etc/rc.d/rc.local.

[[Category:DCS]]