Setup of low-level DCS for TPC Front-end electronics
General layout
The main idea is to provide a local network with all necessary services like DHCP, name server, time server aso. for the DCS boards. The name server running on the server machine provides the link between the board MAC address and the Board hostname. The server machine is connected via a second network to the CERN network and allows IP forwarding which means all machines on the local net have access to the outer network but not the other way.
Logging on to the DCS boards is only possible from the server machine.
The server machine runs also a DIM name server and has also the IntercomLayer installed. See below for a description how to start/configure.
Please refer to the description of Setting up a local DCS network to get further information on the technical details.
DIM in distributed networks
In general it is possible to run the DIM name server on a gateway machine with to networks. The ports are connected to both networks. Since DIM relies on a direct connection between servers and clients, it is in a setup like this not possible to connect client and servers running on different sub networks.
But is possible to have a relay on the gateway machine running, in out case this is the Intercom Layer.
This was there principle, but we didn't get it running this way. We suspect that it is due to firewall rules since there gateway machine was registered as a portable and not as a fixed machine. This will change.
So far the Windows machine running PVSS must thus be connected to the local network.
Naming scheme of the DCS boards
Currently the name and dhcp servers can provide hostnames/ ip addresses to the boards 061 to about 250. The default name is like dcs0061.
In order to introduce a geographic naming scheme the Boards will be named
tpc-fee_<side>_<sector>_<partition> e.g. tpc-fee_0_09_03
Currently the DCS boards 61 to 69 get some arbitrary names tpc-fee_*_*_*. The hostname dcs0xxx can still be used to log on.
Accounts on the master node
A local user fee can be used for general purpose tasks like sending commands to the FeeServers (e.g. configuration files) or monitor the DIM system, ask Dag, Roberto or me for the password.
In addition we will enable a couple of CERN accounts in order to give access to developers.
Network disks
The master provides network disks for the DCS boards. The /dcs-share folder on the master is mounted on /dcs-share on all the boards. The directory is mounted in read/write mode. In order to give write access also to the boards you have to make sure that the access rights for the certain sub-directory is set to rwx for all access groups.
DIM setup
DIM name server
Choose dimdns as DIM_DNS_NODE on the local network. Choose the name of the server machine (e.g. tpcfee02.cern.ch) in order to access it from the outside. The latter has to be tested so far but it was confirmed by the DIM developers that this should work.
Looking at the system with a DIM tool
On the master just type dimInformationDisplay and you will get the unix DIM monitoring tool. Choose
View -> All servers
Investigating services provided by the Server
in order to display all servers known to the system. By right click on the server icon you can choose Services and you will get a window with the list of the services provided by the server. Select e.g XX_TEMP which is the temperature of FEC XX and press View. Press Subscribe in the window popping up. Note: a value -2000 -2E+02 means no link, which could be that the FEC is in state off (service XX_STATE is 0)
Limitations
The Dim Information Display can only show one value at a time. So, this is only suitable for debugging. The full monitoring will be done from PVSS.
The Intercom Layer
When logged on to the master node, start the Intercom Layer by typing
startInterComLayer
This will start the ICL in the current directory. The ICL will look for the following configuration files in the rundir:
Profile.txt Servers.txt Services.txt
The startInterComLayer command is based on the default TPC configuration files, which contain all the 216 servers. Several options can be used to customize the configuration:
- --help: print a command description
- --stdconf: run in the default configuration
- --server=<pattern>: run with a sub-set of the standard configuration. pattern can be used to select some servers from the default configuration. The wildcard '*' can be used for group addressing. If no server of the default configuration matches the pattern, the pattern is interpreted as server name. Multiple invocations of the --server option specify a list of servers
- --rundir=<dirname>: specify the rundir, the default dir is /tmp/${USER}/ICL-rundir.
# start only for server tpc-fee_0_00_0 startInterComLayer --server='tpc-fee_0_00_0'
# start only for all server on side 1 startInterComLayer --server='tpc-fee_1_*_*'
# start only for all server in sector 10 on side 0 startInterComLayer --server='tpc-fee_0_10_*'
# start only for servers in sector 10 and 11 on side 0 startInterComLayer --server='tpc-fee_0_1[0,1]_*'
# start only for servers dcs0070 and dcs0071 startInterComLayer --server=dcs0070 --server=dcs0071
To start it permanently in the background:
screen -S InterComLayer startInterComLayer <options> &
In the current configuration the ICL tries to connects to all servers following the naming scheme tpc-fee_*_*_*. For each it subscribes to the services
- MAIN_STATE
- XX_STATE
- XX_TEMP
- XX_AV
- XX_AC
- XX_DV
- XX_DC,
where XX denotes the FEC. Branch A FECs 00 to 12, Branch B FECs 16 to 27.
Stoping the InterComLayer
The ICL can be killed by any user by executing:
sudo killInterComLayer
Stand alone FEC monitoring on the DCS board
A stand-alone monitoring script exploiting the rcu-sh can be used to read out Data Points from the Monitoring and Safety Module. After logon to a DCS board try:
# gives you a help tpc-feemonitor.sh --help # reads default services on FEC A01 tpc-feemonitor.sh --fec=A01 # monitors default services on FEC A01 5 times with sleep of 10 sec tpc-feemonitor.sh --fec=A01 --loop=5 --sleep=10 # reads temperature on FEC A01 tpc-feemonitor.sh --fec=A01 --data-point=TEMP
Unfortunately the script interpreter on the ARM linux is quite slow, so the processing takes a little while before the actual read-out starts. Taking all FECs is only suitable if you want to start a longer monitoring process.
The tool can be used to create rcu-sh scripts for the read-out of individual or an assembly of FECs. This script can than simply be re-used on and on with the rcu-sh, e.g.
rcu-sh b <script path> -s -l 5 -w 1 sec
Note: The tool doesn't switch on the continuous conversion on the FEC board controllers. The default value for the TEMP data point i 40. If this doesn't change, you should suspect, that the continuous conversion is off.
# broad-cast to all FECs to switch on continuous conversion rcu-sh w 0xc411 0x7ff
FeeServer
If the network disks can be mounted, the FeeServer starts at bootup. It gets the hostname of the board as server name.
Provided services
Each FeeServer publishes the following the services:
- MAIN_STATE
- XX_STATE
- XX_TEMP
- XX_AV
- XX_AC
- XX_DV
- XX_DC,
where XX denotes the FEC. Branch A FECs 00 to 12, Branch B FECs 16 to 27.
A FEC can be activated or deactivated by sending a high level command to the FeeServer:
<fee> FEESVR_SET_FERO_DATA32 XX_STATE 0/1 </fee>
This issues the FEESVR_SET_FERO_DATA command to the channel XX_STATE, where XX has to be replaced by the FEC number and either 1 or 0 have to be used to enable and disable the FEC respectively. E.g. with the feeserver-ctrl
echo '<fee> FEESVR_SET_FERO_DATA32 00_STATE 1 </fee>' | feeserver-ctrl --server tpc-fee_0_00_0
Command line tool for sending of commands to a FeeServer
The feeserver-ctrl program is ment to test and debug the command execution on the FeeServer and the RCU sequencer. The tool reads from std input or a file and writes the result to std output or a file. Sending a configuration can be done by
cat my_configuration | feeserver-ctrl --server <servername> # e.g. cat my_configuration | feeserver-ctrl --server tpc-fee_0_00_0
There are further options to send control commands to the FeeServer CE, try
feeserver-ctrl --help feeserver-ctrl --ce-command help
A few ce commands are picked out because of their importance during debugging, replace <server name> by the name of the server you want to connect to:
# set the logging level of the ControlEngine # level 0: all # level 1: debug or higher (essentially the same as all) # level 2: info or higher (default) # level 3: warning or higher feeserver-ctrl --server <server name> --ce-command CE_SET_LOGGING_LEVEL <level>
# enable the service update feeserver-ctrl --server <server name> --ce-command CEDBG_EN_SERVICE_UPDATE 1
# disable the service update feeserver-ctrl --server <server name> --ce-command CEDBG_EN_SERVICE_UPDATE 0
Log files of the FeeServers
The FeeServers publishs all messages on the message channel provided by the FEE API. In addition, logging messages are written to log files. They are available on the master node at /dcs-share/fee-log/<server name>. To listen for incoming messages do
tail -f <file name>
Setting the logging level of the FeeServer CE to a higher verbosity can be done by
feeserver-ctrl --server <server name> --ce-command CE_SET_LOGGING_LEVEL 0
Configuration of the Name server (DNS)
The DNS provides the hostname to the DCS board. The hostname is automatically used as FeeServer name. The hostnames are built from 3-dimensional geographical addresses, e.g. tpc-fee_0_04_3.
To change the DNS configuration create a simple routing file containing board number and geographical address, e.g.
# sample configuration for the TPC
# boardno side sector partition
11 1 6 4
1 1 0 5
10 0 0 0
12 0 1 5
13 1 7 3
105 0 9 5
and configure the fee-network
fee-names.sh --configure-named --add-configuration <config file>
where <config file> has to be replaced by the file name.
Further information can be found on the DCS network page.
The RCU Gui
The RCU Gui developed by Christian H. Christensen is installed on the server machine. The Gui can connect to one FeeServer at a time, but several Gui's can be started at the same time. To start the Gui logon the server machine as user fee and type:
rcugui fee://<servername>
where <servername> has to be replaced by the name of the FeeServer you want to connect to, e.g.
rcugui fee://dcs0080
The Gui provides comfortable access to the RCU memories as well as to the FECs.
The setup in the RCU lab
A laptop was used as server machine (master). The built-in network (eth0) is used for the uplink to the CERN network. The hostname is tpcfee02.cern.ch with IP address 137.138.54.220.
The setup for the clean room
An identical system was set up on a Linux PC providing to Ethernet interfaces. The machine has the hostname tpcfee01.cern.ch and is currently configured with IP :137.138.253.151 for the RCU lab. After completion and testing of the setup in the lab it will be moved to the clean room together with the big switch and one of the Windows machines from the DCS group intended to host the PVSS.