Get readout box up and running

Hardware requirements

• Focal read-out box
• Linux computer with Gb network capability for taking data
• Windows PC for software/firmware debugging
• One or more patch panel PCBs
• One or more flex PCBs with mounted MIMOSA ASICs

Software requirements and drivers

• Xilinx Lab tools v13.2, including XMD, iMPACT and ChipScope
• Serial console software like PuTTY
• TFTP server for Linux kernel downloading

Configure the FPGA(s)

Using XMD

XMD could run from SDK or a normal cmd command window, for the later we need to set up the path environment firstly with batch file "settings32.bat"("settings64.bat" for a 64 bit PC) from the Xilinx software installation directory like "D:\Xilinx\13.2\ISE_DS\settings32.bat", for Linux, there are corresponded shell files. Copy the file to your work directory where firmware, boot-loader and Linux kernel are, replace the XIL_SCRIPT_LOC variable with an absolute path:

set XIL_SCRIPT_LOC=D:\\Xilinx\\13.2\\ISE_DS\\

Open a cmd window, switch to your work directory, run:

O:\phase1\xilinx\test> settings32.bat
O:\phase1\xilinx\test> xmd

After entering into XMD prompt window, using "fpga -f" command to program Virtex FPGA:

XMD% fpga -f system.bit

If there are more than one device in the JTAG chain, use "-debugdevice devicenr" to specify the component to be programmed, here we need to configure one Virtex6 and two Spartan6 FPGAs:

XMD% fpga -f spartan6u1.bit -debugdevice devicenr 1
XMD% fpga -f spartan6u2.bit -debugdevice devicenr 2
XMD% fpga -f system.bit -debugdevice devicenr 4

Using iMPACT

Run iMPACT from windows Start -> Xilinx ISE Design Suite 13.2 -> ISE Design Tools -> Tools -> iMPACT, or from the cmd window opened above,

XMD% impact

In the iMPACT window, select File -> Initialize Chain, then you will find the devices in the JTAG chain on the "Boundary Scan" window, right click on the FPGA you need to program, Assign a configuration file, then program it following the commands.

Using Chipscope

Sometimes we need chipscope to look inside FPGA by probing some of the internal signals, then it's convenient to program these FPGAs in Chipscope Analyzer. There is a chipscope project file together with a Chipscope definition and connection file - "chipscope.cpj" and "chipscope.cdc". Open the project file from Chipscope pro Analyzer version 13.2 or newer, after opening JTAG cable, all device in the JTAG chain will be shown in the left navigation window, right click on a device and select configure, then operate as indicated to program FPGA and run chipscope.

Configure TFTP server

Open TFTP Server, select File -> Properties..., in "Preferences" page, enable "Allow Read Requests", in the "Directories" page, add the directory where the Linux kernel is placed. This configration is only needed at the first time.

Connect to debug module on Processor Bus

In the XMD prompt window, you can connect to the Microblaze debug module to download and run your bootloader:

 XMD% connect mb mdm -debugdevice devicenr 4
 XMD% dow u-boot.elf
 XMD% run

Open a serial console like Putty, select the serial port which you can find from you windows Device Manager -> Ports(COM & LPT) -> Silicon Labs CP210x USB to UART Bridge(COM4), then com4 is the one we need to use, set Baud rate 115200, No Parity, 8 data bits, 1 stop bit.

In the U-Boot command prompt window of Putty, modify the Virtex MAC/IP address and the TFTP server ip address at the first time, the MAC address could be found from the top side of Virtex6 DEV board, if there is a binding between the MAC and IP address, ask you network administrator for help, if the TFTP server and the readout box are in a private network, just pick a free IP address.

U-Boot-PetaLinux> setenv ethaddr 00:0a:35:02:31:1f
U-Boot-PetaLinux> setenv ipaddr 129.177.39.131
U-Boot-PetaLinux> setenv serverip 129.177.39.230
U-Boot-PetaLinux> saveenv

Make sure the TFTP server is running and the kernel image is in the right directory, normally you should be able to ping each other. Then download the linux kernel from TFTP server:

U-Boot-PetaLinux> tftp

After the image being successfully downloaded, Boot Linux kernel:

U-Boot-PetaLinux> bootm

If you want to burn the kernel image to the on-board flash memory, run:

U-Boot-PetaLinux> run update_kernel

Running PetaLinux

After PetaLinux boots up, log in and set up network with your Virtex MAC/IP:

~ # ifconfig eth0 hw ether 00:0a:35:02:31:1f
~ # ifconfig eth0 129.177.39.131

If your network support dhcp:

~ # udhcpc

To enable SSH service:

~ # dropbear

To set up jumbo frame support for data shipment

~ # ifconfig eth0 down
~ # ifconfig eth0 mtu 9000
~ # ifconfig eth0 txqueuelen 10000
~ # /sbin/sysctl -w net.ipv4.tcp_sack=0
~ # ifconfig eth0 up

Before running the DAQ software, if the device driver has not been compiled into Linux kernle, then manually load it and make new device nodes in the /dev directory:

~ # /home/uib/load.sh

All of the above configurations are now being put into an initiative script file, you can run it after logging in the system:

~ # /home/test/init.sh

The above steps can also be done from a telnet terminal, after being initiated, the ssh server is running, you can connect the box from another pc with ssh connection.

Running DAQ

Start the server side DAQ software in the Linux DAQ server firstly, it will be waiting there for a connection:

$ ./server

Run the client side DAQ software:

~ # focal 0-3

The argument specifies the operation mode of the readout box:

• 0: Virtex test patterns as the input of DMA engine with FIFOs being bypassed, for every 256-bit word, there is a 32-bit counter and some other fixed long word patterns, as below:

Data in [0x67000000..0x6700001f): 01 00 00 00 44 33 22 11 BB CC DD EE CC BB AA 99 00 FF EE DD 11 11 11 11 22 22 22 22 33 33 33 33
Data in [0x67000020..0x6700003f): 02 00 00 00 44 33 22 11 BB CC DD EE CC BB AA 99 00 FF EE DD 11 11 11 11 22 22 22 22 33 33 33 33

• 1: Virtex test patterns as the FIFO input, they are the test signals grouped by 96 bits(one 32-bit counter and two long word patterns) for each FIFO. Every 256-bit word is divided into two parts from the two FIFOs respectively, as below:

Data in [0x67000000..0x6700001f): 41 91 01 00 44 33 22 11 BB CC DD EE 42 91 01 00 | 41 91 01 00 CC BB AA 99 00 FF EE DD 42 91 01 00
Data in [0x67000020..0x6700003f): 44 33 22 11 BB CC DD EE 43 91 01 00 44 33 22 11 | CC BB AA 99 00 FF EE DD 43 91 01 00 CC BB AA 99
Data in [0x67000040..0x6700005f): BB CC DD EE 44 91 01 00 44 33 22 11 BB CC DD EE | 00 FF EE DD 44 91 01 00 CC BB AA 99 00 FF EE DD

• 2: Spartan test patterns as the FIFO input, they are the test signals grouped by 96 bits(one 32-bit counter plus ASCII codes of "SPARTAN1" or "SPARTAN2") for each FIFO. They are very similar as in mode 1 except that these signals are from Spartan FPGAs, as below:

Data in [0x67000000..0x6700001f): 42 91 01 00 31 4E 41 54 52 41 50 53 43 91 01 00 | 42 91 01 00 32 4E 41 54 52 41 50 53 43 91 01 00
Data in [0x67000020..0x6700003f): 31 4E 41 54 52 41 50 53 44 91 01 00 31 4E 41 54 | 32 4E 41 54 52 41 50 53 44 91 01 00 32 4E 41 54
Data in [0x67000040..0x6700005f): 52 41 50 53 45 91 01 00 31 4E 41 54 52 41 50 53 | 52 41 50 53 45 91 01 00 32 4E 41 54 52 41 50 53

• 3: The real data mode, at the input side of the two FIFOs, there are 96 bits from the 96 Mimosa data channels, at the output side of the FIFOs, the data is readout in 128 bits and the two FIFO outputs are connected together to form a 256-bit AXI4 word. With only the first 4 chips(16 channels) of the first Spartan FPGA being enabled and the other Spartan FPGA working in mode 2, the readout data format is as below:

Data in [0x67000000..0x6700001f): 3F 3F 00 00 00 00 00 00 00 00 00 00 3F FF 00 00 | 41 91 01 00 32 4E 41 54 52 41 50 53 42 91 01 00
Data in [0x67000020..0x6700003f): 00 00 00 00 00 00 00 00 3F FF 00 00 00 00 00 00 | 32 4E 41 54 52 41 50 53 43 91 01 00 32 4E 41 54
Data in [0x67000040..0x6700005f): 00 00 00 00 7F 7F 00 00 00 00 00 00 00 00 00 00 | 52 41 50 53 44 91 01 00 32 4E 41 54 52 41 50 53

DAQ Configurations

Slow control setup

The slow control parameters are read in from a configration file - "/home/test/parameters.txt". If this file can't be found/opened or errors occur during reading time, the default values which are hard coded will be selected.

0x00000F  // U1_CHIP_ENABLE        1 bit for 1 chip [23 downto 0].  24 bits, HEX
0x000     // U1_READOUT_CLK_SRC    1 bit for 1 layer[35 downto 24]. 12 bits, HEX
0x0       // U1_FRAME_SYNC_SRC     use the MK_CLKD from the 1st chip as the source of frame_sync signal, [38 downto 36], 3 bits
0x1       // U1_PATTERN_LINE_EN    enable pattern line for MAPS.   1 bit, [39]
0x0       // U1_PATTERN_MODE       pattern mode selection, 0: id/cnt, 1: toggle values(55/aa). [40]
0x1FF     // U1_BKUP               9 bits for future use. [49 downto 41], HEX
0x000000  // U2_CHIP_ENABLE        1 bit for 1 chip [23 downto 0], HEX
0x000     // U2_READOUT_CLK_SRC    1 bit for 1 layer, HEX
0x0       // U2_FRAME_SYNC_SRC     use the MK_CLKD from the 1st chip as the source of frame_sync signal
0x1       // U2_PATTERN_LINE_EN    enable pattern line for MAPS.
0x0       // U2_PATTERN_MODE       pattern mode selection, 0: id/cnt, 1: toggle values(55/aa).
0x1FF     // U2_BKUP               9 bits for future use, HEX

Mimosa JTAG configuration

Before DAQ starts to read out real detector data, all Mimosa asics need to be configured. This is done by a Linux program - "playxsvf", it reads in JTAG configurations from an xsvf format file and generates JTAG pin signals which are connected to a Spartan FPGA, these signals are then multiplexed to different patch pannel pcbs according to the address specified. To run this program:

~ # playxsvf
XSVF Player v5.01, Xilinx, Inc.
USAGE:  playxsvf [-l layer] filename.xsvf
where:  -l layer      = layer, no = 0-15 (default=0)
        filename.xsvf = the XSVF file to execute.

~ # playxsvf -l 1 /home/xsvf/pattern_4chips.xsvf

Four Mimosa chips in one detector layer share the same JTAG interface, for different number of chips in the JTAG chain, there are different xsvf files, the position of the working chips in the JTAG also matters if not all chips are working. The enable/disable of the chips for JTAG configuration is determined by the slow control parameter "CHIP_ENABLE", if one chip is disabled, then it will also be bypassed from the JTAG chain.

Threshold voltages setup

Normally the threshold voltages are set chip by chip with the values in the xsvf files. To change the values we need to regenerate the corresponded xsvf files with SVF to XSVF converter: "svf2xsvf502.exe". To automatically scan these voltages, it is possible to hack into the xsvf files and change the corresponded binary values and run it with xsvfplayer looply, but it is difficult when there are some chips not working inside a JTAG chain because there are different bit shifts inside the xsvf file for different chip position. This part is still under investigation.