Get readout box up and running

Overview

This page contains step by step informations to setup the Focal readout electronics for MAPS detectors.

Hardware requirements

• Focal read-out box
• Linux computer with Gb network capability for taking data
• Windows PC for software/firmware debugging
• One or more patch panel PCBs
• One or more flex PCBs with mounted MIMOSA ASICs

Software requirements and drivers

• Xilinx Lab tools v13.2, including XMD, iMPACT and ChipScope
• Serial console software like PuTTY
• TFTP server for Linux kernel downloading

Configure the FPGA(s)

Using XMD

XMD could run from SDK or a normal cmd command window, for the later we need to set up the path environment firstly with batch file "settings32.bat"("settings64.bat" for a 64 bit PC) from the Xilinx software installation directory like "D:\Xilinx\13.2\ISE_DS\settings32.bat", for Linux, there are corresponded shell files. Copy the file to your work directory where firmware, boot-loader and Linux kernel are, replace the XIL_SCRIPT_LOC variable with an absolute path:

set XIL_SCRIPT_LOC=D:\\Xilinx\\13.2\\ISE_DS\\

Open a cmd window, switch to your work directory, run:

O:\phase1\xilinx\test> settings32.bat
O:\phase1\xilinx\test> xmd

After entering into XMD prompt window, using "fpga -f" command to program Virtex FPGA:

XMD% fpga -f system.bit

If there are more than one device in the JTAG chain, use "-debugdevice devicenr" to specify the component to be programmed, here we need to configure one Virtex6 and two Spartan6 FPGAs:

XMD% fpga -f spartan6u1.bit -debugdevice devicenr 1
XMD% fpga -f spartan6u2.bit -debugdevice devicenr 2
XMD% fpga -f system.bit -debugdevice devicenr 4

Using iMPACT

Run iMPACT from windows Start -> Xilinx ISE Design Suite 13.2 -> ISE Design Tools -> Tools -> iMPACT, or from the cmd window opened above,

XMD% impact

In the iMPACT window, select File -> Initialize Chain, then you will find the devices in the JTAG chain on the "Boundary Scan" window, right click on the FPGA you need to program, Assign a configuration file, then program it following the commands.

Using Chipscope

Sometimes we need chipscope to look inside FPGA by probing some of the internal signals, then it's convenient to program these FPGAs in Chipscope Analyzer. There is a chipscope project file together with a Chipscope definition and connection file - "chipscope.cpj" and "chipscope.cdc". Open the project file from Chipscope pro Analyzer version 13.2 or newer, after opening JTAG cable, all device in the JTAG chain will be shown in the left navigation window, right click on a device and select configure, then operate as indicated to program FPGA and run chipscope.

Configure TFTP server

Open TFTP Server, select File -> Properties..., in "Preferences" page, enable "Allow Read Requests", in the "Directories" page, add the directory where the Linux kernel is placed. This configration is only needed at the first time.

Microblaze debug module and Boot loader

In the XMD prompt window, you can connect to the Microblaze debug module to download and run your bootloader:

 XMD% connect mb mdm -debugdevice devicenr 4
 XMD% dow u-boot.elf
 XMD% run

If there is something wrong, like no response from the console, you can restart the system without reconfiguring Virtex6 FPGA:

 XMD% stop
 XMD% run

Open a serial console like Putty, select the serial port which you can find from you windows Device Manager -> Ports(COM & LPT) -> Silicon Labs CP210x USB to UART Bridge(COM4), then com4 is the one we need to use, set Baud rate 115200, No Parity, 8 data bits, 1 stop bit.

In the U-Boot command prompt window of Putty, modify the Virtex MAC/IP address and the TFTP server ip address at the first time, the MAC address could be found from the top side of Virtex6 DEV board, if there is a binding between the MAC and IP address, ask you network administrator for help, if the TFTP server and the readout box are in a private network, just pick a free IP address.

U-Boot-PetaLinux> setenv ethaddr 00:0a:35:02:31:1f
U-Boot-PetaLinux> setenv ipaddr 129.177.39.131
U-Boot-PetaLinux> setenv serverip 129.177.39.230
U-Boot-PetaLinux> saveenv

Make sure the TFTP server is running and the kernel image is in the right directory, normally you should be able to ping each other. Then download the linux kernel from TFTP server:

U-Boot-PetaLinux> tftp

After the image being successfully downloaded, Boot Linux kernel:

U-Boot-PetaLinux> bootm

If you want to burn the kernel image to the on-board flash memory, run:

U-Boot-PetaLinux> run update_kernel

Running PetaLinux

After PetaLinux boots up, log in and set up network with your Virtex MAC/IP:

~ # ifconfig eth0 hw ether 00:0a:35:02:31:1f
~ # ifconfig eth0 129.177.39.131

If your network support dhcp:

~ # udhcpc

To enable SSH service:

~ # dropbear

To set up jumbo frame support for data shipment

~ # ifconfig eth0 down
~ # ifconfig eth0 mtu 9000
~ # ifconfig eth0 txqueuelen 10000
~ # /sbin/sysctl -w net.ipv4.tcp_sack=0
~ # ifconfig eth0 up

Before running the DAQ software, if the device driver has not been compiled into Linux kernle, then manually load it and make new device nodes in the /dev directory:

~ # /home/uib/load.sh

All of the above configurations are now being put into an initiative script file, you can run it after logging in the system:

~ # /home/test/init.sh

The above steps can also be done from a telnet terminal, after being initiated, the ssh server is running, you can connect the box from another pc with ssh connection.

Startup scripts

For the automatic running of the user initialization scripts, you can't add them directly to the romfs script files like /etc/rc.sysinit, because there files are generated every time during Petalinux compilation and all custom modification will be overwritten. The source files for these initialization scrips are located in "~petalinux-dist/user/sys_init/data/etc/rc", normally you should put your custom startup commands in "start" file, but according to the Makefile, this file doesn't work properly at the moment, so we add our scripts to the end of "sysinit" file:

echo "Running user init scripts." 
/home/test/init.sh

After running "make romfs", you will find that these scripts are added into the romfs file "/etc/rc.sysinit".

Running DAQ

Start the server side DAQ software in the Linux DAQ server firstly, it will be waiting there for a connection:

$ ./server

Run the client side DAQ software:

~ # focal 0-3

The argument specifies the operation mode of the readout box:

• 0: Virtex test patterns as the input of DMA engine with FIFOs being bypassed, for every 256-bit word, there is a 32-bit counter and some other fixed long word patterns, as below:

Data in [0x67000000..0x6700001f): 01 00 00 00 44 33 22 11 BB CC DD EE CC BB AA 99 00 FF EE DD 11 11 11 11 22 22 22 22 33 33 33 33
Data in [0x67000020..0x6700003f): 02 00 00 00 44 33 22 11 BB CC DD EE CC BB AA 99 00 FF EE DD 11 11 11 11 22 22 22 22 33 33 33 33

• 1: Virtex test patterns as the FIFO input, they are the test signals grouped by 96 bits(one 32-bit counter and two long word patterns) for each FIFO. Every 256-bit word is divided into two parts from the two FIFOs respectively, as below:

Data in [0x67000000..0x6700001f): 41 91 01 00 44 33 22 11 BB CC DD EE 42 91 01 00 | 41 91 01 00 CC BB AA 99 00 FF EE DD 42 91 01 00
Data in [0x67000020..0x6700003f): 44 33 22 11 BB CC DD EE 43 91 01 00 44 33 22 11 | CC BB AA 99 00 FF EE DD 43 91 01 00 CC BB AA 99
Data in [0x67000040..0x6700005f): BB CC DD EE 44 91 01 00 44 33 22 11 BB CC DD EE | 00 FF EE DD 44 91 01 00 CC BB AA 99 00 FF EE DD

• 2: Spartan test patterns as the FIFO input, they are the test signals grouped by 96 bits(one 32-bit counter plus ASCII codes of "SPARTAN1" or "SPARTAN2") for each FIFO. They are very similar as in mode 1 except that these signals are from Spartan FPGAs, as below:

Data in [0x67000000..0x6700001f): 42 91 01 00 31 4E 41 54 52 41 50 53 43 91 01 00 | 42 91 01 00 32 4E 41 54 52 41 50 53 43 91 01 00
Data in [0x67000020..0x6700003f): 31 4E 41 54 52 41 50 53 44 91 01 00 31 4E 41 54 | 32 4E 41 54 52 41 50 53 44 91 01 00 32 4E 41 54
Data in [0x67000040..0x6700005f): 52 41 50 53 45 91 01 00 31 4E 41 54 52 41 50 53 | 52 41 50 53 45 91 01 00 32 4E 41 54 52 41 50 53

• 3: The real data mode, at the input side of the two FIFOs, there are 96 bits from the 96 Mimosa data channels, at the output side of the FIFOs, the data is readout in 128 bits and the two FIFO outputs are connected together to form a 256-bit AXI4 word. With only the first 4 chips(16 channels) of the first Spartan FPGA being enabled and the other Spartan FPGA working in mode 2, the readout data format is as below:

Data in [0x67000000..0x6700001f): 3F 3F 00 00 00 00 00 00 00 00 00 00 3F FF 00 00 | 41 91 01 00 32 4E 41 54 52 41 50 53 42 91 01 00
Data in [0x67000020..0x6700003f): 00 00 00 00 00 00 00 00 3F FF 00 00 00 00 00 00 | 32 4E 41 54 52 41 50 53 43 91 01 00 32 4E 41 54
Data in [0x67000040..0x6700005f): 00 00 00 00 7F 7F 00 00 00 00 00 00 00 00 00 00 | 52 41 50 53 44 91 01 00 32 4E 41 54 52 41 50 53

DAQ Configurations

Slow control setup

The slow control setup is necessay for proper DAQ operations and it is done in the initialization part of the program before starting DAQ. All the parameters are read in from a configration file - "/home/test/parameters.txt". If this file can't be found/opened or errors occur during reading time, the default values which are hard coded will be selected.

0x00000F  // U1_CHIP_ENABLE        1 bit for 1 chip [23 downto 0].  24 bits, HEX
0x000     // U1_READOUT_CLK_SRC    2 bits for 1 layer[35 downto 24]. 12 bits, HEX
0x0       // U1_FRAME_SYNC_SRC     use the MK_CLKD from the 1st layer as the source of frame_sync signal, [38 downto 36], 3 bits
0x1       // U1_PATTERN_LINE_EN    enable pattern line for MAPS.   1 bit, [39]
0x0       // U1_PATTERN_MODE       pattern mode selection, 0: id/cnt, 1: toggle values(55/aa). [40]
0x1FF     // U1_BKUP               9 bits for future use. [49 downto 41], HEX
0x000000  // U2_CHIP_ENABLE        2 bits for 1 chip [23 downto 0], HEX, LSB = chip 0
0x000     // U2_READOUT_CLK_SRC    1 bit for 1 layer, HEX
0x0       // U2_FRAME_SYNC_SRC     use the MK_CLKD from the 1st layer as the source of frame_sync signal
0x1       // U2_PATTERN_LINE_EN    enable pattern line for MAPS.
0x0       // U2_PATTERN_MODE       pattern mode selection, 0: id/cnt, 1: toggle values(55/aa).
0x1FF     // U2_BKUP               9 bits for future use, HEX

Notes:

• Chip_enable: 1=enable;0=disable. LSB = chip 1;
• Readout_clk_src: 0b00=chip1(J10); 0b10=chip2(J11); 0b01=chip3(J13); 0b11=chip4(J12). LSB = Layer 1
• Frame_sync_src: 0=layer1...5=layer6. Make sure there is a working MK_CLKD signal for the selected layer.

Slow control parameters will be saved inside Spartan FPGAs, so normally only one time of slow control operation is needed.

Mimosa JTAG configuration

Before DAQ starts to read out real detector data, all Mimosa asics need to be configured. This is done by a Linux program - "playxsvf", it reads in JTAG configurations from an xsvf format file and generates JTAG pin signals which are connected to a Spartan FPGA, these signals are then multiplexed to different patch pannel pcbs according to the address specified. To run this program:

~ # playxsvf
XSVF Player v5.01, Xilinx, Inc.
USAGE:  playxsvf [-l layer] filename.xsvf
where:  -l layer      = layer, no = 0-15 (default=0)
        filename.xsvf = the XSVF file to execute.

~ # playxsvf -l 1 /home/xsvf/pattern_4chips.xsvf

Layer number:

• 1~6 for detector layer 1 to 6, 7 for all, these 6 layers are connected to Spartan U1 via the upper fanout board.
• 9~14 for detector layer 7 to 12, 15 for all, these 6 layers are connected to Spartan U2 via the lower fanout board.

Four Mimosa chips in one detector layer share the same JTAG interface, they are configured one by one with others being bypassed, for different number of chips in the JTAG chain, there are different xsvf files, the position of the working chips in the JTAG also matters if not all chips are working. The enable/disable of the chips for JTAG configuration is determined by the slow control parameter "CHIP_ENABLE", if one chip is disabled, then it will also be bypassed from the JTAG chain. So it is important to finish the slow control before running JTAG configuration. Normally for the first time after the Spartan FPGAs are programmed or there are any modifications for the slow control parameters, we need to run "focal" program with a pattern mode firstly, then run JTAG configurtion before we can read out real detector data finally.

All the JTAG configurations are saved inside Mimosa chips, so only one time of JTAG download is needed.

Threshold voltage setup

Normally the threshold voltages are set chip by chip with the values in the xsvf files. To change the values we need to regenerate the corresponded xsvf files with SVF to XSVF converter: "svf2xsvf502.exe". To automatically scan these voltages, it is possible to hack into the xsvf files and change the corresponded binary values and run it with xsvfplayer looply, but it is difficult when there are some chips not working inside a JTAG chain because there are different bit shifts inside the xsvf file for different chip position. This part is still under investigation.

Running Chipscope

After programming FPGAs and configuring Mimosa chips, if something doesn't work properly, chipscope is the effective way to look into FPGAs to debug firmwares. For the current system, the Virtex and the two Spartan FPGAs share the same JTAG download cable, so they can work together in the same Chispcope project. After opening the Chipscope project file and initializing the JTAG cable, we can find the Spartan U1 and virtex chipscope interfaces, by setting up proper trigger conditions we could get the waveform of all the signals we want.

• For Spartan U1, the chipscope signals include the frame_sync signal, the data ouput of the first chip(4 channels), the pattern verification results of these 4 channels and the total "not and" signal of all pattern match signals from the chips being enabled. If we trigger with "fram_sync_int", which is the signal for the start of the last line in a frame, we could see the pattern signal being programmed via JTAG, followed by the first several lines of data output from the first Mimosa chip, pattern_match value "1" means the pattern for current channel is matched. All match signals are "not anded" to an LED - "LED2_OBUF", if it is "0", all working chips have a matched pattern.

• For Virtex, there are signals related with the Focal unit and DMA engine, like the current counts of FIFOs, the FIFO status, external and internal(for test) frame_sync/spill_valid signals and so on. Normally it can be triggered with frame_sync, if you want to look know the status of FIFO, you can trigger with the FIFO_full/FIFO_empty or FIFO_count signals.

Auto boot the system

After the firmware/bootloader/Linux developments are finished, we can burn these files into the on-board flash memories, to make the system boot up automatically after power on. There are three different kinds of flash memories available, a 2GB CF card, a 16MB platform flash and a 32MB BPI linear flash. We can put the FPGA configuration file in CF card or in the platform flash, and the BPI flash is the one for bootloader and Linux image.

Program the CF card

The CF card supports up to 8 configuration files, which is selected by the S1 switch on the Virtex board. We can generate a new .ace file with iMPACT and overwrite one of the 8 files in CF card, then enable the SysACE CF load and give the correct SysACE address:

S1:
4 ON  (SysACE Mode = 1)   // Enable configuration load from CF card
3 OFF (SysACE Addr 2 = 0) // Three address bits, 0 means the first one(.ace file locates in /XILINX/cfg0/ directory in CF card)
2 OFF (SysACE Addr 1 = 0)
1 OFF (SysACE Addr 0 = 0)
S2:
6 OFF (FLASH_A23 = 0)
5 ON  (M2 = 1)            // JTAG mode
4 OFF (M1 = 0)
3 ON  (M0 = 1)
2 ON  (CS_SEL = 1)        // Select Linear flash access
1 OFF (EXT_CCLK = 0)

For the switch setups, please refer to ML605 Hardware User Guide.

Program the Platform flash

The platform flash supports in system programing, we can program it in iMPACT, with the .mcs file also generated in iMPACT. Please refer to Platform Flash XL Configuration and Storage Device. Switches should be set up as below:

S1:
4 OFF (SysACE Mode = 0)   // Disable configuration load from CF card
3 OFF (SysACE Addr 2 = 0) // Three address bits, don't care here
2 OFF (SysACE Addr 1 = 0)
1 OFF (SysACE Addr 0 = 0)
S2:
6 OFF (FLASH_A23 = 0)     // Don't care in this mode
5 ON  (M2 = 1)            // Slave SelectMAP mode
4 ON  (M1 = 1)
3 OFF (M0 = 0)
2 OFF (CS_SEL = 0)        // Select Platform flash access
1 ON  (EXT_CCLK = 1)      

Burn bootloader and Linux image

This should be done in u-boot prompt window, for bootloader, we need binary file "u-boot-s.bin", which can be find in the petalinux-dist/images/ directory, you need to copy this file together with the linux kernel image "image.ub" to the TFTP server root directory.

U-Boot-PetaLinux> run update_uboot
U-Boot-PetaLinux> run update_kernel

LED definitions

• D1: on: Test pattern out mode (mode 2) of Spartan1, off: normal mode;
• D2: on: All enabled channels have correct patterns, off: at least one channel(1/4 of a chip) of pattern mismatch;
• D3: flash: 1Hz from Spartan1 to Spartan2, maybe soldered up side down for some boxes;
• D4: As D1, for Spartan2;
• D5: As D2, for Spartan2;

Useful links

1. Getting Started with the Xilinx Virtex-6 FPGA ML605 Evaluation Kit
2. ML605 Restoring Flash Contents
3. Virtex-6 FPGA Configuration User Guide