Important Things to Check

On this page, you'll find a selection of things you should keep an eye out for when you're on expert shift.

Empty fragments

One way you can tell a run may be bad or about to crash is the percentage of runs with empty fragments. Usually, this should only be a few percent, but if it is greater than around 20%, ping the Run Co and let them know. The image below shows you which part of Granana you need to look at to see the percentage of runs with empty fragments (boxed in pink):

The DAQ Shifter panel on Grafana, with the "Events with empty fragments [%]" display shown in the middle left.

FTrig/ETrig ratio

The FTrig/ETrig ratio is another important metric to keep an eye on when on shift. This can be found in the "TDC Sample Rate" box on the top left (highlighted in red below).

The DAQ Shifter panel on Grafana, with the "TDC Sample Rate" display shown in the top right.

The ratio is the final selection on the bottom panel, which, when selected, looks as seen below:

The FTrig/ETrig ratio selected in Grafana.

This should be stable throughout the run (and climbing to a stable rate when the run is starting).

If the FTrig/ETrig ratio begins to spike, or ride/fall above/below the expected number given by the trigger experts, notify them as soon as possible!

Run records

Sometimes, if a run crashes or you suspect there's a problem with a run, you should look at the run records. You can cd into the directory below to find them:

cd /daq/run_records/

If you ssh onto a DAQ machine, you can find the FHiCL configuration settings for a given run by navigating to

cd /daq/run_records/<runnumber>

If you ls within the directory, you should see something similar to what I've popped down below:

boot.txt           EventBuilder3.fcl            ranks.txt     tpc03.fcl  tpc08.fcl   wib102.fcl  wib201.fcl  wib206.fcl  wib305.fcl  wib404.fcl
Dispatcher1.fcl    known_boardreaders_list.txt  settings.txt  tpc04.fcl  tpc09.fcl   wib103.fcl  wib202.fcl  wib301.fcl  wib306.fcl  wib405.fcl
environment.txt    mbb02.fcl                    setup.txt     tpc05.fcl  tpc10.fcl   wib104.fcl  wib203.fcl  wib302.fcl  wib401.fcl  wib406.fcl
EventBuilder1.fcl  metadata.txt                 tpc01.fcl     tpc06.fcl  tpc11.fcl   wib105.fcl  wib204.fcl  wib303.fcl  wib402.fcl
EventBuilder2.fcl  ntb01.fcl                    tpc02.fcl     tpc07.fcl  wib101.fcl  wib106.fcl  wib205.fcl  wib304.fcl  wib403.fcl

To find the name of the database configuration that was run, what detector components were used, and the paths to all the log messages, you can open metadata.txt, which will provide you with all this information!

You can also find the log files in

/daq/log

but it's faster to find the logs in metadata.txt.

Old log files get moved to

/daq/log_archive

if you ever need to find them!