DAQ Message Viewer Errors

Here, you'll find a selection of commonly encountered DAQ errors, and how to fix them.

Nevis TPC Header Out of Sync

Bad NTB Timestamp

If there is a problem with the NTB timestamps, you might see an error like this pop up:

EventBuilder7_p2 | %MSG-e EventBuilder7_p2_SharedMemoryEventManager:  Early  04-May-2025 12:00:12 CDT pre-events SharedMemoryEventManager.cc:1144
Event 439 has already been marked "Incomplete" and sent to art!
EventBuilder7_p2 | %MSG-e EventBuilder7_p2_SharedMemoryEventManager:  Early  04-May-2025 12:00:12 CDT pre-events SharedMemoryEventManager.cc:236
Dropping fragment with sequence id 439 and fragment id 37123 because data taking has already passed this event.

In this case the NTB is what is "pushing" the DAQ (as a DAQ expert, you can change what component pushes). Once the NTB timestamps go bad, the EVBs can't build evenets because there are no NTB timestamps.

The DAQ will run for a new minutes with some errors, including:

• Missing fragments from all other subsystems

• SBNDGateFilter cannot find HLT

• Causality timestamp issues

Eventually, the DAQ will crash or be stopped by the auto-stop script due to missing fragments.

Some symptoms that can be checked in the EVB logs are:

1. Error: Missing all other fragments except the NTB fragment. Cause: Bad NTB timestamp.

3662 Event 1062: total_fragments=1, fragmentIDs for 330 missing_fragments: 8, 37123, 37124, 37125, 37126, 37127, 37128, 37129, 37130, 37131, 
     37132, 37133, 37134, 37135, 37136, 37137, 37138, 37379, 37380, 37381, 37382, 37383, 37384, 37385, 37386, 37387, 37388, 37389, 37390, 373
     91, 37392, 37393, 37394, 37635, 37636, 37637, 37638, 37639, 37640, 37641, 37642, 37643, 37644, 37645, 37646, 37647, 37648, 37649, 37650,
      37891, 37892, 37893, 37894, 37895, 37896, 37897, 37898, 37899, 37900, 37901, 37902, 37903, 37904, 37905, 37906, 38147, 38148, 38149, 38
     150, 38151, 38152, 38153, 38154, 38155, 38156, 38157, 38158, 38159, 38160, 38161, 38162, 38403, 38404, 38405, 38406, 38407, 38408, 38409
     , 38410, 38411, 38412, 38413, 38414, 38415, 38416, 38417, 38418, 38659, 38660, 38661, 38662, 38663, 38664, 38665, 38666, 38667, 38668, 3
     8669, 38670, 38671, 38672, 38673, 38674, 38915, 38916, 38917, 38918, 38919, 38920, 38921, 38922, 38923, 38924, 38925, 38926, 38927, 3892
     8, 38929, 39171, 39172, 39173, 39174, 39175, 39176, 39177, 39178, 39179, 39180, 39181, 39182, 39183, 39184, 39185, 39186, 39427, 39428, 
     39429, 39430, 39431, 39432, 39433, 39434, 39435, 39436, 39437, 39438, 39439, 39440, 39441, 39442, 39683, 39684, 39685, 39686, 39687, 396
     88, 39689, 39690, 39691, 39692, 39693, 39694, 39695, 39696, 39697, 39698, 40960, 40961, 40962, 40963, 40964, 40965, 40966, 40967, 40968,
      41223, 41229, 41232, 41235, 45133, 45134, 45138, 45139, 45140, 45142, 45143, 45146, 45147, 45149, 45150, 45151, 45153, 45154, 45155, 45
     156, 45157, 45158, 45159, 45160, 45336, 45345, 45346, 45368, 45369, 45371, 45372, 45373, 45478, 45480, 45481, 45482, 45483, 45484, 45485
     , 45488, 45626, 45641, 45656, 45702, 45703, 45704, 45717, 45718, 45719, 45720, 45721, 45723, 45724, 45725, 45726, 45727, 45750, 45806, 4
     5843, 45866, 45869, 45879, 45944, 45954, 45962, 45972, 45987, 45988, 45989, 45998, 46021, 46022, 46023, 46026, 46027, 46031, 46098, 4612
     4, 46159, 46161, 46165, 46211, 46212, 46213, 46223, 46226, 46227, 46240, 46242, 46280, 46284, 46286, 46300, 46302, 46352, 46371, 46372, 
     46384, 46386, 46387, 46407, 46411, 46441, 46442, 46443, 46447, 46450, 46451, 46454, 46455, 46459, 46461, 46462, 46465, 46481, 46523, 465
     41, 46559, 46564, 46631, 46632, 46639, 46641, 46664, 46701, 46708, 46709, 46713, 46719, 46720, 46733, 46736, 46770, 46771, 46772, 46775,
      46776, 46777, 46801, 46802, 46803, 46804, 46805, 46808, 55099

1. Error: Cannot find HLT reported by SBNDGateFilter module. Cause: The PTB fragment is missing.

3658 %MSG-e _TRACE_:  SBNDGateFilter:offbeamlightfilter@BeginModule  04-May-2025 12:18:40 CDT run: 18465 subRun: 1 event: 1062 SBNDGateFilter_module.cc:202
3659 Run 18465, subrun 1, event 1062 has zero HLT word Fragments in module, not separating by beam type!

How to fix this

1. Reset the NTB using Phoebus (insert link to instructions for Phoebus here)

2. Kill the DAQ

3. Restart the run in the shifter GUI (write some notes for this!)