Control and monitoring software of LHAASO DAQ

Purpose The Large High Altitude Air Shower Observatory (LHAASO) is a hybrid extensive air shower (EAS) array. It consists of 3 types of detector arrays and about 9500 detector units in total. Status information generated by data acquisition system (DAQ), including signal count rates of detectors, is stored temporarily in Redis and updated periodically. Redis is an open-source in-memory database. The control interface of DAQ’s data flow software is also based on Redis. According to the monitoring and control requirements, status data from DAQ system need to be stored. Overview status of data flow software and detectors should be displayed. For automatic operation mode, common software and hardware faults should be detected and recovered automatically.
Methods Status information collection, analysis and automatic operation control are implemented in multiple services. The information collection programs collect the operation status data from the DAQ and monitor the operation status of the computing nodes. All information is cached in Kafka. Kafka is an open-source distributed event streaming platform. The DAQ data flow controller program obtains the status data from Kafka, analyzes the operation status, and tries to recover the fault by modifying the DAQ configuration file and restarting DAQ. In this architecture, the automatic control logic can be easily updated by modifying and restarting a single service. Finally, the monitoring data are structured and stored in MongoDB.
Results and Conclusions The software can monitor the detectors, electronics and computing nodes in LHAASO, save monitoring data in MongoDB, and control the DAQ data flow software. When a fault occurs, it can restart the DAQ data flow software. The software meets the auto-control operation requirements of LHAASO.