EP-FXT workflow management system

Purpose The Einstein Probe (EP) satellite is equipped with the Wide-field X-ray Telescope (WXT) and the Follow-up X-ray Telescope (FXT). The EP-FXT Workflow Management System is part of the FXT-SDC and is responsible for managing the workflow involved in data product generation. It is utilized for the fully automated processing of FXT data within the pipeline for generating data products.
Methods The EP-FXT Workflow Management System invokes computing servers by leveraging the computing resource scheduling platform HTCondor, constructs workflows through the workflow orchestration platform Dolphineschedule, detects data via the central control website, automatically triggers workflows, monitors the workflow processing procedure, and provides a feedback and redo mechanism.
Results The workflow processing was automatically initiated 16,900 times for 4000 observations within 270 days, and approximately 40,000 files are archived daily. The system handles both standalone servers and computing clusters. Under the circumstances of sufficient resources where tasks do not need to wait, when utilizing the resources of standalone server, the average time for task switching is 1.4 s. In contrast, when using the computing cluster, the average time for task switching is 2.7 s. Regarding the Observation Data Workflow, 26.91% of the tasks are completed within 10 min, and 71.7% are completed within 100 min.
Conclusion The EP-FXT Workflow Management System is capable of effectively expediting the generation of data products.