A real time processing system for big data in astronomy: Applications to Hera
Loading...
Date
2021
Journal Title
Journal ISSN
Volume Title
Publisher
Elsevier
Abstract
As current- and next-generation astronomical instruments come online, they will generate an unprecedented deluge of data. Analyzing these data in real time presents unique conceptual and computational
challenges, and their long-term storage and archiving is scientifically essential for generating reliable,
reproducible results. We present here the real-time processing (RTP) system for the Hydrogen Epoch of Reionization Array (HERA), a radio interferometer endeavoring to provide the first detection of the
highly redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an interferometer.
The RTP system consists of analysis routines run on raw data shortly after they are acquired, such
as calibration and detection of radio-frequency interference (RFI) events. RTP works closely with the
Librarian, the HERA data storage and transfer manager which automatically ingests data and transfers
copies to other clusters for post-processing analysis. Both the RTP system and the Librarian are
public and open source software, which allows for them to be modified for use in other scientific
collaborations. When fully constructed, HERA is projected to generate over 50 terabytes (TB) of data
each night, and the RTP system enables the successful scientific analysis of these data.
Description
Keywords
Physical sciences, Engineering, Astronomy, Big data, Software
Citation
La Plante, P. et al. (2021). A real time processing system for big data in astronomy: Applications to Hera. Astronomy and Computing,36, 100489. https://doi.org/10.1016/j.ascom.2021.100489