A real time processing system for big data in astronomy: Applications to Hera

Simple item page
dc.contributor.authorLa Plante, Paul
dc.contributor.authorWilliams, Peter KG
dc.contributor.authorKittiwisit, Piyanat
dc.date.accessioned2021-09-23T10:38:33Z
dc.date.available2021-09-23T10:38:33Z
dc.date.issued2021
dc.description.abstractAs current- and next-generation astronomical instruments come online, they will generate an unprecedented deluge of data. Analyzing these data in real time presents unique conceptual and computational challenges, and their long-term storage and archiving is scientifically essential for generating reliable, reproducible results. We present here the real-time processing (RTP) system for the Hydrogen Epoch of Reionization Array (HERA), a radio interferometer endeavoring to provide the first detection of the highly redshifted 21 cm signal from Cosmic Dawn and the Epoch of Reionization by an interferometer. The RTP system consists of analysis routines run on raw data shortly after they are acquired, such as calibration and detection of radio-frequency interference (RFI) events. RTP works closely with the Librarian, the HERA data storage and transfer manager which automatically ingests data and transfers copies to other clusters for post-processing analysis. Both the RTP system and the Librarian are public and open source software, which allows for them to be modified for use in other scientific collaborations. When fully constructed, HERA is projected to generate over 50 terabytes (TB) of data each night, and the RTP system enables the successful scientific analysis of these data.en_US
dc.identifier.citationLa Plante, P. et al. (2021). A real time processing system for big data in astronomy: Applications to Hera. Astronomy and Computing,36, 100489. https://doi.org/10.1016/j.ascom.2021.100489en_US
dc.identifier.issn2213-1337
dc.identifier.urihttps://doi.org/10.1016/j.ascom.2100489
dc.identifier.urihttp://hdl.handle.net/10566/6790
dc.language.isoenen_US
dc.publisherElsevieren_US
dc.subjectPhysical sciencesen_US
dc.subjectEngineeringen_US
dc.subjectAstronomyen_US
dc.subjectBig dataen_US
dc.subjectSoftwareen_US
dc.titleA real time processing system for big data in astronomy: Applications to Heraen_US
dc.typeArticleen_US

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
A-Real-Time-Processing-system-for-big-data-in-astronomy-Applications-to-HERAAstronomy-and-Computing.pdf
Size:
1010.79 KB
Format:
Adobe Portable Document Format
Description:
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.71 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

Research Articles (Physics)