JIVE-toolchain-verify

At JIVE several data conversions take place. The most-used ones are from correlator output to CASA MeasurementSet v2¹ (j2ms2), and from MeasurementSet v2 to FITS-IDI (tConvert), the FITS Interferometry Data Interchange, format².

No tools existed yet to verify whether those data format transformation utilities (j2ms2, tConvert) actually did their work correctly - i.e. losing, adding, or corrupting (meta-)data.

This repository contains a number of tools that can be used to compare different data formats and test important quantities for equality to asseess loss, addition or changes in those.

Getting Started

After

    > cd /path/to/somewhere
    > git clone <this repo>

several Python2/3 compatible scripts become available:

check-multipart-fits.py
compare-ms-idi.py
compare-ms-idi-meta.py
fix-uvf.py

whose purpose will be explained below.

Running the scripts using a specific (local?) Python can be easily done like so

    > /path/to/python-X.Y.Z /path/to/compare-ms-idi.py [options]

fix-uvf.py

Not a checking tool, actually. It is used to fix UV-FITS files.

	> fix-uvf.py /path/to/file.UVF

Will do the following edits in-place in file.UVF

• Change the name of the FREQSEL column into FRQSEL since that is what AIPS is looking for
• Add the SORTORDR = 'TB' header keyword. JIVE has always written data in time, baseline order but failed to indicate that in the file header, causing an unnecessary resort after delivery to the PI.

UV-FITS is a deprecated flavour of FITS for radio-interferometric data transport. The format has no proper documentation and the data cannot be broken in smaller chunks - all data is in one (1) file.

check-multipart-fits.py

Checks if there is data loss between the boundaries of subsequent IDI files produced from a single MeasurementSet. Reports how much observing time (in seconds) was lost across all boundaries.

Full documentation and explanation is in check-multipart-fits.md

compare-ms-idi.py

Use this script to verify tConvert's operation when it comes to exporting all MeasurementSet data to a (set of) FITS-IDI files or j2ms2 operation by comparing different MeasurementSets. This script collects the integrated weights for all (baseline, source) combinations (“key” hereafter), and counts how many times each integration is present.

Full documentation and explanation is in compare-ms-idi.md

compare-ms-idi-meta.py

This script is similar to compare-ms-idi.py in operation, only it compares the meta data in all given data sets:

• antenna properties (position, offset, mount, diameter)
• source properties (position)
• spectral windows (lowest, highest frequency, number of spectral channels)

Use this script to verify tConvert's operation when it comes to exporting all MeasurementSet data to a (set of) FITS-IDI files and/or check different j2ms2 versions producing the same (or explictly verify expected different!) meta data in the MeasurementSets.

Full documentation and explanation is in compare-ms-idi-meta.md

Prerequisites

All tools require extra Python modules to access data in CASA MeasurementSet and FITS format:

• Python casacore module:

  • pyrap (casacore < 2.0), or
  • python-casacore (casacore >= 2.0)

  The python-casacore package can easily be installed or built from source, see the project’s github page

• astropy, for astropy.io.fits only See the astropy documentation.

Locally, we have good experiences installing anaconda(use the free option) and adding the packages to that - which is a very quick way of setting up your system with a batteries-included scientific software suite.

References

1: https://casa.nrao.edu/Memos/229.html
2: https://fits.gsfc.nasa.gov/registry/fitsidi.html