A set of tools to verify the operation of the j2ms2 and tConvert programs
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
haavee e09d4c3be0 More subtle SafeCompare() 2 years ago
README.md Documentation 2 years ago
check-multipart-fits.md Documentation 2 years ago
check-multipart-fits.py Import the multi-part IDI file checker 4 years ago
compare-ms-idi-meta.md diameter can be None 2 years ago
compare-ms-idi-meta.py More subtle SafeCompare() 2 years ago
compare-ms-idi.md Documentation 2 years ago
compare-ms-idi.py Remove unnecessary global function(s) 4 years ago
fix-uvf.py Be consistent with other tools 3 years ago



At JIVE several data conversions take place. The most-used ones are from correlator output to CASA MeasurementSet v21 (j2ms2), and from MeasurementSet v2 to FITS-IDI (tConvert), the FITS Interferometry Data Interchange, format2.

No tools existed yet to verify whether those data format transformation utilities (j2ms2, tConvert) actually did their work correctly - i.e. losing, adding, or corrupting (meta-)data.

This repository contains a number of tools that can be used to compare different data formats and test important quantities for equality to asseess loss, addition or changes in those.

Getting Started


    > cd /path/to/somewhere
    > git clone <this repo>

several Python2/3 compatible scripts become available:


whose purpose will be explained below.

Running the scripts using a specific (local?) Python can be easily done like so

    > /path/to/python-X.Y.Z /path/to/compare-ms-idi.py [options]


Not a checking tool, actually. It is used to fix UV-FITS files.

	> fix-uvf.py /path/to/file.UVF

Will do the following edits in-place in file.UVF

  • Change the name of the FREQSEL column into FRQSEL since that is what AIPS is looking for
  • Add the SORTORDR = 'TB' header keyword. JIVE has always written data in time, baseline order but failed to indicate that in the file header, causing an unnecessary resort after delivery to the PI.

UV-FITS is a deprecated flavour of FITS for radio-interferometric data transport. The format has no proper documentation and the data cannot be broken in smaller chunks - all data is in one (1) file.


Checks if there is data loss between the boundaries of subsequent IDI files produced from a single MeasurementSet. Reports how much observing time (in seconds) was lost across all boundaries.

Full documentation and explanation is in check-multipart-fits.md


Use this script to verify tConvert's operation when it comes to exporting all MeasurementSet data to a (set of) FITS-IDI files or j2ms2 operation by comparing different MeasurementSets. This script collects the integrated weights for all (baseline, source) combinations ("key" hereafter), and counts how many times each integration is present.

Full documentation and explanation is in compare-ms-idi.md


This script is similar to compare-ms-idi.py in operation, only it compares the meta data in all given data sets:

  • antenna properties (position, offset, mount, diameter)
  • source properties (position)
  • spectral windows (lowest, highest frequency, number of spectral channels)

Use this script to verify tConvert's operation when it comes to exporting all MeasurementSet data to a (set of) FITS-IDI files and/or check different j2ms2 versions producing the same (or explictly verify expected different!) meta data in the MeasurementSets.

Full documentation and explanation is in compare-ms-idi-meta.md


All tools require extra Python modules to access data in CASA MeasurementSet and FITS format:

  • Python casacore module:

    • pyrap (casacore < 2.0), or
    • python-casacore (casacore >= 2.0)

    The python-casacore package can easily be installed or built from source, see the project's github page

  • astropy, for astropy.io.fits only See the astropy documentation.

Locally, we have good experiences installing anaconda(use the free option) and adding the packages to that - which is a very quick way of setting up your system with a batteries-included scientific software suite.


1: https://casa.nrao.edu/Memos/229.html
2: https://fits.gsfc.nasa.gov/registry/fitsidi.html