A set of tools to verify the operation of the j2ms2 and tConvert programs
Não pode escolher mais do que 25 tópicos Os tópicos devem começar com uma letra ou um número, podem incluir traços ('-') e podem ter até 35 caracteres.
haavee e09d4c3be0 More subtle SafeCompare() há 8 meses
README.md Documentation há 8 meses
check-multipart-fits.md Documentation há 8 meses
check-multipart-fits.py Import the multi-part IDI file checker há 2 anos
compare-ms-idi-meta.md diameter can be None há 8 meses
compare-ms-idi-meta.py More subtle SafeCompare() há 8 meses
compare-ms-idi.md Documentation há 8 meses
compare-ms-idi.py Remove unnecessary global function(s) há 2 anos
fix-uvf.py Be consistent with other tools há 1 ano

README.md

JIVE-toolchain-verify

At JIVE several data conversions take place. The most-used ones are from correlator output to CASA MeasurementSet v21 (j2ms2), and from MeasurementSet v2 to FITS-IDI (tConvert), the FITS Interferometry Data Interchange, format2.

No tools existed yet to verify whether those data format transformation utilities (j2ms2, tConvert) actually did their work correctly - i.e. losing, adding, or corrupting (meta-)data.

This repository contains a number of tools that can be used to compare different data formats and test important quantities for equality to asseess loss, addition or changes in those.

Getting Started

After

    > cd /path/to/somewhere
    > git clone <this repo>

several Python2/3 compatible scripts become available:

check-multipart-fits.py
compare-ms-idi.py
compare-ms-idi-meta.py
fix-uvf.py

whose purpose will be explained below.

Running the scripts using a specific (local?) Python can be easily done like so

    > /path/to/python-X.Y.Z /path/to/compare-ms-idi.py [options]

fix-uvf.py

Not a checking tool, actually. It is used to fix UV-FITS files.

	> fix-uvf.py /path/to/file.UVF

Will do the following edits in-place in file.UVF

  • Change the name of the FREQSEL column into FRQSEL since that is what AIPS is looking for
  • Add the SORTORDR = 'TB' header keyword. JIVE has always written data in time, baseline order but failed to indicate that in the file header, causing an unnecessary resort after delivery to the PI.

UV-FITS is a deprecated flavour of FITS for radio-interferometric data transport. The format has no proper documentation and the data cannot be broken in smaller chunks - all data is in one (1) file.

check-multipart-fits.py

Checks if there is data loss between the boundaries of subsequent IDI files produced from a single MeasurementSet. Reports how much observing time (in seconds) was lost across all boundaries.

Full documentation and explanation is in check-multipart-fits.md

compare-ms-idi.py

Use this script to verify tConvert's operation when it comes to exporting all MeasurementSet data to a (set of) FITS-IDI files or j2ms2 operation by comparing different MeasurementSets. This script collects the integrated weights for all (baseline, source) combinations (“key” hereafter), and counts how many times each integration is present.

Full documentation and explanation is in compare-ms-idi.md

compare-ms-idi-meta.py

This script is similar to compare-ms-idi.py in operation, only it compares the meta data in all given data sets:

  • antenna properties (position, offset, mount, diameter)
  • source properties (position)
  • spectral windows (lowest, highest frequency, number of spectral channels)

Use this script to verify tConvert's operation when it comes to exporting all MeasurementSet data to a (set of) FITS-IDI files and/or check different j2ms2 versions producing the same (or explictly verify expected different!) meta data in the MeasurementSets.

Full documentation and explanation is in compare-ms-idi-meta.md

Prerequisites

All tools require extra Python modules to access data in CASA MeasurementSet and FITS format:

Locally, we have good experiences installing anaconda(use the free option) and adding the packages to that - which is a very quick way of setting up your system with a batteries-included scientific software suite.

References

1: https://casa.nrao.edu/Memos/229.html
2: https://fits.gsfc.nasa.gov/registry/fitsidi.html