A set of tools to verify the operation of the j2ms2 and tConvert programs
選択できるのは25トピックまでです。 トピックは、先頭が英数字で、英数字とダッシュ('-')を使用した35文字以内のものにしてください。
haavee e09d4c3be0 More subtle SafeCompare() 8ヶ月前
README.md Documentation 8ヶ月前
check-multipart-fits.md Documentation 8ヶ月前
check-multipart-fits.py Import the multi-part IDI file checker 2年前
compare-ms-idi-meta.md diameter can be None 8ヶ月前
compare-ms-idi-meta.py More subtle SafeCompare() 8ヶ月前
compare-ms-idi.md Documentation 8ヶ月前
compare-ms-idi.py Remove unnecessary global function(s) 2年前
fix-uvf.py Be consistent with other tools 1年前

README.md

JIVE-toolchain-verify

At JIVE several data conversions take place. The most-used ones are from correlator output to CASA MeasurementSet v21 (j2ms2), and from MeasurementSet v2 to FITS-IDI (tConvert), the FITS Interferometry Data Interchange, format2.

No tools existed yet to verify whether those data format transformation utilities (j2ms2, tConvert) actually did their work correctly - i.e. losing, adding, or corrupting (meta-)data.

This repository contains a number of tools that can be used to compare different data formats and test important quantities for equality to asseess loss, addition or changes in those.

Getting Started

After

    > cd /path/to/somewhere
    > git clone <this repo>

several Python2/3 compatible scripts become available:

check-multipart-fits.py
compare-ms-idi.py
compare-ms-idi-meta.py
fix-uvf.py

whose purpose will be explained below.

Running the scripts using a specific (local?) Python can be easily done like so

    > /path/to/python-X.Y.Z /path/to/compare-ms-idi.py [options]

fix-uvf.py

Not a checking tool, actually. It is used to fix UV-FITS files.

	> fix-uvf.py /path/to/file.UVF

Will do the following edits in-place in file.UVF

  • Change the name of the FREQSEL column into FRQSEL since that is what AIPS is looking for
  • Add the SORTORDR = 'TB' header keyword. JIVE has always written data in time, baseline order but failed to indicate that in the file header, causing an unnecessary resort after delivery to the PI.

UV-FITS is a deprecated flavour of FITS for radio-interferometric data transport. The format has no proper documentation and the data cannot be broken in smaller chunks - all data is in one (1) file.

check-multipart-fits.py

Checks if there is data loss between the boundaries of subsequent IDI files produced from a single MeasurementSet. Reports how much observing time (in seconds) was lost across all boundaries.

Full documentation and explanation is in check-multipart-fits.md

compare-ms-idi.py

Use this script to verify tConvert's operation when it comes to exporting all MeasurementSet data to a (set of) FITS-IDI files or j2ms2 operation by comparing different MeasurementSets. This script collects the integrated weights for all (baseline, source) combinations (“key” hereafter), and counts how many times each integration is present.

Full documentation and explanation is in compare-ms-idi.md

compare-ms-idi-meta.py

This script is similar to compare-ms-idi.py in operation, only it compares the meta data in all given data sets:

  • antenna properties (position, offset, mount, diameter)
  • source properties (position)
  • spectral windows (lowest, highest frequency, number of spectral channels)

Use this script to verify tConvert's operation when it comes to exporting all MeasurementSet data to a (set of) FITS-IDI files and/or check different j2ms2 versions producing the same (or explictly verify expected different!) meta data in the MeasurementSets.

Full documentation and explanation is in compare-ms-idi-meta.md

Prerequisites

All tools require extra Python modules to access data in CASA MeasurementSet and FITS format:

Locally, we have good experiences installing anaconda(use the free option) and adding the packages to that - which is a very quick way of setting up your system with a batteries-included scientific software suite.

References

1: https://casa.nrao.edu/Memos/229.html
2: https://fits.gsfc.nasa.gov/registry/fitsidi.html