A set of tools to verify the operation of the j2ms2 and tConvert programs
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

6.9 KiB


This script is similar to compare-ms-idi.py in operation, only it compares the meta data in all given data sets:

  • antenna properties (position, offset, mount, diameter)
  • source properties (position)
  • spectral windows (lowest, highest frequency, number of spectral channels)

Use this script to verify tConvert's operation when it comes to exporting all MeasurementSet data to a (set of) FITS-IDI files and/or check different j2ms2 versions producing the same (or explictly verify expected different!) meta data in the MeasurementSets.

The keys on which data is compared

The keys used to compare the meta data are antenna name, source name and lowest frequency of spectral window and are printed as ('antenna', 'station name'), ('source', 'source name') and ('frequency', 'frequency-in-Hz')

Operation with one or more input files

If only one data set is given, or there are no keys with more than one value, all keys and their associated properties are just displayed. This mode can be used for quick meta data inspection.

If more than one data set is specified the tool will report:

  • extra keys that are not common to all data sets, and in which data set they occur and how many times

  • for each keys common across all given data sets compares the properties found in each data set. Note that since each FITS-IDI chunk (see above) has its own antenna, source and frequency table; the tool treats each individual *.IDIn chunk as individual data set in order to verify that all FITS-IDI chunks contain the same meta data.

    The report compresses the multi-way diff by taking one reference value (indicating which data set it came from), computes the diff wrt to all other values and aggregates data sets by equivalent diff.

    Only the properties that are different are displayed and in which data set(s) these specific values were found

The usage is exactly the same as compare-ms-idi.py so only the different output is shown.

An extreme example

A use case for this appeared recently. A request to propagate the antenna diameter from the VEX file, into the MeasurementSet and consequently into the FITS-IDI file(s) was requested.

A VEX file was hand-edited to supply an antenna diameter of 100.0 m for the Ef antenna.

In total six files were created:

  • a MeasurementSet by current production j2ms2 which does not handle antenna diameter at all
  • id. by a fixed j2ms2 that should propagate from VEX to MeasurementSet
  • Each MeasurementSet was converted to FITS-IDI by two versions of tConvert:
    • the current production tConvert, which does not handle antenna diameter at all
    • the new one that should propagate the antenna diameter from MeasurementSet to FITS-IDI

Running the six-way meta-data diff, with, for good measures, a (partially) disjoint different dataset thrown in to highlight the tool’s operation, a seven-way meta data diff in one go:

    > compare-ms-idi-meta.py --ms rsm02-prod-j2ms2.ms
                             --ms rsm02-antdiam-j2ms2.ms
                             --idi RSM02-PRODJ2MS2-PRODTCONVERT.IDI
                             --idi RSM02-PRODJ2MS2-ANTDIAMTCONVERT.IDI
                             --idi RSM02-ANTDIAMJ2MS2-PRODTCONVERT.IDI
                             --idi RSM02-ANTDIAMJ2MS2-ANTDIAMTCONVERT.IDI 
                             --idi ../../eg063d/eg063-prod-dev.IDI* 

yields a fair bit of output.

The extra keys

As with compare-ms-idi.py, any keys not common across all data sets are listed first: Their values are not reported.

==== Problem report ====
  IDI: ../../eg063d: eg063-prod-dev.IDI* 
 Extra keys:
    ('antenna', 'Ar') found 3 times
    ... snip ...
    ('frequency', '4926990000.000') found 3 times
    ... snip ...
    ('source', '0133+476') found 3 times
    ... snip ...

The diff report

The most interesting output is for the Ef station - for that is the one station where the antenna diameter was added in the VEX file. It is expected that propagation of this value into the actual FITS-IDI file(s) depends on specifically which combination of j2ms2 and tConvert versions is used.

This is what the tool reports for the key (antenna, Ef):

('antenna', 'Ef') :
    ANT Ef: xyz=[4033947.2616  486990.7866 4900430.9915]  (0)
            d=0.0 offset=[0.0145 0.     0.    ]
            mount=alt-az found in:
          MS: rsm02-prod-j2ms2.ms
    DIFF: diameter: 100.0 vs 0.0 found in:                (1)
          MS: rsm02-antdiam-j2ms2.ms
    DIFF: diameter: None vs 0.0,                          (2)
          mount: UnknownMNTSTA#0 vs alt-az found in:
    DIFF: mount: UnknownMNTSTA#0 vs alt-az found in:      (3)
    DIFF: diameter: 100.0 vs 0.0,                         (4)
          mount: UnknownMNTSTA#0 vs alt-az found in:
    DIFF: diameter: None vs 0.0,                          (5)
          offset: [0.013 0. 0.] vs [0.0145 0. 0.],
          mount: UnknownMNTSTA#0 vs alt-az found in:
         IDI: ../../eg063d/eg063-prod-dev.IDI[1-3]

Comparing the reference value (0) - the first value found, in rsm03-prod-j2ms2.ms- to values found in the other data sets, five (5) unique diffs are found.

  • (0) The reference value. It contains default antenna diameter of 0.0, written in the MeasurementSet by an antenna diameter unaware VEX file accessor in j2ms2

  • (1) The antenna diameter aware j2ms2 has propagated the 100.0 m diameter to the MeasurementSet - that is the only diff between the two MeasurementSets apparently: completely to expectation

  • (2) The diameter unaware tConvert does not write this column in the FITS-IDI file, hence the diameter is None

  • (3) The diameter aware tConvert does propagate the value to the FITS-IDI file but the source MeasurementSet did not contain the correct 100.0 m value on account of being created by the diameter unaware j2ms2 so the only difference is the station mount

  • (4) This is the fully updated antenna diameter toolchain: antenna aware j2ms2 and tConvert. The difference with the reference data set includes the 100.0 m diameter being found in this FITS-IDI file

  • (5) The rather disjoint FITS-IDI files from a different experiment thrown in show that the station offset used in experiment EG063 was different than the one used in the reference data set and that the FITS-IDI files were created by a tConvert that does not write the antenna diameter

Finally it shows that the station mount from the MeasurementSet (“alt-az”) does not get translated to a known FITS-IDI enumerated value (see the FITS-IDI standard1 in any of the FITS-IDI files. This may need looking into.


1: https://fits.gsfc.nasa.gov/registry/fitsidi.html