This script is similar to
compare-ms-idi.py in operation, only it
compares the meta data in all given data sets:
Use this script to verify
tConvert's operation when it comes to exporting
all MeasurementSet data to a (set of) FITS-IDI files and/or check different
j2ms2 versions producing the same (or explictly verify expected different!) meta data in the MeasurementSets.
The keys used to compare the meta data are antenna name, source name and lowest frequency of spectral window and are printed as
('antenna', 'station name'),
('source', 'source name') and
If only one data set is given, or there are no keys with more than one value, all keys and their associated properties are just displayed. This mode can be used for quick meta data inspection.
If more than one data set is specified the tool will report:
extra keys that are not common to all data sets, and in which data set they occur and how many times
for each keys common across all given data sets compares the properties
found in each data set.
Note that since each FITS-IDI chunk (see above)
has its own antenna, source and frequency table; the tool treats each
*.IDIn chunk as individual data set in order to verify that
all FITS-IDI chunks contain the same meta data.
The report compresses the multi-way diff by taking one reference value (indicating which data set it came from), computes the diff wrt to all other values and aggregates data sets by equivalent diff.
Only the properties that are different are displayed and in which data set(s) these specific values were found
The usage is exactly the same as
compare-ms-idi.py so only the different
output is shown.
A use case for this appeared recently. A request to propagate the antenna diameter from the VEX file, into the MeasurementSet and consequently into the FITS-IDI file(s) was requested.
A VEX file was hand-edited to supply an antenna diameter of 100.0 m for the Ef antenna.
In total six files were created:
j2ms2which does not handle antenna diameter at all
j2ms2that should propagate from VEX to MeasurementSet
tConvert, which does not handle antenna diameter at all
Running the six-way meta-data diff, with, for good measures, a (partially) disjoint different dataset thrown in to highlight the tool’s operation, a seven-way meta data diff in one go:
> compare-ms-idi-meta.py --ms rsm02-prod-j2ms2.ms --ms rsm02-antdiam-j2ms2.ms --idi RSM02-PRODJ2MS2-PRODTCONVERT.IDI --idi RSM02-PRODJ2MS2-ANTDIAMTCONVERT.IDI --idi RSM02-ANTDIAMJ2MS2-PRODTCONVERT.IDI --idi RSM02-ANTDIAMJ2MS2-ANTDIAMTCONVERT.IDI --idi ../../eg063d/eg063-prod-dev.IDI*
yields a fair bit of output.
compare-ms-idi.py, any keys not common across all data sets are listed first:
Their values are not reported.
==== Problem report ==== IDI: ../../eg063d: eg063-prod-dev.IDI* Extra keys: ('antenna', 'Ar') found 3 times ... snip ... ('frequency', '4926990000.000') found 3 times ... snip ... ('source', '0133+476') found 3 times ... snip ...
The most interesting output is for the Ef station - for that is the one
station where the antenna diameter was added in the VEX file. It is expected
that propagation of this value into the actual FITS-IDI file(s) depends on
specifically which combination of
tConvert versions is used.
This is what the tool reports for the key
('antenna', 'Ef') : ANT Ef: xyz=[4033947.2616 486990.7866 4900430.9915] (0) d=0.0 offset=[0.0145 0. 0. ] mount=alt-az found in: MS: rsm02-prod-j2ms2.ms DIFF: diameter: 100.0 vs 0.0 found in: (1) MS: rsm02-antdiam-j2ms2.ms DIFF: diameter: None vs 0.0, (2) mount: UnknownMNTSTA#0 vs alt-az found in: IDI: RSM02-PRODJ2MS2-PRODTCONVERT.IDI IDI: RSM02-ANTDIAMJ2MS2-PRODTCONVERT.IDI DIFF: mount: UnknownMNTSTA#0 vs alt-az found in: (3) IDI: RSM02-PRODJ2MS2-ANTDIAMTCONVERT.IDI DIFF: diameter: 100.0 vs 0.0, (4) mount: UnknownMNTSTA#0 vs alt-az found in: IDI: RSM02-ANTDIAMJ2MS2-ANTDIAMTCONVERT.IDI DIFF: diameter: None vs 0.0, (5) offset: [0.013 0. 0.] vs [0.0145 0. 0.], mount: UnknownMNTSTA#0 vs alt-az found in: IDI: ../../eg063d/eg063-prod-dev.IDI[1-3]
Comparing the reference value (0) - the first value found, in
rsm03-prod-j2ms2.ms- to values found in the other data sets, five (5)
unique diffs are found.
(0) The reference value. It contains default antenna diameter of 0.0,
written in the MeasurementSet by an antenna diameter unaware VEX file
(1) The antenna diameter aware
j2ms2 has propagated the 100.0 m diameter
to the MeasurementSet - that is the only diff between the two
MeasurementSets apparently: completely to expectation
(2) The diameter unaware
tConvert does not write this column in the
FITS-IDI file, hence the diameter is
(3) The diameter aware
tConvert does propagate the value to the
FITS-IDI file but the source MeasurementSet did not contain the correct
100.0 m value on account of being created by the diameter unaware
so the only difference is the station mount
(4) This is the fully updated antenna diameter toolchain: antenna aware
tConvert. The difference with the reference data set
includes the 100.0 m diameter being found in this FITS-IDI file
(5) The rather disjoint FITS-IDI files from a different experiment thrown
in show that the station offset used in experiment EG063 was different
than the one used in the reference data set and that the FITS-IDI files
were created by a
tConvert that does not write the antenna diameter
Finally it shows that the station mount from the MeasurementSet (“alt-az”) does not get translated to a known FITS-IDI enumerated value (see the FITS-IDI standard1 in any of the FITS-IDI files. This may need looking into.