This script is similar to
compare-ms-idi.py in operation, only it
compares the meta data in all given data sets:
- antenna properties (position, offset, mount, diameter)
- source properties (position)
- spectral windows (lowest, highest frequency, number of spectral channels)
Use this script to verify
tConvert's operation when it comes to exporting
all MeasurementSet data to a (set of) FITS-IDI files and/or check different
j2ms2 versions producing the same (or explictly verify expected different!) meta data in the MeasurementSets.
The keys on which data is compared
The keys used to compare the meta data are antenna name, source name and lowest frequency of spectral window and are printed as
('antenna', 'station name'),
('source', 'source name') and
Operation with one or more input files
If only one data set is given, or there are no keys with more than one value, all keys and their associated properties are just displayed. This mode can be used for quick meta data inspection.
If more than one data set is specified the tool will report:
extra keys that are not common to all data sets, and in which data set they occur and how many times
for each keys common across all given data sets compares the properties found in each data set. Note that since each FITS-IDI chunk (see above) has its own antenna, source and frequency table; the tool treats each individual
*.IDInchunk as individual data set in order to verify that all FITS-IDI chunks contain the same meta data.
The report compresses the multi-way diff by taking one reference value (indicating which data set it came from), computes the diff wrt to all other values and aggregates data sets by equivalent diff.
Only the properties that are different are displayed and in which data set(s) these specific values were found
The usage is exactly the same as
compare-ms-idi.py so only the different
output is shown.
An extreme example
A use case for this appeared recently. A request to propagate the antenna diameter from the VEX file, into the MeasurementSet and consequently into the FITS-IDI file(s) was requested.
A VEX file was hand-edited to supply an antenna diameter of 100.0 m for the Ef antenna.
In total six files were created:
- a MeasurementSet by current production
j2ms2which does not handle antenna diameter at all
- id. by a fixed
j2ms2that should propagate from VEX to MeasurementSet
- Each MeasurementSet was converted to FITS-IDI by two versions of
- the current production
tConvert, which does not handle antenna diameter at all
- the new one that should propagate the antenna diameter from MeasurementSet to FITS-IDI
- the current production
Running the six-way meta-data diff, with, for good measures, a (partially) disjoint different dataset thrown in to highlight the tool's operation, a seven-way meta data diff in one go:
> compare-ms-idi-meta.py --ms rsm02-prod-j2ms2.ms --ms rsm02-antdiam-j2ms2.ms --idi RSM02-PRODJ2MS2-PRODTCONVERT.IDI --idi RSM02-PRODJ2MS2-ANTDIAMTCONVERT.IDI --idi RSM02-ANTDIAMJ2MS2-PRODTCONVERT.IDI --idi RSM02-ANTDIAMJ2MS2-ANTDIAMTCONVERT.IDI --idi ../../eg063d/eg063-prod-dev.IDI*
yields a fair bit of output.
The extra keys
compare-ms-idi.py, any keys not common across all data sets are listed first:
Their values are not reported.
==== Problem report ==== IDI: ../../eg063d: eg063-prod-dev.IDI* Extra keys: ('antenna', 'Ar') found 3 times ... snip ... ('frequency', '4926990000.000') found 3 times ... snip ... ('source', '0133+476') found 3 times ... snip ...
The diff report
The most interesting output is for the Ef station - for that is the one
station where the antenna diameter was added in the VEX file. It is expected
that propagation of this value into the actual FITS-IDI file(s) depends on
specifically which combination of
tConvert versions is used.
This is what the tool reports for the key
('antenna', 'Ef') : ANT Ef: xyz=[4033947.2616 486990.7866 4900430.9915] (0) d=0.0 offset=[0.0145 0. 0. ] mount=alt-az found in: MS: rsm02-prod-j2ms2.ms DIFF: diameter: 100.0 vs 0.0 found in: (1) MS: rsm02-antdiam-j2ms2.ms DIFF: diameter: None vs 0.0, (2) mount: UnknownMNTSTA#0 vs alt-az found in: IDI: RSM02-PRODJ2MS2-PRODTCONVERT.IDI IDI: RSM02-ANTDIAMJ2MS2-PRODTCONVERT.IDI DIFF: mount: UnknownMNTSTA#0 vs alt-az found in: (3) IDI: RSM02-PRODJ2MS2-ANTDIAMTCONVERT.IDI DIFF: diameter: 100.0 vs 0.0, (4) mount: UnknownMNTSTA#0 vs alt-az found in: IDI: RSM02-ANTDIAMJ2MS2-ANTDIAMTCONVERT.IDI DIFF: diameter: None vs 0.0, (5) offset: [0.013 0. 0.] vs [0.0145 0. 0.], mount: UnknownMNTSTA#0 vs alt-az found in: IDI: ../../eg063d/eg063-prod-dev.IDI[1-3]
Comparing the reference value (0) - the first value found, in
rsm03-prod-j2ms2.ms- to values found in the other data sets, five (5)
unique diffs are found.
(0) The reference value. It contains default antenna diameter of 0.0, written in the MeasurementSet by an antenna diameter unaware VEX file accessor in
(1) The antenna diameter aware
j2ms2has propagated the 100.0 m diameter to the MeasurementSet - that is the only diff between the two MeasurementSets apparently: completely to expectation
(2) The diameter unaware
tConvertdoes not write this column in the FITS-IDI file, hence the diameter is
(3) The diameter aware
tConvertdoes propagate the value to the FITS-IDI file but the source MeasurementSet did not contain the correct 100.0 m value on account of being created by the diameter unaware
j2ms2so the only difference is the station mount
(4) This is the fully updated antenna diameter toolchain: antenna aware
tConvert. The difference with the reference data set includes the 100.0 m diameter being found in this FITS-IDI file
(5) The rather disjoint FITS-IDI files from a different experiment thrown in show that the station offset used in experiment EG063 was different than the one used in the reference data set and that the FITS-IDI files were created by a
tConvertthat does not write the antenna diameter
Finally it shows that the station mount from the MeasurementSet ("alt-az") does not get translated to a known FITS-IDI enumerated value (see the FITS-IDI standard1 in any of the FITS-IDI files. This may need looking into.