jive-toolchain-verify/compare-ms-idi.md

compare-ms-idi.py

Use this script to verify tConvert's operation when it comes to exporting all MeasurementSet data to a (set of) FITS-IDI files or j2ms2 operation by comparing different MeasurementSets. This script collects the integrated weights for all (baseline, source) combinations (“key” hereafter), and counts how many times each integration is present.

If more than one data set is specified it will report:

• extra keys that are not common to all data sets, and in which data set these occur
• for each key common across all given data sets, the tool will compare the integrated weights and report differences. If those numbers are not equal some data is missing in some of the data set(s) - it is reported which value was found in which dataset.
• if a key has duplicated time stamps this means that the same data was present in the data set more than once and these occurrences are reported too

Usage:

    > compare-ms-idi.py --ms /path/to/input.ms
                        --idi /path/to/output.idi*

Note: the wildcards should not be escaped in this case.

Multiple --idi and/or --ms options are supported. The script effectively performs a multiway diff across all datasets given on the command line. It is also possible to compare only MeasurementSets, or only FITS-IDI files - the script doesn’t care, even when they’re (partially) disjoint.

If only one data set is passed on the command line no diff will be computed but the found keys are displayed - effectively summarizing the data set for exposure time per baseline per source.

The script has -h/--help.

Example output

The uninteresting case

This run compares two MeasurementSets that end up comparing equal, despite being created by two different versions of j2ms2 - the changes in j2ms2 have not affected the data in the produced MeasurementSet:

    > compare-ms-idi.py --ms rsm02-prod-j2ms2.ms
                        --ms rsm02-antdiam-j2ms2.ms
    Successful readonly open of default-locked table rsm02-prod-j2ms2.ms: 22 columns, 94752 rows
    Successful readonly open of default-locked table rsm02-antdiam-j2ms2.ms: 22 columns, 94752 rows
    Checked 2 data sets, 90 common keys                                  
    >

This is a more interesting case - an instrumented one.

For one experiment a subset of the data was converted to a second (much) smaller MeasurementSet, which was subsequently converted to FITS-IDI as well.

The comparison compared the full MeasurementSet, the partial MeasurementSet and the FITS-IDI file produced from the partial MeasurementSet.

    > compare-ms-idi.py --ms rsm02-dev.ms
                        --ms rsm02-prod-j2ms2.ms
                        --idi RSM02-PRODJ2MS2.IDI

It produces a lot of output, which can be summarized in three sections:

The extra keys

The full data set had some stations coming in later and/or sources that were only observed later than those present in the partial MeasurementSet. As such there are keys that are only present in the full MeasurementSet.

These types of keys - those that cannot be compared - are listed first:

==== Problem report ====
   MS: rsm02-dev.ms 
 Extra keys:
    ('T6T6', 'J1310+3220') found     89 times
    ('JbTr', 'J1310+3220') found     89 times
    ('McYs', 'J1310+3220') found     89 times
    ...

The common keys that have different values

For keys whose integrated weight differs between any of the data sets, the following is displayed:

('EfTr', 'J1427+2632') :
      6656.00s wgt= 53246.27   3328 times in   MS: rsm02-dev.ms
       416.00s wgt=  3328.00    208 times in   MS: rsm02-prod-j2ms2.ms
       416.00s wgt=  3328.00    208 times in  IDI: RSM02-PRODJ2MS2.IDI*
('EfYs', 'J1419+2706') :
      2550.00s wgt= 40775.69   1275 times in   MS: rsm02-dev.ms
       152.00s wgt=  2432.00     76 times in   MS: rsm02-prod-j2ms2.ms
       152.00s wgt=  2432.00     76 times in  IDI: RSM02-PRODJ2MS2.IDI*
('EfYs', 'J1427+2632') :
      6448.00s wgt=103158.98   3224 times in   MS: rsm02-dev.ms
       208.00s wgt=  3328.00    104 times in   MS: rsm02-prod-j2ms2.ms
       208.00s wgt=  3328.00    104 times in  IDI: RSM02-PRODJ2MS2.IDI*

It also shows that the data from the partial MeasurementSet and the corresponding FITS-IDI file is consistent but that a significant amount of data is missing compared to the full MeasurementSet.

The summary line

The tool always displays a one-line summary:

Checked 3 data sets, 90 common keys with 90 problems identified and 108
non-common keys in 1 formats