Browse Source

Test description in README.md format

master
haavee 10 months ago
parent
commit
fb47dfea63
  1. 51
      ES085A/README.md
  2. 64
      ES085A/test_description.txt

51
ES085A/README.md

@ -0,0 +1,51 @@
# Test numerical reproduceability of j2ms2 and tConvert
This test contains three raw datasets:
- raw SFXC output files from job 24427 (`24427/*.cor`), from experiment ES085A.
This data was chosen since it has a strong maser line, easily observable off center in subband 3.
- known-good MeasurementSet produced from that: `es085a.ms`
- known-good FITS-IDI file produced from `es085a.ms`: `ES085A.IDI`
And two types of support files:
- VEX file(s) for the experiment (`ES085A.vix` and `es085a.vix`), who are identical/one is symlink of the other; the `j2ms2` tool expects a VEX file name of the name of the directory it is run in under certain circumstances.
- The `lis` file, used to control the input to j2ms2 to exactly (re)produce a given MeasurementSet
This ~1 GB of data is stored on the [JIVE archive](https://archive.jive.nl/regression) and will be automatically downloaded the first time this regression test is run.
Then, starting from raw correlator output, VEX file and "lis" file four data sets are created in a temporary location:
- `j2ms2` is run with lisfile; lisfile incorporates all `*.cor` files from the job, producing MeasurementSet `es085a_cont.ms`
- `j2ms2` is run "manually" by specifying `-o es085a.ms job/*.cor` on the command line; producing MeasurementSet `es085a.ms`
- both MeasurementSets are converted to FITS-IDI using `tConvert`:
- `es085a_cont.ms` => `ES085A_CONT.IDI`
- `es085a.ms` => `ES085A.IDI`
## The following tests are performed
- `j2ms2` and `tConvert` should run succesfully - i.e. terminate with exit code 0
- Verify that "all data" from the correlator files is written to the MeasurementSets and FITS-IDI files.
The [jive-toolchain-verify](https://code.jive.eu/verkout/jive-toolchain-verify) tool [compare-ms-idi.py](https://code.jive.eu/verkout/jive-toolchain-verify/src/branch/master/compare-ms-idi.md) is used to compute a six-way diff between the two "gold" datasets and the four newly created ones.
The tool collects integrated weight and integration time per baseline per source, but does not look at the data content itself. Any differences reported are due to the toolchain-under-test skipping (or duplicating) data in either of the data formats; the diff output will tell what happened in which format(s).
- Verify that all meta data (antenna, source, frequency setup properties) are correctly propagated. The [jive-toolchain-verify](https://code.jive.eu/verkout/jive-toolchain-verify) tool [compare-ms-idi-meta.py](https://code.jive.eu/verkout/jive-toolchain-verify/src/branch/master/compare-ms-idi-meta.md) tool is used to compute a six-way diff between the meta data from the two "gold" datasets and the four data files generated by the toolchain under test. Any differences reported should be due to the new toolchain writing different meta data to (one of) the other formats.
- Verification of numerical equality of the data written to the MeasurementSet(s):
A batch script in combination of a postprocessing module for the [jiveplot](https://github.com/haavee/jiveplot) tool is run.
The jiveplot tool is used to:
- select three time ranges (`scan 18 19 33` of ES085A)
- compute amplitude+phase (y-axes) as function of frequency (x-axis)
- those are integrated (summed) for each baseline/subband/polarization/source/scan
These values are extracted from two MeasurementSets: gold and toolchain-under-test produced MeasurementSet.
A [special-purpose written postprocessing module for jiveplot](https://code.jive.eu/verkout/jive-toolchain-regression/src/branch/master/python/compare_ms_data.py) compares the datasets in two successive plot actions.
It keeps the last plot data set and if a new dataset is plotted, it computes the difference between the x- and y-values found in all data sets and accumulates those differences separately for x and y. If any of the accumulated differences exceeds the tolerance (currently `1e-7`) the postprocessing tool prints a warning.
# Todo
Extract the same data from FITS-IDI and compare for numerical equivalence

64
ES085A/test_description.txt

@ -1,64 +0,0 @@
Test numerical reproduceability of j2ms2 and tConvert
-----------------------------------------------------
This test contains three raw datasets:
- raw SFXC output files from job 24427 (24427/*.cor), from experiment ES085A.
This data was chosen since it has a strong maser line, easily observable off center in subband 3.
- known-good MeasurementSet produced from that: gold/es085a_cont.ms
- known-good FITS-IDI file produced from gold/es085a_cont.ms: gold/ES085A_CONT.IDI
And two support files:
- VEX file(s) for the experiment ES085A.vix and es085a.vix, who are identical/one is symlink of the other
(The j2ms2 tool expects a VEX file name of the name of the directory it is run in under certain circumstances).
- The "lis" file, used to control the input to j2ms2 to exactly (re)produce a given MeasurementSet
Then, starting from raw correlator output, VEX file and "lis" file four data sets are created:
- j2ms2 is run with lisfile; lisfile incorporates all cor files from the job, producing MeasurementSet "es085a_cont.ms"
- j2ms2 is run "manually" by specifying "-o es085a.ms job/*.cor" on the command line; producing MeasurementSet "es085a.ms"
- both MeasurementSets are converted to FITS-IDI using `tConvert`:
es085a_cont.ms => ES085A_CONT.IDI
es085a.ms => ES085A.IDI
The following tests are performed
=================================
Verify that "all data" from the correlator files is written to the MeasurementSets and FITS-IDI files
-----------------------------------------------------------------------------------------------------
The jive-toolchain-verify tool "compare-ms-idi.py" is used to compute a six-way diff between the two "gold" datasets and the four newly created ones. The tool collects integrated weight and integration time per baseline per source, but does not look at the data content itself.
Any differences reported are due to the toolchain-under-test skipping (or duplicating) data in either of the data formats; the diff output will tell what happened in which format(s).
Verify that all meta data (antenna, source, frequency setup properties) are correctly propagated
-------------------------------------------------------------------------------------------------
The jive-toolchain-verify tool "compare-ms-idi-meta.py" tool is used to compute a six-way diff between the meta data from the two "gold" datasets and the four data files generated by the toolchain under test.
Any differences reported should be due to the new toolchain writing different meta data to (one of) the other formats.
Verification of numerical equality of the data written to the MeasurementSet(s)
-------------------------------------------------------------------------------
A batch script in combination of a postprocessing module for the jiveplot tool is run.
The jiveplot tool is used to:
- select three time ranges (scan 18 19 33 of ES085A)
- compute amplitude+phase (y-axes) as function of frequency (x-axis)
- those are integrated (summed) for each baseline/subband/polarization/source/scan
These values are extracted from two MeasurementSets: gold and toolchain-under-test produced MeasurementSet.
A special-purpose written postprocessing module for jiveplot compares the datasets in two successive plot actions.
It keeps the last plot data set and if a new dataset is plotted, it computes the difference between the x- and y-values found in all data sets and accumulates those differences separately for x and y.
If any of the accumulated differences exceeds the tolerance (currently 1e-7) the postprocessing tool prints a warning.
############################################################
# Todo #
############################################################
Extract the same data from FITS-IDI and compare for numerical equivalence
Loading…
Cancel
Save