A set of tools to verify the operation of the j2ms2 and tConvert programs
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

1.8 KiB

check-multipart-fits.py

Checks if there is data loss between the boundaries of subsequent IDI files produced from a single MeasurementSet. Reports how much observing time (in seconds) was lost across all boundaries.

Usage

The script does not have a -h/--help command line option. In fact it has no options at all.

Run as follows:

    > check-multipart-fits.py '/path/to/exp/*.IDI*' '/path/to/other/*.IDI*'

and wait. The ''s are necessary in order to let the script expand the wildcards in stead of the shell.

Note that hierarchical wildcards are supported too to dig through directories of directories (etc) of FITS-IDI files:

    > check-multipart-fits.py '/path/to/*/*/*.IDI*'

Background

JIVE limits the IDI fits files to <= 2 GB per IDI file in order to not upset 32-bit software. If a MeasurementSet is larger than this, tConvert will break it up into chunks of at most 2 GB / piece, naming the files

    > tConvert input.ms OUTPUT.IDI
    ...
    > ls OUTPUT*
    OUTPUT.IDI1
    OUTPUT.IDI2
    ...
    OUTPUT.IDIn

Due to a bug/erroneous assumption made inside tConvert, an unfixed version could trigger data loss between OUTPUT.IDIm and OUTPUT.IDI(m+1).

This tool checks the time stamps at start of m+1 with the last time stamp in m and sums up all the differences across all m-1 boundaries.

There is a caveat that if a scan boundary/gap would be such that all data from the last scan before the gap is in chunks <=m and data for the first scan after the gap is in chunk m+1, that would count as a loss of gap duration seconds.

However the chances of that are minimal and such occurrences have not yet been observed checking the whole EVN archive (approx. 111 TB of FITS files at the time of writing this).