Browse Source

diameter can be None

To be able to differentiate between FITS-IDI files containing the default
"0.0" diameter and those FITS-IDI files that contain no DIAMETER column at
all, allow antenna diameter to remain None if the value not found in the
FITS-IDI file.

This shows more diffs but give more information about which information was
actually found, thus more useful in checking/debugging.

Documentation in the accompanying .md file changed accordingly to explain
this
diff-output-compression
haavee 8 months ago
parent
commit
1cef4cf5c3
2 changed files with 57 additions and 26 deletions
  1. +34
    -21
      compare-ms-idi-meta.md
  2. +23
    -5
      compare-ms-idi-meta.py

+ 34
- 21
compare-ms-idi-meta.md View File

@@ -102,49 +102,62 @@ This is what the tool reports for the key `(antenna, Ef)`:

```
('antenna', 'Ef') :
ANT Ef: xyz=[4033947.2616 486990.7866 4900430.9915] d=0.0 (0)
offset=[0.0145 0. 0. ] mount=alt-az found in:
ANT Ef: xyz=[4033947.2616 486990.7866 4900430.9915] (0)
d=0.0 offset=[0.0145 0. 0. ]
mount=alt-az found in:
MS: rsm02-prod-j2ms2.ms
DIFF: diameter: 100.0 vs 0.0 found in: (1)
DIFF: diameter: 100.0 vs 0.0 found in: (1)
MS: rsm02-antdiam-j2ms2.ms
DIFF: mount: UnknownMNTSTA#0 vs alt-az found in: (2)
IDI: RSM02-ANTDIAMJ2MS2-PRODTCONVERT.IDI
DIFF: diameter: None vs 0.0, (2)
mount: UnknownMNTSTA#0 vs alt-az found in:
IDI: RSM02-PRODJ2MS2-PRODTCONVERT.IDI
IDI: RSM02-ANTDIAMJ2MS2-PRODTCONVERT.IDI
DIFF: mount: UnknownMNTSTA#0 vs alt-az found in: (3)
IDI: RSM02-PRODJ2MS2-ANTDIAMTCONVERT.IDI
DIFF: diameter: 100.0 vs 0.0, (3)
DIFF: diameter: 100.0 vs 0.0, (4)
mount: UnknownMNTSTA#0 vs alt-az found in:
IDI: RSM02-ANTDIAMJ2MS2-ANTDIAMTCONVERT.IDI
DIFF: offset: [0.013 0. 0. ] vs [0.0145 0. 0. ],(4)
mount: UnknownMNTSTA#0 vs alt-az found in:
DIFF: diameter: None vs 0.0, (5)
offset: [0.013 0. 0.] vs [0.0145 0. 0.],
mount: UnknownMNTSTA#0 vs alt-az found in:
IDI: ../../eg063d/eg063-prod-dev.IDI[1-3]
```

Comparing the reference value **(0)** - the first value found, in
`rsm03-prod-j2ms2.ms`- to values found in the other data sets, four (4)
`rsm03-prod-j2ms2.ms`- to values found in the other data sets, five (5)
unique diffs are found.

- **(0)** The reference value. It contains default antenna diameter of 0.0,
which is used if the diameter is not found in the data set or the default
of 0.0 was written by code that does not propagate the actual value
written in the MeasurementSet by an antenna diameter unaware VEX file
accessor in `j2ms2`

- **(1)** The antenna diameter aware `j2ms2` has propagated the 100.0 m diameter
to the MeasurementSet - that is the only diff between the two
MeasurementSets apparently: completely to expectation

- **(2)** The two versions of `tConvert` can produce the same FITS-IDI file
differences. The diameter unaware `tConvert` does not write it in the
FITS-IDI file and the antenna diameter aware `tConvert` that converts a
MeasurementSet that was produced by the diameter unaware `j2ms2` also
writes the default value of 0.0. Again this is expected.
- **(2)** The diameter unaware `tConvert` does not write this column in the
FITS-IDI file, hence the diameter is `None`

- **(3)** This is the fully updated antenna diameter toolchain: antenna aware
- **(3)** The diameter aware `tConvert` does propagate the value to the
FITS-IDI file but the source MeasurementSet did not contain the correct
100.0 m value on account of being created by the diameter unaware `j2ms2`
so the only difference is the station mount

- **(4)** This is the fully updated antenna diameter toolchain: antenna aware
`j2ms2` and `tConvert`. The difference with the reference data set
includes the 100.0 m diameter being found in this FITS-IDI file

- **(4)** The rather disjoint FITS-IDI files from a different experiment thrown
- **(5)** The rather disjoint FITS-IDI files from a different experiment thrown
in show that the station offset used in experiment EG063 was different
than the one used in the reference data set.
than the one used in the reference data set and that the FITS-IDI files
were created by a `tConvert` that does not write the antenna diameter

Finally it shows that the station mount from the MeasurementSet ("alt-az")
does not get translated to a known FITS-IDI enumerated value (see the FITS-IDI standard[^1]).
This may need looking into.
does not get translated to a known FITS-IDI enumerated value (see the
FITS-IDI standard<sup>[1](#fitsidi)</sup> in any of the FITS-IDI files. This may need looking
into.


# References

<a name="fitsidi">1</a>: https://fits.gsfc.nasa.gov/registry/fitsidi.html

+ 23
- 5
compare-ms-idi-meta.py View File

@@ -19,6 +19,7 @@ choice = lambda p, t, f : lambda x: t(x) if p(x) else f(x)
const = lambda c : lambda *_: c
method = lambda f : lambda *args, **kwargs: f(*args, **kwargs)
wrap = lambda f, n=1 : lambda *args, **kwargs: f(*args[0:n]) # transform f to allow many parameters but pass only first n
is_none = partial(operator.is_, None)
Map = lambda fn : partial(map, fn)
Group = lambda n : operator.methodcaller('group', n)
GroupBy = lambda keyfn : partial(itertools.groupby, key=keyfn)
@@ -67,7 +68,7 @@ Niets = NietsNut()

# Use this as keyfn in "sorted(key=...)" to transparently replace a None key
# with Niets, but leaving the sorted objects intact - i.e. leaving the None where it was
sorteable_none = choice(partial(operator.is_, None), const(Niets), identity)
sorteable_none = choice(is_none, const(Niets), identity)

##################################################################################
#
@@ -76,11 +77,28 @@ sorteable_none = choice(partial(operator.is_, None), const(Niets), identity)
#
##################################################################################

# If you want to figure out if an element in a tuple/list happens to be None
# and there is a risk that one of the elements is a numpy.array() you're up
# Sh*t Creek - without a paddle:
#
# >>> import numpy
# >>> a1 = numpy.arange(3)
# >>> None in (a1, a1)
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
# ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
#
# This behaviour is numpy version dependend - older versions will work, newer will give this
# exception FFS.
#
# So we just have to roll our own "is the value None present in this list/tuple?"
contains_none = compose(any, Map(is_none))
SafeCompare = lambda f: lambda *args: False if contains_none(args) else f(*args)
SumAttr = lambda a: lambda o0, o1: setattr(o0, a, getattr(o0, a)+getattr(o1, a))
AppendAttr = lambda a: lambda o0, o1: setattr(o0, a, getattr(o0, a).append(getattr(o1, a)))
FloatEq = lambda l, r: abs(l-r)<1e-3
ArrayEq = lambda l, r: numpy.allclose(l, r, atol=1e-3)
StringEq = lambda l, r: l.lower() == r.lower()
FloatEq = SafeCompare(lambda l, r: abs(l-r)<1e-3)
ArrayEq = SafeCompare(lambda l, r: numpy.allclose(l, r, atol=1e-3))
StringEq = SafeCompare(lambda l, r: l.lower() == r.lower())

###################################3
# For meta-data comparison we need:
@@ -302,7 +320,7 @@ def acc_idi_antenna_table(acc, idi):
an, ag = idi_tbl('ANTENNA', 'ARRAY_GEOMETRY')(idi)
# array geometry table has anname column to link to antenna table
agidx = anname_key_l(ag['ANNAME'])
diameter = lambda i: 0.0 if 'DIAMETER' not in ag.columns.names else ag['DIAMETER'][i]
diameter = lambda i: None if 'DIAMETER' not in ag.columns.names else ag['DIAMETER'][i]
stabxyz, mntsta, staxof = GetN('STABXYZ', 'MNTSTA', 'STAXOF')(ag)
def mk_idi_antenna(acc, station):
name = station.capitalize()


Loading…
Cancel
Save