Small Bug: Not filling sparse empty time stamps #2

Closed
opened 8 months ago by marcote · 2 comments
marcote commented 8 months ago

I have seen this bug in a few cases, all of them replicated by the following steps:

  • Opening antab_editor for a given observation, loading multiple antab files.
  • For a given telescope, flag a few outlier Tsys measurements. In that sense that there is only one or two columns for a given time stamp without values. The rest of IFs are kept.
  • Select "fill empty values" > "Interpolate".
  • The empty time stamps / IFs do not get filled.

I guess it may be due to the way that antab_editor checks for empty values, so it may not get realized about these empty cells in particular.

You can test this issue with the experiment EM144B stored at jop83:/jop83_0/pipe/in/marcote/em144b. You can directly load the current *.editor files, where the one for Ir contain a few of these empty cells (flagged from existing data in the original antabfs file) and the interpolation seems to not work to fill them.

I have seen this bug in a few cases, all of them replicated by the following steps: - Opening `antab_editor` for a given observation, loading multiple `antab` files. - For a given telescope, flag a few outlier Tsys measurements. In that sense that there is only one or two columns for a given time stamp without values. The rest of IFs are kept. - Select "fill empty values" > "Interpolate". - The empty time stamps / IFs do not get filled. I guess it may be due to the way that `antab_editor` checks for empty values, so it may not get realized about these empty cells in particular. You can test this issue with the experiment `EM144B` stored at `jop83:/jop83_0/pipe/in/marcote/em144b`. You can directly load the current `*.editor` files, where the one for `Ir` contain a few of these empty cells (flagged from existing data in the original `antabfs` file) and the interpolation seems to not work to fill them.
Owner

Thanks for the report. The problem turned out to be double timestamp values. This caused the interpolation algorithm to return "Not a number" values, which is exactly what we use to represent non existing values (the internal representation is done using pandas).

I added a warning message indicating the double timestamps, when this problem occurs. This will stop the interpolation, so it will not fix the problem, since I'm not sure which of the 2 timestamps to use. I can imagine that if this occurs frequently a button of some sort to remove double values mgiht be useful (maybe just use the last of the duplicate timestamps?).

Thanks for the report. The problem turned out to be double timestamp values. This caused the interpolation algorithm to return "Not a number" values, which is exactly what we use to represent non existing values (the internal representation is done using pandas). I added a warning message indicating the double timestamps, when this problem occurs. This will stop the interpolation, so it will not fix the problem, since I'm not sure which of the 2 timestamps to use. I can imagine that if this occurs frequently a button of some sort to remove double values mgiht be useful (maybe just use the last of the duplicate timestamps?).
Poster

Ok! Sorry I actually did not noticed that the timestamps were exactly the same. I imagine that there would belong to TPI measurements taken in a fraction of a second or a few seconds (so intrinsically in two different time stamps). But then during the creation of the ANTAB file the writen timestamps were the same as they are typically rounded to 0.01 min (~6 s).

The warning will be fine. Thanks!

Ok! Sorry I actually did not noticed that the timestamps were exactly the same. I imagine that there would belong to TPI measurements taken in a fraction of a second or a few seconds (so intrinsically in two different time stamps). But then during the creation of the ANTAB file the writen timestamps were the same as they are typically rounded to 0.01 min (~6 s). The warning will be fine. Thanks!
marcote closed this issue 8 months ago
Sign in to join this conversation.
No Label
No Milestone
No Assignees
2 Participants
Notifications
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.