Using aoquality

The aoquality program is a command-line tool to query or modify the quality statistics in the table. It is more or less a command-line counterpart to aoqplot.

What quality statistics are

Quality statistics are optional tables in a measurement set that describe statistics of the visibility in the measurement set. These can be computed from the visibilities and are therefore redundant, but they are much smaller in size, and therefore alleviate the need to read an entire set just to get some first idea of the quality of the visibilities. They can be calculated during flagging (using the lua collect_statistics() function), which avoids reading the data again just for getting the statistics.

Note

If the visibilities are changed, the quality statistics become out of date. This can be desirable in some cases, e.g. to keep the statistics of the raw, high-resolution data before averaging.

(Re)collecting statistics

To collect statistics, the following command can be used:

aoquality collect [-d <column name>] observation.ms

This will read all the visibilities and add the quality statistics tables to the measurement set afterwards. If quality tables already exist, they are overwritten. If no column name is specified, the DATA column is used.

Extracting and combining statistics

Normally, statistics are written inside a measurement set. The aoquality tool allows the extraction of statistics and to write these into separate (small) files, so that it becomes possible to store statistics separately from the measurement set. The stored statistics have the form of an empty measurement set with quality tables and just enough metadata to interpret the statistics. The aoquality combine option can be used to create such separate statistic sets. The recommended extension for extracted statistics is .qs. For example, to extract statistics from a measurement set and write them in a separate, use:

aoquality combine mystatistics.qs input.ms

This would create mystatistics.qs and fill it with the quality statistics from input.ms. It is also possible to specify multiple measurement sets. In this case, the statistics are combined and then written to the output file, for example:

aoquality combine mystatistics.qs \
  observation1.ms observation2.ms observation3.ms

This would combine the statistics from the three observations and write these to mystatistics.qs. For quality statistics operations, the .qs statistics can be used any time that a .ms can be used, so aoquality query commands can be applied to it, they can be further combined with aoquality combine, and aoqplot can be used to view the contents of the .qs statistics.

Querying statistics

There are several dimensions over which statistics, such as the standard deviation, are stored and can be queried: over time (integrated over all other dimensions), frequency or baseline. To query these, the aoquality tool accepts these commands:

  • aoquality query_b: Query the statistics per baseline (integrated over all other dimensions)
  • aoquality query_f: Query the statistics per channel (integrated over all other dimensions)
  • aoquality query_t: Query the statistics over time (integrated over all other dimensions)
  • aoquality query_g: Query a statistic integrated over all dimensions

Each of these commands uses the same syntax. We will use query_f as an example. To query the standard deviation per channel, one would issue this command:

aoquality query_f StandardDeviation observation.ms

As shown, after query_f the type of statistic is given, followed by the name of the measurement set. A few other common statistics are Mean, Variance, Count (=nr of visibilities), RFIRatio and DStandardDeviation (=frequency-differenced standard deviation).

Here is an example output:

This shows the frequency in the first column, the standard deviation of the real values of XX in the second column, followed by imaginary, then real of XY, etc.

Other options

The aoquality command has a few more options. These can be queried by running aoquality without parameters.