Daily Pipeline rev_08 Config

[ljgray]

Updated RFI masking

• Uses the new RFIStokesIMask task, with significant improvements to RFI flagging
• Adds RFI flagging based on a high-delay chi-squared metric
• Adds RFI flagging based on sensitivity metric

Sidereal rebinning

Uses the new SiderealRebinner task in place of regridding, which is significantly simpler than the previous regridding task and eliminates the artifacts introduced at this stage. Also removes the ThresholdVisWeightBaseline task which was required to remove regridding artifacts.

The rebinning process results in a difference between the true and effective RA bin centres, so a correction is required. On the daily level, a local gradient is calculated based on a stack dataset with full sidereal and frequency coverage. This correction will be applied after stacking when producing a stack from this data.

Restrict run order for BeamFormCat tasks

Slightly modifies the run order around BeamFormCat tasks. This task can store a large copy of the sidereal stream data in its setup method, so these changes try to (a) block the setup from happening until next can run right afterwards and (b) restrict the number of outputs based on the number of catalogs being loaded so that the task moves to its finish state as soon as the last iteration is done.

Fixes and improvements to file online/offline management

Properly manage weather files, flag checks, and file pad lag.

Dependencies

• Pipeline runner improvements radiocosmology/caput#258 to implement priority and output limits
• radiocosmology/caout#255 and Include module load options and defaults #215 to fix a potential bug when loading modules
• Exclude NaNs in weather data when interpolating #226 to fix a NaN bug in some weather data
• Updates to daily pipeline infrastructure #239 to implement automatic online/nearline file management and additional data checks
• Add a new RFI Flagging task radiocosmology/draco#265 and Chime-specific version of new RFI mask #241 to implement the new RFI mask
• Baseline-dependent rebinner radiocosmology/draco#269 to implement the new rebinning feature
• fix: apply filter to inherited datasets radiocosmology/draco#260 to fix a bug with dataset axis selections
• Save out multiple files per task and tweak Stokes I RFI task radiocosmology/draco#271 to implement multi-output saving for tasks and minor RFI fixes
• ss/sens mask radiocosmology/draco#274 to implement improved sensitivity-based RFI masking
• Maximum likelihood delay power spectrum estimator radiocosmology/draco#262 to implement max-likelihood delay power spectrum estimator and fixes to Gibbs delay power spectrum estimator
• Assorted bug fixes radiocosmology/draco#275 for residual bug fixes
• Additional validation notebook updates Daily_validation#24 for final validation notebook changes
• https://github.com/chime-experiment/cedar-environment/pull/6 for file clear automation
• Modifications to BlendStack and frequency mask radiocosmology/draco#280 tweak to flagging and blending

[sjforeman]

Suggested changes for updated rainfall flagging:

1. Remove rain1mm from flags in type: ch_pipeline.analysis.flagging.DataFlagger block
2. Add FlagRainfall (feat(flagging): add pipeline task for rainfall flagging #244) after DataFlagger:
   • type: ch_pipeline.analysis.flagging.FlagRainfall
     in: sstream_mask3
     out: sstream_mask4
     params:
     accumulation_time: 30.0
     threshold: 1.0
3. Updated later sstream_maskN references for consistency

[ljgray]

Suggested changes for updated rainfall flagging:

1. Remove rain1mm from flags in type: ch_pipeline.analysis.flagging.DataFlagger block

2. Add FlagRainfall (feat(flagging): add pipeline task for rainfall flagging #244) after DataFlagger:

   • type: ch_pipeline.analysis.flagging.FlagRainfall
     in: sstream_mask3
     out: sstream_mask4
     params:
     accumulation_time: 30.0
     threshold: 1.0

3. Updated later sstream_maskN references for consistency

Done

[ssiegelx]

The new data quality metric and rfi mask introduced in #246 and draco #268 be incorporated into the configuration file as follows:

    # Smooth the noise estimates which suffer from sample variance
    - type: draco.analysis.flagging.SmoothVisWeight
      in: tstream_thermal_corrected
      out: tstream_day_smoothweight

    # Mask out the short baselines.  Make a copy since the next task
    # will apply a delay filter in place.
    - type: draco.analysis.flagging.MaskBaselines
      requires: manager
      in: tstream_day_smoothweight
      out: tstream_day_noshort
      params:
        zero_data: No
        mask_short: 20.0
        share: "none"

    # Apply an aggressive delay filter.
    - type: draco.analysis.dayenu.DayenuDelayFilterFixedCutoff
      in: tstream_day_noshort
      out: tstream_day_filtered
      params:
        tauw: 0.400
        single_mask: false
        atten_threshold: 0.0

    # Check consistency of data with noise at high delay.
    - type: draco.analysis.transform.ReduceChisq
      in: tstream_day_filtered
      out: chisq_day_filtered
      params:
        axes:
          - "stack"
        dataset: "vis"
        save: true
        output_name: "chisq_{tag}.h5"

    # Generate an RFI mask from the chi-squared test statistic.
    - type: draco.analysis.flagging.RFIMaskChisqHighDelay
      in: chisq_day_filtered
      out: rfimask2
      params:
        save: true
        output_name: "rfi_mask_chisq_{tag}.h5"

    # Apply the RFI mask. This will modify the data in place.
    - type: draco.analysis.flagging.ApplyTimeFreqMask
      in: [tstream_day_smoothweight, rfimask2]
      out: tstream_day_rfi

Although we have to make a copy of the timestream in order to filter it, which could potentially cause memory issues.

[ljgray]

The new data quality metric and rfi mask introduced in #246 and draco #268 be incorporated into the configuration file as follows:

    # Smooth the noise estimates which suffer from sample variance
    - type: draco.analysis.flagging.SmoothVisWeight
      in: tstream_thermal_corrected
      out: tstream_day_smoothweight

    # Mask out the short baselines.  Make a copy since the next task
    # will apply a delay filter in place.
    - type: draco.analysis.flagging.MaskBaselines
      requires: manager
      in: tstream_day_smoothweight
      out: tstream_day_noshort
      params:
        zero_data: No
        mask_short: 20.0
        share: "none"

    # Apply an aggressive delay filter.
    - type: draco.analysis.dayenu.DayenuDelayFilterFixedCutoff
      in: tstream_day_noshort
      out: tstream_day_filtered
      params:
        tauw: 0.400
        single_mask: false
        atten_threshold: 0.0

    # Check consistency of data with noise at high delay.
    - type: draco.analysis.transform.ReduceChisq
      in: tstream_day_filtered
      out: chisq_day_filtered
      params:
        axes:
          - "stack"
        dataset: "vis"
        save: true
        output_name: "chisq_{tag}.h5"

    # Generate an RFI mask from the chi-squared test statistic.
    - type: draco.analysis.flagging.RFIMaskChisqHighDelay
      in: chisq_day_filtered
      out: rfimask2
      params:
        save: true
        output_name: "rfi_mask_chisq_{tag}.h5"

    # Apply the RFI mask. This will modify the data in place.
    - type: draco.analysis.flagging.ApplyTimeFreqMask
      in: [tstream_day_smoothweight, rfimask2]
      out: tstream_day_rfi

Although we have to make a copy of the timestream in order to filter it, which could potentially cause memory issues.

Looks good, I'll test it out. I have a feeling that we will end up having memory issues. If we do, how much more computationally expensive would it be to do the delay filter for all baselines and then add a min/max baseline length parameter to ReduceChisq? We could potentially get around having to make a copy that way.

[ljgray]

Ah never mind, I see now that the dayenu filter is applied in place

[ssiegelx]

Yeah exactly. I think if it was an issue, then we could just create a single task that does everything (baseline masking, filtering, and sum over the stack axis). It would do this for each time sample and then output a (freq, time) container for masking.

[ljgray]

@ssiegelx I tested with the standard pipeline config and it did unfortunately produce a memory error during analysis.dayenu.DayenuDelayFilterFixedCutoff for the standard pipeline (12 nodes).

If it's straightforward to combine the tasks then that would probably be the best way to go. I can also try increasing the number of nodes per pipeline job to 16, but with our lower allocation this year it wouldn't be ideal

[ssiegelx]

Ok thanks! I'll combine the tasks. I think I should have something by tomorrow.

[ssiegelx]

The updated task has been pushed to draco #268. I am still waiting to get a test job to run. The config should now look like:

    # Smooth the noise estimates which suffer from sample variance
    - type: draco.analysis.flagging.SmoothVisWeight
      in: tstream_thermal_corrected
      out: tstream_day_smoothweight

    # Apply an aggressive delay filter and
    # check consistency of data with noise at high delay.
    - type: draco.analysis.dayenu.DayenuDelayFilterFixedCutoff
      in: tstream_day_smoothweight
      out: chisq_day_filtered
      params:
        tauw: 0.400
        single_mask: false
        atten_threshold: 0.0
        reduce_baseline: true
        mask_short: 20.0
        save: true
        output_name: "chisq_{tag}.h5"

    # Generate an RFI mask from the chi-squared test statistic.
    - type: draco.analysis.flagging.RFIMaskChisqHighDelay
      in: chisq_day_filtered
      out: rfimask2
      params:
        save: true
        output_name: "rfi_mask_chisq_{tag}.h5"

    # Apply the RFI mask. This will modify the data in place.
    - type: draco.analysis.flagging.ApplyTimeFreqMask
      in: [tstream_day_smoothweight, rfimask2]
      out: tstream_day_rfi