feat(beamform): delete some class list attributes in 'process_finish'.

[ljgray]

These lists can hold fairly large chunks of data which will not get cleared from memory until one pipeline iteration after .finish is called, which often happens out of order

[jrs65]

Oh great, did this work in the end?

[ljgray]

Oh great, did this work in the end?

It helps, but the root of the issue was with the pipeline run order. It calls setup on both BeamFormCat tasks as soon as the sidereal stream is made, which causes both to generate these vis copies. So I'll have another PR to tweak the config

[tristpinsm]

If this is a common performance issue, would it make sense to purge all attributes in SingleTask.finish?

[ljgray]

If this is a common performance issue, would it make sense to purge all attributes in SingleTask.finish?

I don't see any issues with doing that off the top of my head, other than keeping the attribute that handles pipeline state

[jrs65]

Purging all attributes is a little risky as you don't know if the user might want access to them later (for instance for people running the pipeline in a notebook for testing).

I think for the moment doing what you are doing here is good. I think a more general fix needs to go into caput.pipeline. I wondered if there would be a minimal fix that would allow us to remove tasks early, but I don't think there is. We would need to change the way the pipeline tasks signal they are done in a fairly invasive way.

[tristpinsm]

does finish get called if you are running a task in a notebook?

[jrs65]

Yup

[ljgray]

Purging all attributes is a little risky as you don't know if the user might want access to them later (for instance for people running the pipeline in a notebook for testing).

I think for the moment doing what you are doing here is good. I think a more general fix needs to go into caput.pipeline. I wondered if there would be a minimal fix that would allow us to remove tasks early, but I don't think there is. We would need to change the way the pipeline tasks signal they are done in a fairly invasive way.

I've been playing with the run order a bit, and I do think we could at least remove finished tasks after they finish without too much of a change, but if we're going to be making changes like that I'd rather look at modifying the pipeline runner as a whole

[ljgray]

Purging all attributes is a little risky as you don't know if the user might want access to them later (for instance for people running the pipeline in a notebook for testing).

I think for the moment doing what you are doing here is good. I think a more general fix needs to go into caput.pipeline. I wondered if there would be a minimal fix that would allow us to remove tasks early, but I don't think there is. We would need to change the way the pipeline tasks signal they are done in a fairly invasive way.

Assume this means its good to merge?