[FEA] Find and un-nest withResource where appropriate

[abellina]

We have a nice helper function withResource that has helped us emulate "try-with-resources" semantics from Java in Scala.

This can be abused, however. Here's an example:

withResource(iter.next()) { batch => 
  withResource(GpuColumnVector.from(batch)) { table => 
    withResource(table.callCudfFunction()) { table2 => 
      withResource(table2.callAnotherCudfFunction()) { table3 => 
      }
    }
  }
}

In this code we first transform batch into table, this doesn't incur new GPU memory, but table.callCudfFunction() returns a new table, table2, and so when we call the second cuDF function on table2 we now have in GPU memory: table, table2 and now table3.

Ideally the code above can be broken down into multiple chunks. Note that this isn't possible or clean in all cases, so this isn't an easy task.

val table = withResource(iter.next()) { batch => 
  GpuColumnVector.from(batch)
} 

val table2 = withResource(table) { _ =>
  table.callCudfFunction()
}

// we now only have `table2` in memory, instead of `table` and `table2` prior to making `table3`.

val table3 = withResource(table2) { _ =>
  table2.callAnotherCudfFunction()
}

First work on this issue should probably be to find big parts of the code where we want to focus, perhaps with some smarts to detect the issue. I could see several PRs being generated, per exec that suffers from the above, as we go through the code.

[abellina]

The original intent of this issue was to figure out how to instrument withResource to figure out memory growth, but we made inroads into memory waste by looking at the nested level, and by brute force running benchmarks with larger scales and less resources. That said, I think we should move towards measuring memory explosion using the Rmm apis for scoped maximum usage added in 22.12 via #6745, and enforce using #7257