pantab 4.0 checklist

[WillAyd]

• Improve error messages (don't just use RuntimeError)
• Add tests for writing other libraries than pandas
• Work upstream in pandas to improve dataframe interchange (or just use Arrow C Data)
  • date32 support
  • nullable integer support
  • nullable boolean support
  • proper serialization of arrow dtypes
• proper benchmarks
• communication around improvements

some nice to haves, but not blockers:

• reuse hyper connection

[WillAyd]

The current PR in #218 leverages the dataframe interchange protocol. However, after talking with Joris on the pandas team I am more convinced that using the new Arrow C Data protocol is a better bet. I have started that redevelopment which does simplify things, but I think am also hitting bugs with the PyCapsule exchange of information (this was just released in PyArrow 14.0)

[WillAyd]

• Add tests for writing other libraries than pandas

The latest design ended up using the Arrow C Data interchange, which hasn't been added upstream in pandas yet. So for now we require pyarrow and require pandas -> pyarrow -> pantab, but in the future we can just inspect a frame object for __arrow_array_stream__ without pyarrow and serialize from there.

See pandas-dev/pandas#56587 for pandas