WIP Initial scoping for upgrade to Python 3

[AmyOctoCat]

Initial work to investigate very roughly how long a migration to python 3 would take.

Note, this is based off of Matt Archer's branch to add integration testing.

Plans for a migration:

• upgrade integration test to run in python3
• update dependencies for compatibility with python3, note that it is not currently possible to bump numpy to v2.x.x because statsmodels isn't compatible with versions above and including numpy 2.
• update the first section of the syntax in source code to python3, this included approx 285 lines of oneflux/partition/nighttime.py which is 966 lines long, ie about 1/4 of nighttime.py, plus the other imported utilities it touches. This included fixing an encoding issue in library.py which was to do with changes to the way python3 represents byte strings (second answer down in this post: https://stackoverflow.com/questions/21957667/numpy-dtype-issues-in-genfromtxt-reads-string-in-as-bytestring). Without detailed knowledge of the structure of the code base, it is difficult to say what fraction of the total code base this has covered. Used pygount to get an idea of the total size of the codebase (pygount excludes whitespace which is why this is different from above):

Summary:

┏━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━━━━┳━━━━━━┓
┃ Language   ┃ Files ┃     % ┃ Code ┃    % ┃ Comment ┃    % ┃
┡━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━━━━╇━━━━━━┩
│ Python     │    26 │  89.7 │ 6925 │ 50.0 │    3547 │ 25.6 │
│ __binary__ │     3 │  10.3 │    0 │  0.0 │       0 │  0.0 │
├────────────┼───────┼───────┼──────┼──────┼─────────┼──────┤
│ Sum        │    29 │ 100.0 │ 6925 │ 50.0 │    3547 │ 25.6 │
└────────────┴───────┴───────┴──────┴──────┴─────────┴──────┘

Breakdown:

121	Python	oneflux	oneflux/__init__.py
0	__binary__	oneflux	oneflux/__init__.pyc
0	Python	oneflux	oneflux/graph/__init__.py
289	Python	oneflux	oneflux/graph/compare.py
8	Python	oneflux	oneflux/local_settings_template.py
0	Python	oneflux	oneflux/partition/__init__.py
113	Python	oneflux	oneflux/partition/auxiliary.py
25	Python	oneflux	oneflux/partition/compu.py
1478	Python	oneflux	oneflux/partition/daytime.py
81	Python	oneflux	oneflux/partition/ecogeo.py
765	Python	oneflux	oneflux/partition/library.py
477	Python	oneflux	oneflux/partition/nighttime.py
9	Python	oneflux	oneflux/pipeline/__init__.py
379	Python	oneflux	oneflux/pipeline/aux_info_files.py
337	Python	oneflux	oneflux/pipeline/common.py
1067	Python	oneflux	oneflux/pipeline/site_data_product.py
575	Python	oneflux	oneflux/pipeline/site_plots.py
695	Python	oneflux	oneflux/pipeline/variables_codes.py
1460	Python	oneflux	oneflux/pipeline/wrappers.py
0	Python	oneflux	oneflux/tools/__init__.py
0	__binary__	oneflux	oneflux/tools/__init__.pyc
92	Python	oneflux	oneflux/tools/partition_dt.py
80	Python	oneflux	oneflux/tools/partition_nt.py
0	__binary__	oneflux	oneflux/tools/partition_nt.pyc
81	Python	oneflux	oneflux/tools/pipeline.py
0	Python	oneflux	oneflux/utils/__init__.py
180	Python	oneflux	oneflux/utils/files.py
4	Python	oneflux	oneflux/utils/helper_fns.py
46	Python	oneflux	oneflux/utils/strings.py

Very roughly I would estimate that the PR in it's current state has updated about 5% of the code base.

• run python 2to3 tool, this picked up some of the more standard syntax changes but has not found more subtle issues and currently still getting runtime issues.
• run pytest-cov plugin to generate code coverage report which is as follows:

---------- coverage: platform darwin, python 3.12.3-final-0 ----------
Name                                            Stmts   Miss  Cover
-------------------------------------------------------------------
tests/python/__init__.py                            0      0   100%
tests/python/integration/test_partitioning.py      66     31    53%
tests/python/test_context.py                        3      1    67%
-------------------------------------------------------------------
TOTAL                                              69     32    54%

This is a fairly blunt instrument, and it is not clear exactly what the coverage is measuring, so it would be good to get more insight into this or try a different tool.

• add unit tests, as there are currently only integration tests, there are 13 modules which look like candidates for unit testing in the oneflux directory, these are very long, with library being 1300 lines.
• add a README for running the tests (a stub has been added so far)
• update the remaining syntax in the project - very rough estimate 95% remaining

My feeling is that the time needed to add appropriate tests and update the syntax to Python3 would be in the ball park of 6 months, rather than 3.

These resources are quite helpful for finding common changes between python2 and python3:
https://sebastianraschka.com/Articles/2014_python_2_3_key_diff.html#python-2-3
https://portingguide.readthedocs.io/en/latest/index.html
issues found so far to look out for:

• https://portingguide.readthedocs.io/en/latest/strings.html
• deprecation of long type, can now use int instead
• deprecation of distutils
• changed urllib request syntax