[TASK] Migrate most regexp scala tests to pytest integration tests

[NVnavkumar]

Is your feature request related to a problem? Please describe.
Scala unit tests in the build process do not run in parallel, and therefore slow down the build process. Also, Scala tests are only testing on the unit level and testing against the pure JVM version of regular expressions, which Spark does follow for the most part, except for a few exceptions (see REGEXP_REPLACE(...) which has a position parameter, but Java does not, so this incurs special handling and potential bugs like https://issues.apache.org/jira/browse/SPARK-39107).

Describe the solution you'd like
We should migrate these to integration tests for performance reasons in the build process and for correctness against Spark as a whole as opposed to just versus the JVM.