[BUG] Shims inadvertently missed for various artifacts

[gerashegalov]

Describe the bug

It became too easy to miss shims due to different explicit lists maintained for different type of shims. E.g., see #10465, #11052

This is due to the complexity of maintaining the shim lists manually stemming from:

• Number of Shims
• Explicit list of snapshot vs noSnapshot
• Upstream vs Vendor Shims
• Subset of shims such as Upstream [3.3, 4.0) released for Scala 2.12 and Scala 2.13
• Subset of shims such future 4.0 released just for Scala 2.13
• All vendor shims just for Scala 2.12

It is clear that this is getting out of control even with the approaching cut of Spark 3.1.x shims per #10955

This issue proposes to generate these lists from a single source of truth instead of manually updating multiple lists. Each new shim requires a profile in the pom, thus the list of such profiles is a natural choice to be a single source of truth.

So all.buildvers can be obtained via

$ mvn help:all-profiles | awk '/Profile Id: release/ { print substr($3, 8) }' | sort -u
311
312
313
314
320
321
321cdh
322
323
324
330
330cdh
330db
331
332
332cdh
332db
333
334
340
341
341db
342
343
350
351

We can iterate through this list and easily determine whether it points to a SNAPSHOT or not instead of maintaining lists (no)Snapshot.buildvers

$ mvn -pl dist help:evaluate -Dexpression=spark.version -Dbuildver=343
3.4.3

We also have a deterministic convention for Databricks profiles

As for for Scala we have a relatively simple set of rules to code

• [3.1.0, 3.3.0): only 2.12
• [3.3.0, 4.0) 2.12 and 2.13
• [4.0.0,) only 2.13
• Databricks or CDH: only 2.12