[SPARK-49249][SPARK-49122][Connect][SQL] Add addArtifact API to the Spark SQL Core

[xupefei]

What changes were proposed in this pull request?

This PR improves Spark SQL Core by adding a bunch of addArtifact APIs to SparkSession. These APIs were first introduced to Spark Connect a while ago.

The follow-up task is for PySpark.

Why are the changes needed?

To close the API compatibility gap between Spark Connect and Spark Classic.

Does this PR introduce any user-facing change?

Yes, users will be able to use some new APIs.

How was this patch tested?

Added new tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[hvanhovell]

Look pretty good!

[hvanhovell]

@xupefei can you check if this works with UDFs?

[xupefei]

@xupefei can you check if this works with UDFs?

I checked and found it doesn't work. The Spark Core session is currently not using the class loader provided by ArtifactManager:

spark/common/utils/src/main/scala/org/apache/spark/util/SparkClassUtils.scala

Lines 28 to 47 in 11b682c

	def getContextOrSparkClassLoader: ClassLoader =
	Option(Thread.currentThread().getContextClassLoader).getOrElse(getSparkClassLoader)
	// scalastyle:off classforname
	/**
	* Preferred alternative to Class.forName(className), as well as
	* Class.forName(className, initialize, loader) with current thread's ContextClassLoader.
	*/
	def classForName[C](
	className: String,
	initialize: Boolean = true,
	noSparkClassLoader: Boolean = false): Class[C] = {
	if (!noSparkClassLoader) {
	Class.forName(className, initialize, getContextOrSparkClassLoader).asInstanceOf[Class[C]]
	} else {
	Class.forName(className, initialize, Thread.currentThread().getContextClassLoader).
	asInstanceOf[Class[C]]
	}
	// scalastyle:on classforname
	}

For now we have to wrap the code with artifactManager.withResources to replace the class loader. THis issue needs to be fixed.

[hvanhovell]

One small nit...

[hvanhovell]

LGTM