[SPARK-49741][DOCS] Add spark.shuffle.accurateBlockSkewedFactor to config docs page

[timlee0119]

What changes were proposed in this pull request?

spark.shuffle.accurateBlockSkewedFactor was added in Spark 3.3.0 in https://issues.apache.org/jira/browse/SPARK-36967 and is a useful shuffle configuration to prevent issues where HighlyCompressedMapStatus wrongly estimates the shuffle block sizes when the block size distribution is skewed, which can cause the shuffle reducer to fetch too much data and OOM. This PR adds this config to the Spark config docs page to make it discoverable.

Why are the changes needed?

To make this useful config discoverable by users and make them able to resolve shuffle fetch OOM issues themselves.

Does this PR introduce any user-facing change?

Yes, this is a documentation fix. Before this PR there's no spark.sql.adaptive.skewJoin.skewedPartitionFactor in the Shuffle Behavior section on the Configurations page and now there is.

How was this patch tested?

On the IDE:

Updated:

Was this patch authored or co-authored using generative AI tooling?

No

[JoshRosen]

+1 on documenting this, as this is a useful configuration. Looks like the docs generally match

spark/core/src/main/scala/org/apache/spark/internal/config/package.scala

Lines 1387 to 1397 in f3785fa

	private[spark] val SHUFFLE_ACCURATE_BLOCK_SKEWED_FACTOR =
	ConfigBuilder("spark.shuffle.accurateBlockSkewedFactor")
	.internal()
	.doc("A shuffle block is considered as skewed and will be accurately recorded in " +
	"HighlyCompressedMapStatus if its size is larger than this factor multiplying " +
	"the median shuffle block size or SHUFFLE_ACCURATE_BLOCK_THRESHOLD. It is " +
	"recommended to set this parameter to be the same as SKEW_JOIN_SKEWED_PARTITION_FACTOR." +
	"Set to -1.0 to disable this feature by default.")
	.version("3.3.0")
	.doubleConf
	.createWithDefault(-1.0)

so looks good overall to me, just left one suggestion about moving this down a few rows so it's next to a related configuration in the table.

[JoshRosen]

I see that the spark.shuffle.accurateBlockThreshold configuration is already documented in this table. It looks like we're preexistingly inconsistent about alphabetizing this list.

What do you think about either moving this new configuration a bit further down so it's next to spark.shuffle.accurateBlockThreshold?

[timlee0119]

Done, also updated the test section

[dongjoon-hyun]

To @timlee0119 and @JoshRosen , shall we remove .internal() from the configuration definition together?

spark/core/src/main/scala/org/apache/spark/internal/config/package.scala

Lines 1387 to 1389 in bdea091

	private[spark] val SHUFFLE_ACCURATE_BLOCK_SKEWED_FACTOR =
	ConfigBuilder("spark.shuffle.accurateBlockSkewedFactor")
	.internal()

[dongjoon-hyun]

In addition, the PR title looks wrong to me because we are touching spark.shuffle.accurateBlockSkewedFactor instead of spark.sql.adaptive.....

[HyukjinKwon]

Yeah, I would match the description

[timlee0119]

Sorry for the typo, I've fixed the PR title and removed internal()

[dongjoon-hyun]

+1, LGTM. Thank you.

[HyukjinKwon]

Merged to master.