grpc request may hang when error message is too large after bumping to tonic v0.12

[BugenZhao]

e2e-sink-test now consistently hangs here:

risingwave/e2e_test/sink/kafka/avro.slt

Lines 189 to 196 in 8d8c136

	statement error test-rw-sink-upsert-avro-err-key
	create sink sink_err from into_kafka with (
	connector = 'kafka',
	topic = 'test-rw-sink-upsert-avro-err',
	properties.bootstrap.server = 'message_queue:29092',
	primary_key = 'int32_field,string_field')
	format upsert encode avro (
	schema.registry = 'http://schemaregistry:8082');

I find that by disabling SCHEMA_REGISTRY_DEBUG here, this issue is gone.

risingwave/ci/docker-compose.yml

Line 269 in 8d8c136

SCHEMA_REGISTRY_DEBUG: 'true'

The only difference is that there won't be backtraces from the schema registry in the error message.

failed to validate sink: config error: all request confluent registry all timeout, req path ["subjects", "test-rw-sink-upsert-avro-err-key", "versions", "latest"], urls http://schemaregistry:8082/
	confluent schema registry error 40401: Subject 'test-rw-sink-upsert-avro-err-key' not found. io.confluent.rest.exceptions.RestNotFoundException: Subject 'test-rw-sink-upsert-avro-err-key' not found.
- io.confluent.rest.exceptions.RestNotFoundException: Subject 'test-rw-sink-upsert-avro-err-key' not found.
-	at io.confluent.kafka.schemaregistry.rest.exceptions.Errors.subjectNotFoundException(Errors.java:78)
-	at io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource.getSchemaByVersion(SubjectVersionsResource.java:154)
-	at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)

Following this insight, I suppose it's because we always encode the ServerError in gRPC (HTTP2) headers (#13282), and there's an outstanding issue where tonic 0.12 will hang forever when the header size exceeds some limit.

risingwave/src/error/src/tonic.rs

Lines 68 to 71 in 8d8c136

	let serialized = bincode::serialize(&source).unwrap();
	let mut metadata = MetadataMap::new();
	metadata.insert_bin(ERROR_KEY, MetadataValue::from_bytes(&serialized));

Upstream issues:

• long headers from the server hang tonic hyperium/tonic#1834
• hang when client receives header response that's too big in http2 hyperium/hyper#3724
• streams: notify recv task upon reset hyperium/h2#791

ATM there seems to be no fix. I'll disable SCHEMA_REGISTRY_DEBUG now as a workaround and open an issue for this.

Originally posted by @BugenZhao in #17889 (comment)

[BugenZhao]

With this configuration exposed, we're able to workaround this issue:

hyperium/tonic#1835

Waiting for a new version to be released.

[BugenZhao]

Workarounded with #18639