Memory Limiter processor should be configurable to drop the data.

[Harnoor-se7en]

Is your feature request related to a problem? Please describe.
We use Kong API gateway 2.8x and its Zipkin plugin to export traces. Unfortunately, this Zipkin plugin uses a batch queue with no max size limit (i.e. we can't specify the max number of batches that can be queued before the oldest batch is dropped when a new one is queued).
So in case, Otel-collector is overloaded (the batch queue is full), the memory limiter would refuse the data and the source(sender: Zipkin plugin) would retry sending the same batch. This will put pressure on the source.

Describe the solution you'd like
There should be a configurable option such that the memory limiter simply drops data instead of refusing it so that the sender does not receive backpressure.

[andrzej-stencel]

What response would you expect the OpenTelemetry Collector to return to the Kong's Zipkin plugin in such case? A 200 OK? This doesn't seem right to me 😅

Is using the Kong OpenTelemetry plugin instead of the Zipkin plugin an option?

[Harnoor-se7en]

Yes 200 OK isn't right, but the response returned is 500. I think the client should receive 429 and Retry-After header maybe? So that client is aware that the error is non-permanent.

Is using the Kong OpenTelemetry plugin instead of the Zipkin plugin an option?

Sadly currently we are not using Kong 3.x version, hence cannot use the above.

[andrzej-stencel]

Sadly currently we are not using Kong 3.x version, hence cannot use the above.

That's fair, thanks for the explanation.

Can you share your collector configuration (simplified if possible)? I'm assuming you are using the Zipkin receiver, is this correct?

The Zipkin receiver indeed always returns the 500 Internal Server Error regardless of the downstream issue in the pipeline (source). I believe this is incorrect. Similar issue is currently being fixed for the OTLP receiver: #7486, I believe this needs to be fixed in the Zipkin receiver too, and possibly in other receivers.

[andrzej-stencel]

This seems to be a wider problem that is currently not solved, here's the open issue:

• Ensure reliable data delivery in erroneous situations #7460

and more specifically for OTLP receiver:

• Ensure OTLP receiver handles consume errors correctly #4335

The Zipkin receiver suffers from the same issue.

[Harnoor-se7en]

I'm assuming you are using the Zipkin receiver, is this correct?

Yes, we are using Zipkin receiver.

Thanks a lot, @astencel-sumo for sharing similar issues. I see this issue has been discussed considerably and multiple receivers are not strictly implemented according to the receiver's API contract.

I will see if I can fix this issue in the Zipkin receiver.