Jaeger Ingester crashes V1.32

[HaroonSaid]

Describe the bug
jaeger ingester crashing in 1.31 and 1.32 (debug trace)

{"level":"debug","ts":1648068260.686414,"caller":"consumer/consumer.go:138","msg":"Got msg","msg":{"Headers":null,"Timestamp":"0001-01-01T00:00:00Z","BlockTimestamp":"0001-01-01T00:00:00Z","Key":null,"Value":null,"Topic":"tools_tools_us_dev1_jaeger_spans","Partition":0,"Offset":56823}}
{"level":"debug","ts":1648068260.686429,"caller":"consumer/consumer.go:138","msg":"Got msg","msg":{"Headers":null,"Timestamp":"0001-01-01T00:00:00Z","BlockTimestamp":"0001-01-01T00:00:00Z","Key":"MDhlMjI5MDBkODZlN2M3NWZmZTc1MzBkNmNhYzIwZmE=","Value":"ChAI4ikA2G58df/nUw1srCD6EggMX9Cgwek2ghoRL21haW4vZW52LmV4YW1wbGUyCwiq9+SRBhCArbxkOgQQoMIFQj0KEW90ZWwubGlicmFyeS5uYW1lGihPcGVuVGVsZW1ldHJ5Lkluc3RydW1lbnRhdGlvbi5Bc3BOZXRDb3JlQh8KFG90ZWwubGlicmFyeS52ZXJzaW9uGgcxLjAuMC4wQhoKCWh0dHAuaG9zdBoNNTIuMTMuMjEwLjE3NEISCgtodHRwLm1ldGhvZBoDR0VUQiAKC2h0dHAudGFyZ2V0GhEvbWFpbi9lbnYuZXhhbXBsZUIxCghodHRwLnVybBolaHR0cDovLzUyLjEzLjIxMC4xNzQvbWFpbi9lbnYuZXhhbXBsZUJZCg9odHRwLnVzZXJfYWdlbnQaRk1vemlsbGEvNS4wIChjb21wYXRpYmxlOyBDZW5zeXNJbnNwZWN0LzEuMTsgK2h0dHBzOi8vYWJvdXQuY2Vuc3lzLmlvLylCEwoLZW52aXJvbm1lbnQaBGRldjFCEgoGdGVuYW50Ggh0b29sc191c0IXChBodHRwLnN0YXR1c19jb2RlEAIolANCEwoJc3Bhbi5raW5kGgZzZXJ2ZXJCGQoQb3RlbC5zdGF0dXNfY29kZRoFRVJST1JCCwoFZXJyb3IQASABQh0KFGludGVybmFsLnNwYW4uZm9ybWF0GgVwcm90b1KOAQoGcHJlYm90EiIKEXNlcnZpY2UubmFtZXNwYWNlGg10b29sc191cy1kZXYxEiMKD3NlcnZpY2UudmVyc2lvbhoQMS4wLjAtZGV2ZWxvcC04NBI7ChNzZXJ2aWNlLmluc3RhbmNlLmlkGiQzN2YxNTE0OC0wYzBmLTQwZjQtOTc4ZC00ZjlkMWEwZGQxYWQ=","Topic":"tools_tools_us_dev1_jaeger_spans","Partition":0,"Offset":56824}}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x198f046]

goroutine 168 [running]:
github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/dbmodel.FromDomain.convertProcess(...)
	github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/dbmodel/from_domain.go:123
github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/dbmodel.FromDomain.convertSpanEmbedProcess({0x40, 0xc0004f6540, {0x2094088, 0x1b1eaf2}}, 0xc0004234a0)
	github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/dbmodel/from_domain.go:64 +0x126
github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/dbmodel.FromDomain.FromDomainEmbedProcess(...)
	github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/dbmodel/from_domain.go:43
github.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).WriteSpan(0xc0000b01e0, {0x0, 0x0}, 0xc0004234a0)
	github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:152 +0x7a
github.com/jaegertracing/jaeger/cmd/ingester/app/processor.KafkaSpanProcessor.Process({{0x20a8120, 0x29ebfc0}, {0x20a7fe0, 0xc0000b01e0}, {0x0, 0x0}}, {0x20aa520, 0xc0001821e0})
	github.com/jaegertracing/jaeger/cmd/ingester/app/processor/span_processor.go:67 +0xd3
github.com/jaegertracing/jaeger/cmd/ingester/app/processor/decorator.(*retryDecorator).Process(0xc000166700, {0x20aa520, 0xc0001821e0})
	github.com/jaegertracing/jaeger/cmd/ingester/app/processor/decorator/retry.go:110 +0x37
github.com/jaegertracing/jaeger/cmd/ingester/app/consumer.(*comittingProcessor).Process(0xc000205dd0, {0x20aa520, 0xc0001821e0})
	github.com/jaegertracing/jaeger/cmd/ingester/app/consumer/committing_processor.go:44 +0x5e
github.com/jaegertracing/jaeger/cmd/ingester/app/processor.(*metricsDecorator).Process(0xc000290e40, {0x20aa520, 0xc0001821e0})
	github.com/jaegertracing/jaeger/cmd/ingester/app/processor/metrics_decorator.go:44 +0x5b
github.com/jaegertracing/jaeger/cmd/ingester/app/processor.(*ParallelProcessor).Start.func1()
	github.com/jaegertracing/jaeger/cmd/ingester/app/processor/parallel_processor.go:57 +0x42
created by github.com/jaegertracing/jaeger/cmd/ingester/app/processor.(*ParallelProcessor).Start
	github.com/jaegertracing/jaeger/cmd/ingester/app/processor/parallel_processor.go:53 +0x10c

To Reproduce
Steps to reproduce the behavior:

unknown, ingester just trying to read span from Kafka.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Version (please complete the following information):

• OS: [e.g. Linux]
• Jaeger version: [e.g. 1.8]
• Deployment: [e.g. bare metal, Docker, Kubernetes, ...]

What troubleshooting steps did you try?
Try to follow https://www.jaegertracing.io/docs/latest/troubleshooting/ and describe how far you were able to progress and/or which steps did not work.

Additional context
Add any other context about the problem here.

[yurishkuro]

Seems like a duplicate of #3578

[HaroonSaid]

I could do with some help on how to figure out what causes these crashes.
How do I decode debug trace (the above message in Base64) to a protobuf and run it thru a debug flow to re-produce where the crash happens?

[yurishkuro]

closing as duplicate of #3598