Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serialization errors when using JSON messages on MSK Connect together with GSR Confluent JDBC Sink #346

Open
armaseg opened this issue Jun 4, 2024 · 0 comments

Comments

@armaseg
Copy link

armaseg commented Jun 4, 2024

Hello,

I’m having the following error when I’m using MSK Connect together with GSR Confluent JDBC Sink. I tried multiple converters, but I get always the same error:

ERROR [gsr|task-0] WorkerSinkTask{id=gsr-0} Error converting message value in topic 'my-topic' partition 1 at offset 50 and timestamp 1716907277779: Converting byte[] to Kafka Connect data failed due to serialization error: (org.apache.kafka.connect.runtime.WorkerSinkTask:547)

My configuration:

connector.class=io.confluent.connect.jdbc.JdbcSinkConnector

connection.url=jdbc:postgresql://aurora-hostname.cluster-1234.eu-west-1.rds.amazonaws.com:5432/mskconnect
table.name.format=workorder
connection.user=username
connection.password=password
insert.mode=insert
pk.mode=none
connection.attempts=1

auto.evolve=true
auto.create=false
tasks.max=1

topics=my-topic

internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter.schemas.enable=false

key.converter=org.apache.kafka.connect.storage.StringConverter
key.converter.schemas.enable=false

value.converter=com.amazonaws.services.schemaregistry.kafkaconnect.jsonschema.JsonSchemaConverter
value.converter.schemas.enable=true
value.converter.region=eu-west-1
value.converter.avroRecordType=GENERIC_RECORD
value.converter.registry.name=my-schema-registry
value.converter.schemaName=my-schema-name
value.converter.compatibility=BACKWARD
value.converter.schemaAutoRegistrationEnabled=true
value.converter.dataFormat=JSON

value.converter.schemas.log.level=DEBUG

My Schema:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "type": "object",
  "properties": {
    "worker_order_number": {
      "type": "string"
    },
    "timestamp": {
      "type": "string"
    },
    "description": {
      "type": "string"
    },
    "ope_plant": {
      "type": "string"
    },
    "order_number": {
      "type": "string"
    },
    "status": {
      "type": "string"
    },
    "setup_time": {
      "type": "string"
    }
  },
  "required": [
    "worker_order_number",
    "timestamp",
    "description",
    "ope_plant",
    "order_number",
    "status",
    "setup_time"
  ]
}

My Message

{
            "worker_order_number": "123",
            "timestamp":"123456789",
            "description": "DESCRIPTION",
            "ope_plant": "OPE_PLANT",
            "order_number": "ORDER_NUMBER",
            "status": "STATUS",
            "setup_time": "SETUP_TIME",
}

Any advice how to resolve this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant