otel-collector migration fails on new install

[mrj0]

Bug description

Hello, I keep running into this issue while setting up a new cluster and thought I should report it. I had problems with replicas: 2 in the helm override file. Two collectors would start and both try to migrate the database schema, both crash and mark the migration as dirty.

So I decided to start with one collector and then attempted to add a second after the migration completed and everything was running. Every time I do that I get a migration error from the new pod:

2023-09-29T05:51:01.896Z	info	clickhouselogsexporter/exporter.go:455	Running migrations from path: 	{"kind": "exporter", "data_type": "logs", "name": "clickhouselogsexporter", "test": "/logsmigrations"}
Error: failed to build pipelines: failed to create "clickhouselogsexporter" exporter for data type "logs": cannot configure clickhouse logs exporter: clickhouse Migrate failed to run, error: migration failed in line 0: RENAME TABLE IF EXISTS signoz_logs.logs_atrribute_keys TO signoz_logs.logs_attribute_keys on CLUSTER cluster; (details: code: 57, message: There was an error on [clickhouse-shard0-1.clickhouse-headless.platform.svc.cluster.local:9000]: Code: 57. DB::Exception: Table signoz_logs.logs_attribute_keys already exists. (TABLE_ALREADY_EXISTS) (version 23.8.2.7 (official build)))
2023/09/29 05:51:03 application run finished with error: failed to build pipelines: failed to create "clickhouselogsexporter" exporter for data type "logs": cannot configure clickhouse logs exporter: clickhouse Migrate failed to run, error: migration failed in line 0: RENAME TABLE IF EXISTS signoz_logs.logs_atrribute_keys TO signoz_logs.logs_attribute_keys on CLUSTER cluster; (details: code: 57, message: There was an error on [clickhouse-shard0-1.clickhouse-headless.platform.svc.cluster.local:9000]: Code: 57. DB::Exception: Table signoz_logs.logs_attribute_keys already exists. (TABLE_ALREADY_EXISTS) (version 23.8.2.7 (official build)))

Given that information (and really knowing nothing at all about clickhouse), I jumped on the shard and found that both tables exist:

$ k exec -n platform -it clickhouse-shard0-1 -- /bin/bash
I have no name!@clickhouse-shard0-1:/$ clickhouse client
ClickHouse local version 23.8.2.7 (official build).

clickhouse-shard0-1.clickhouse-headless.platform.svc.cluster.local :) use signoz_logs

USE signoz_logs

Query id: c2f528ab-d3b3-486e-ac2d-72fc796f60f9

Ok.

0 rows in set. Elapsed: 0.001 sec. 

clickhouse-shard0-1.clickhouse-headless.platform.svc.cluster.local :) show tables

SHOW TABLES

Query id: 0b441ab5-2963-4307-90f0-2efb85a7fb1d

┌─name────────────────────────────┐
│ atrribute_keys_float64_final_mv │
│ atrribute_keys_int64_final_mv   │
│ atrribute_keys_string_final_mv  │
│ attribute_keys_float64_final_mv │
│ attribute_keys_int64_final_mv   │
│ attribute_keys_string_final_mv  │
│ distributed_logs                │
│ distributed_logs_atrribute_keys │
│ distributed_logs_attribute_keys │
│ distributed_logs_resource_keys  │
│ distributed_tag_attributes      │
│ distributed_usage               │
│ logs                            │
│ logs_atrribute_keys             │
│ logs_attribute_keys             │
│ logs_resource_keys              │
│ resource_keys_string_final_mv   │
│ schema_migrations               │
│ tag_attributes                  │
│ usage                           │
└─────────────────────────────────┘

20 rows in set. Elapsed: 0.002 sec. 

clickhouse-shard0-1.clickhouse-headless.platform.svc.cluster.local :) select * from schema_migrations

SELECT *
FROM schema_migrations

Query id: 38dee4e7-535c-4d97-b069-18ca1aae8142

┌─version─┬─dirty─┬────────────sequence─┐
│       4 │     0 │ 1695966869787088096 │
└─────────┴───────┴─────────────────────┘
┌─version─┬─dirty─┬────────────sequence─┐
│       1 │     1 │ 1695966868450444053 │
│       1 │     0 │ 1695966869224612331 │
│       2 │     1 │ 1695966869226137021 │
│       2 │     0 │ 1695966869341372930 │
│       3 │     1 │ 1695966869342918047 │
│       3 │     0 │ 1695966869672044189 │
└─────────┴───────┴─────────────────────┘
┌─version─┬─dirty─┬────────────sequence─┐
│       5 │     1 │ 1695966869788557737 │
└─────────┴───────┴─────────────────────┘
┌─version─┬─dirty─┬────────────sequence─┐
│       4 │     1 │ 1695966869673624443 │
└─────────┴───────┴─────────────────────┘

9 rows in set. Elapsed: 0.003 sec. 

It appeared that logs_atrribute_keys and logs_attribute_keys had the same values, so I went ahead and dropped logs_atrribute_keys. I still had trouble starting now so I ran:

clickhouse-shard0-1.clickhouse-headless.platform.svc.cluster.local :) drop table logs_atrribute_keys

DROP TABLE logs_atrribute_keys

Query id: 6ac28d8b-0a7a-40e5-9a3b-ea2e7f44fe23

Ok.

clickhouse-shard0-1.clickhouse-headless.platform.svc.cluster.local :) alter table schema_migrations update dirty = 0 where version = 5;

ALTER TABLE schema_migrations
    UPDATE dirty = 0 WHERE version = 5

Query id: cc31d007-6438-45b5-a676-f6147376ce9e

Ok.

Expected behavior

Otel-collector starts more than one pod.

How to reproduce

1. Clean install using helm. I'm using an external clickhouse but I think I had the same issue with the bundled chart
2. Install signoz
3. Edit the collector deploy and change replicas: 2

Version information

NAME: signoz
LAST DEPLOYED: Thu Sep 28 22:37:19 2023
NAMESPACE: platform
STATUS: deployed
REVISION: 1
NOTES:

1. You have just deployed SigNoz cluster:

• frontend version: '0.29.3'
• query-service version: '0.29.3'
• alertmanager version: '0.23.4'
• otel-collector version: '0.79.7'
• otel-collector-metrics version: '0.79.7'

Thank you! Hopefully that helps somebody.

[welcome]

Thanks for opening this issue. A team member should give feedback soon. In the meantime, feel free to check out the contributing guidelines.

[mrj0]

Well, I thought I had made it work but I found errors in the clickhouse logs

"2023.09.29 22:21:56.608701 [ 391 ] {013b2740-8462-42cc-86b5-95a402de3460} <Error> executeQuery: Code: 60. DB::Exception: Table signoz_logs.logs_atrribute_keys does not exist. Maybe you meant logs_attribute_keys?. (UNKNOWN_TABLE) (version 23.8.2.7 (official build)) (from 172.16.200.187:60578) (in query: INSERT INTO signoz_logs.logs (timestamp, observed_timestamp, id, trace_id, span_id, trace_flags, severity_text, severity_number, body, resources_string_key, resources_string_value, attributes_string_key, attributes_string_value, attributes_int64_key, attributes_int64_value, attributes_float64_key, attributes_float64_value) VALUES), Stack trace (when copying this message, always include the lines below):"

Instead I recreated the old since the migration is done anyhow

clickhouse-shard0-1.clickhouse-headless.platform.svc.cluster.local :) create table logs_atrribute_keys as logs_attribute_keys

CREATE TABLE logs_atrribute_keys AS logs_attribute_keys

Query id: 367c0d97-9648-4ca5-a47b-c51a33b210f1

Ok.

0 rows in set. Elapsed: 0.019 sec. 

clickhouse-shard0-1.clickhouse-headless.platform.svc.cluster.local :) insert into logs_atrribute_keys select * from logs_attribute_keys

INSERT INTO logs_atrribute_keys SELECT *
FROM logs_attribute_keys

Query id: 78977440-6031-4e24-976b-18a87419d742

Ok.

0 rows in set. Elapsed: 0.007 sec. 

[srikanthccv]

See SigNoz/signoz-otel-collector#156.