Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cvat-vector deployment crashing on a fresh Kubernetes deployment #8455

Closed
2 tasks done
RedrumSherlock opened this issue Sep 18, 2024 · 2 comments
Closed
2 tasks done
Labels
bug Something isn't working need info Need more information to investigate the issue

Comments

@RedrumSherlock
Copy link

Actions before raising this issue

  • I searched the existing issues and did not find anything similar.
  • I read/searched the docs

Steps to Reproduce

  1. git clone the latest develop branch of cvat
  2. follow the instruction on https://docs.cvat.ai/docs/administration/advanced/k8s_deployment_with_helm/#with-overrides to install the helm chart on Google Kubernetes Engine clusters
  3. Found that all the other deployments are running properly, but the cvat-vector deployment pods are crashing, with the following error messages

2024-09-18T19:16:12.808567Z INFO vector::app: Internal log rate limit configured. internal_log_rate_secs=10 │ │ 2024-09-18T19:16:12.808875Z INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info" │ │ 2024-09-18T19:16:12.809073Z INFO vector::app: Loading configs. paths=["/etc/vector"] │ │ 2024-09-18T19:16:12.809609Z ERROR vector::cli: Configuration error. error=No sources defined in the config. │ │ 2024-09-18T19:16:12.809623Z ERROR vector::cli: Configuration error. error=No sinks defined in the config. │ │ Stream closed EOF for cvat/cvat-vector-0 (vector)

Expected Behavior

All the deployments would start correctly

Possible Solution

No response

Context

Since cvat-vector is down, when I am trying to create a superuser ,I got the following error and was not able to do it. It looks the superuser creation is dependent on the cvat-vector instance. Thus I am not able to use the system at all.

2024-09-18 18:33:11: exception: An error occurred while sending events: HTTPConnectionPool(host='cvat-vector', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7c03a09bd330>: Failed to establish a new connection: [Errno 111] Connection refused'))

Environment

Kuberentes on GKE
Cluster version: 1.28.13
Node OS image: Container-Optimized OS with containerd (cos_containerd)
@RedrumSherlock RedrumSherlock added the bug Something isn't working label Sep 18, 2024
@azhavoro
Copy link
Contributor

azhavoro commented Sep 19, 2024

Please make sure that cvat-vector-config cinfigMap is correctly created and vector config file from this cm is mounted into vector container.
What is output of kubectl exec <your-release-name>-vector-0 -- cat /etc/vector/vector.toml ?

@azhavoro azhavoro added the need info Need more information to investigate the issue label Sep 19, 2024
@RedrumSherlock
Copy link
Author

Actually I found that it is because the symbolic link is broken on my local machine and thus the new vector helm on ../components are not being used. After fixing the symbolic link the vector configmap is correctly configured and it is now working. Thanks for help @azhavoro !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working need info Need more information to investigate the issue
Projects
None yet
Development

No branches or pull requests

2 participants