Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major slowdowns writing to tile38 #697

Open
TheCycoONE opened this issue Jul 21, 2023 · 2 comments
Open

Major slowdowns writing to tile38 #697

TheCycoONE opened this issue Jul 21, 2023 · 2 comments

Comments

@TheCycoONE
Copy link

Describe the bug
We are experiencing significant slow downs in our write operations. For the past 3 days we've been seeing write speeds for points of approximately 10-30/s, down from 10,000 or more per second typically. We've experienced these slow downs a few times in the past. We expect but cannot yet confirm that it may be related to replication.

We are running 6 instances of tile38, split between two geographically separated datacenters with ~94ms round trip ping time between the two datacenters, with one instance as the leader and all others as replicas. The coordination is managed by redis sentinal.

To Reproduce
This is a behaviour we've observed in production several times, but we do not have steps to reproduce outside of production.

Expected behavior
We expect consistent write speed that can keep up with our load of thousands of points per second.

Logs
Not Applicable

Operating System (please complete the following information):

  • OS: [Rocky Linux 8.6]
  • CPU: [Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz x4]
  • Version: [1.31.0]
  • Container: [VMWare ESX]

Additional context
See attached
tile38_slowness_server_cmds.txt

@tidwall
Copy link
Owner

tidwall commented Jul 31, 2023

Sorry, but unable to help without some way to reproduce the issue.

@Kilowhisky
Copy link
Contributor

Are there a large number of points at the same GPS location? I've found that if you reuse the same static point or have tons of points at the same location it will slow things down substantially as the index's get huge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants