Testing Distributed Loader #48
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…so our memory monitor can call aggressivelyEvict in time. Also, with this change, the memory growth is a heck of a lot more controlled.
…nd avoid calling rdf.GetUid. Plus, other changes across the entire code base to accommodate this.
… is found, stop processing immediately. Switch the system to sync.WaitGroup completely, instead of an error channel. Log fatal on error in loader and uidassigner
…lower to run these goroutines, but they keep the memory usage in check.
|
Reviewed 12 of 12 files at r1. Comments from the review on Reviewable.io |
|
Review status: Comments from the review on Reviewable.io |
all files reviewed at latest revision, all discussions resolved, all commit checks successful.
danielmai
pushed a commit
that referenced
this pull request
May 1, 2020
arijitAD
pushed a commit
that referenced
this pull request
Oct 15, 2020
shivaji-dgraph
pushed a commit
that referenced
this pull request
Mar 12, 2024
…ch based on vector indexes (#48) - [Float] data type with an @Embedding directive in GraphQL is mapped to "VFLOAT" data type in dgraph - @Embedding directive is allowed only for [Float] data type - Add querySimilar<Type> query that performs similarity search based on HNSW (vector) indexes for types with fields having @Embedding directive - querySimilar<Type> accepts 3 arguments 1. id - unique id for the object to run similarity search on. 2. <predicate> - An enum of embedding predicates defined in the type to base the search on 3. <topK> - number of nearest neighbors to return
harshil-goel
pushed a commit
that referenced
this pull request
Mar 12, 2024
…ch based on vector indexes (#48) - [Float] data type with an @Embedding directive in GraphQL is mapped to "VFLOAT" data type in dgraph - @Embedding directive is allowed only for [Float] data type - Add querySimilar<Type> query that performs similarity search based on HNSW (vector) indexes for types with fields having @Embedding directive - querySimilar<Type> accepts 3 arguments 1. id - unique id for the object to run similarity search on. 2. <predicate> - An enum of embedding predicates defined in the type to base the search on 3. <topK> - number of nearest neighbors to return
harshil-goel
pushed a commit
that referenced
this pull request
Apr 17, 2024
…ch based on vector indexes (#48) - [Float] data type with an @Embedding directive in GraphQL is mapped to "VFLOAT" data type in dgraph - @Embedding directive is allowed only for [Float] data type - Add querySimilar<Type> query that performs similarity search based on HNSW (vector) indexes for types with fields having @Embedding directive - querySimilar<Type> accepts 3 arguments 1. id - unique id for the object to run similarity search on. 2. <predicate> - An enum of embedding predicates defined in the type to base the search on 3. <topK> - number of nearest neighbors to return
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Don't assign any uids while running distributed loader.
Store any errors encountered while running loader pipeline, and send them back.
Run gentlyMerge via goroutines -- they keep the memory usage in check.
Limit the number of goroutines run for gentlyMerge.
Randomly pick keys from dirtyMap; otherwise, we'll always pick the same keys for merging.
Decrease memory allocations via various optimizations.