Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added batch embedding computing #86

Merged
merged 19 commits into from
Mar 6, 2024
Merged

Added batch embedding computing #86

merged 19 commits into from
Mar 6, 2024

Conversation

Wats0ns
Copy link

@Wats0ns Wats0ns commented Feb 28, 2024

Now the embedding are batch computed when building the KB. This allows for faster KB building

McPatate
McPatate reviewed Feb 28, 2024
View reviewed changes
Copy link
Member

@McPatate McPatate left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks nice!

crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
McPatate
McPatate reviewed Feb 28, 2024
View reviewed changes
crates/llm-ls/src/backend.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/backend.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/backend.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/backend.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/retrieval.rs Show resolved Hide resolved
crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/tinyvec-embed/src/db.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/tinyvec-embed/src/db.rs Outdated Show resolved Hide resolved
McPatate
McPatate reviewed Feb 28, 2024
View reviewed changes
Copy link
Member

@McPatate McPatate left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Starting to look ready!

crates/llm-ls/src/backend.rs Outdated Show resolved Hide resolved
McPatate
McPatate reviewed Feb 29, 2024
View reviewed changes
Copy link
Member

@McPatate McPatate left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last round of review I think 😄

crates/tinyvec-embed/src/db.rs Show resolved Hide resolved
crates/tinyvec-embed/src/db.rs Outdated Show resolved Hide resolved
crates/tinyvec-embed/src/error.rs Outdated Show resolved Hide resolved
Quentin Maire and others added 16 commits March 4, 2024 18:14
KB is now processed in batch
b48539c
From unwrap to ?
d2b5541
@Wats0ns @McPatate
Update crates/llm-ls/src/retrieval.rs
f6d6dcc 
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
@Wats0ns @McPatate
Update crates/llm-ls/src/retrieval.rs
76d9b5e 
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
@Wats0ns @McPatate
Update crates/llm-ls/src/retrieval.rs
2e0a975 
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
@Wats0ns @McPatate
Update crates/llm-ls/src/retrieval.rs
a90089a 
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
@Wats0ns @McPatate
Update crates/llm-ls/src/retrieval.rs
d6d8f7d 
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
@Wats0ns @McPatate
Update crates/llm-ls/src/retrieval.rs
c07e833 
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
Corrections after feedbacks
770e326
Corrections after feedbacks
0fd9d51
Corrections after feedbacks
c005ddf
@Wats0ns @McPatate
Update crates/llm-ls/src/backend.rs
b6f721e 
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
@Wats0ns @McPatate
Update crates/tinyvec-embed/src/db.rs
00d7669 
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
@Wats0ns @McPatate
Update crates/tinyvec-embed/src/db.rs
e9881f2 
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
@Wats0ns @McPatate
Update crates/tinyvec-embed/src/error.rs
17ea54a 
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
Corrections after feedbacks
837c99e
McPatate
McPatate reviewed Mar 6, 2024
View reviewed changes
crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/tinyvec-embed/src/db.rs Outdated Show resolved Hide resolved
McPatate
McPatate reviewed Mar 6, 2024
View reviewed changes
Copy link
Member

@McPatate McPatate left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there!

Could you run cargo fmt before committing please?

crates/llm-ls/src/retrieval.rs Outdated Show resolved Hide resolved
crates/tinyvec-embed/src/error.rs Outdated Show resolved Hide resolved
McPatate
McPatate approved these changes Mar 6, 2024
View reviewed changes
Copy link
Member

@McPatate McPatate left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work!

crates/tinyvec-embed/src/error.rs Outdated Show resolved Hide resolved
@McPatate McPatate merged commit 64a4c38 into huggingface:feat/multi_file_context Mar 6, 2024
0 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@McPatate McPatate McPatate approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Wats0ns @McPatate