Add indexing pipeline type #3461
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vblagoje
requested review from
ZanSara and
julian-risch
and removed request for
a team and
ZanSara
julian-risch
approved these changes
Oct 24, 2022
| fingerprint = sha1(json.dumps(self.get_config(), sort_keys=True).encode()).hexdigest() | ||
| run_total = self.run.counter + self.run_batch.counter | ||
| send_custom_event( | ||
| "pipeline", | ||
| payload={ | ||
| "fingerprint": fingerprint, | ||
| "type": self.get_type(), | ||
| "type": "Indexing" if is_indexing else self.get_type(), |
There was a problem hiding this comment.
It would be nice to get more info about the components used in the indexing pipeline. As inspiration here is the tutorial on indexing: https://haystack.deepset.ai/tutorials/doc-class-index
It would be great if the "type" could contain TextConverter, PreProcessor, FileTypeClassifier, PDFToTextConverter, DocxToTextConverter instead. It would be okay to get this done in a follow-up PR.
There was a problem hiding this comment.
What do you think? For now just add a separate issue for that to the backlog?
There was a problem hiding this comment.
Perhaps, let's get it working with the simplest solution possible and then improve.
ZanSara
added
type:feature
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related Issues
Proposed Changes:
We didn't account for indexing pipelines. Now we detect them based on
file_pathsparameter being used.How did you test it?
Manual telemetry tests are being prepared if the general approach is approved.
Notes for the reviewer
Best to involve a manual test with access to telemetry to verify the actual pipeline is being detected properly
Checklist