Skip to content

Commit

Permalink
docs: Reformat code blocks in docstrings (#3580)
Browse files Browse the repository at this point in the history 
* Fix docstrings for DocumentStores

* Fix docstrings for AnswerGenerator

* Fix docstrings for Connector

* Fix docstrings for DocumentClassifier

* Fix docstrings for LabelGenerator

* Fix docstrings for QueryClassifier

* Fix docstrings for Ranker

* Fix docstrings for Retriever and Summarizer

* Fix docstrings for Translator

* Fix docstrings for Pipelines

* Fix docstrings for Primitives

* Fix Python code block spacing

* Add line break before code block

* Fix code blocks

* fix: discard metadata fields if not set in Weaviate (#3578)

* fix weaviate bug in returning embeddings and setting empty meta fields

* review comment

* Update unstable version and openapi schema (#3584)

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* fix: Flatten `DocumentClassifier` output in `SQLDocumentStore`; remove `_sql_session_rollback` hack in tests (#3273)

* first draft

* fix

* fix

* move test to test_sql

* test: add test to check id_hash_keys is not ignored (#3577)

* refactor: Generate JSON schema when missing (#3533)

* removed unused script

* print info logs when generating openapi schema

* create json schema only when needed

* fix tests

* Remove leftover

Co-authored-by: ZanSara <sarazanzo94@gmail.com>

* move milvus tests to their own module (#3596)

* feat: store metadata using JSON in SQLDocumentStore (#3547)

* add warnings

* make the field cachable

* review comment

* Pin faiss-cpu as 1.7.3 seems to have problems (#3603)

* Update Haystack imports (#3599)

* Update Python version (#3602)

* fix: `ParsrConverter` fails on pages without text (#3605)

* try to fix bug

* remove print

* leftover

* refactor: update Squad data  (#3513)

* refractor the to_squad data class

* fix the validation label

* refractor the to_squad data class

* fix the validation label

* add the test for the to_label object function

* fix the tests for to_label_objects

* move all the test related to squad data to one file

* remove unused imports

* revert tiny_augmented.json

Co-authored-by: ZanSara <sarazanzo94@gmail.com>

* Url fixes (#3592)

* add 2 example scripts

* fixing faq script

* fixing some urls

* removing example scripts

* black reformatting

* add labeler to the repo (#3609)

* convert eval metrics to python float (#3612)

* feat: add support for `BM25Retriever` in `InMemoryDocumentStore` (#3561)

* very first draft

* implement query and query_batch

* add more bm25 parameters

* add rank_bm25 dependency

* fix mypy

* remove tokenizer callable parameter

* remove unused import

* only json serializable attributes

* try to fix: pylint too-many-public-methods / R0904

* bm25 attribute always present

* convert errors into warnings to make the tutorial 1 work

* add docstrings; tests

* try to make tests run

* better docstrings; revert not running tests

* some suggestions from review

* rename elasticsearch retriever as bm25 in tests; try to test memory_bm25

* exclude tests with filters

* change elasticsearch to bm25 retriever in test_summarizer

* add tests

* try to improve tests

* better type hint

* adapt test_table_text_retriever_embedding

* handle non-textual docs

* query only textual documents

* Incorporate Reviewer feedback

* refactor: replace `torch.no_grad` with `torch.inference_mode` (where possible) (#3601)

* try to replace torch.no_grad

* revert erroneous change

* revert other module breaking

* revert training/base

* Fix docstrings for DocumentStores

* Fix docstrings for AnswerGenerator

* Fix docstrings for Connector

* Fix docstrings for DocumentClassifier

* Fix docstrings for LabelGenerator

* Fix docstrings for QueryClassifier

* Fix docstrings for Ranker

* Fix docstrings for Retriever and Summarizer

* Fix docstrings for Translator

* Fix docstrings for Pipelines

* Fix docstrings for Primitives

* Fix Python code block spacing

* Add line break before code block

* Fix code blocks

* Incorporate Reviewer feedback

Co-authored-by: Massimiliano Pippi <mpippi@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Stefano Fiorucci <44616784+anakin87@users.noreply.github.com>
Co-authored-by: Julian Risch <julian.risch@deepset.ai>
Co-authored-by: ZanSara <sarazanzo94@gmail.com>
Co-authored-by: Espoir Murhabazi <espoir.mur@gmail.com>
Co-authored-by: Tuana Celik <tuana.celik@deepset.ai>
Co-authored-by: tstadel <60758086+tstadel@users.noreply.github.com>
  • Loading branch information
@brandenchan @masci
@github-actions @anakin87 @julian-risch @ZanSara @espoirMur @TuanaCelik @tstadel
10 people authored Nov 28, 2022
1 parent c6890c3 commit 4a83b20
Show file tree
Hide file tree
Showing 30 changed files with 608 additions and 510 deletions.
7 changes: 7 additions & 0 deletions haystack/document_stores/base.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ def get_all_documents(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -175,6 +176,7 @@ def get_all_documents_generator(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -255,6 +257,7 @@ def get_all_labels_aggregated(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -713,6 +716,7 @@ def query(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -741,6 +745,7 @@ def query(
optionally a list of dictionaries as value.
__Example__:
```python
filters = {
"$or": [
Expand Down Expand Up @@ -816,6 +821,7 @@ def query_batch(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -844,6 +850,7 @@ def query_batch(
optionally a list of dictionaries as value.
__Example__:
```python
filters = {
"$or": [
Expand Down
6 changes: 6 additions & 0 deletions haystack/document_stores/deepsetcloud.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,7 @@ def get_all_documents(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -227,6 +228,7 @@ def get_all_documents_generator(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -340,6 +342,7 @@ def query_by_embedding(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -368,6 +371,7 @@ def query_by_embedding(
optionally a list of dictionaries as value.
__Example__:
```python
filters = {
"$or": [
Expand Down Expand Up @@ -446,6 +450,7 @@ def query(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -474,6 +479,7 @@ def query(
optionally a list of dictionaries as value.
__Example__:
```python
filters = {
"$or": [
Expand Down
2 changes: 2 additions & 0 deletions haystack/document_stores/elasticsearch.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,7 @@ def query_by_embedding(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -334,6 +335,7 @@ def query_by_embedding(
optionally a list of dictionaries as value.
__Example__:
```python
filters = {
"$or": [
Expand Down
2 changes: 2 additions & 0 deletions haystack/document_stores/opensearch.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -369,6 +369,7 @@ def query_by_embedding(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -397,6 +398,7 @@ def query_by_embedding(
optionally a list of dictionaries as value.
__Example__:
```python
filters = {
"$or": [
Expand Down
7 changes: 7 additions & 0 deletions haystack/document_stores/pinecone.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,7 @@ def get_document_count(
operator is provided, `"$eq"` (or `"$in"` if the comparison value is a list) is used as default
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -445,6 +446,7 @@ def update_embeddings(
operator is provided, `"$eq"` (or `"$in"` if the comparison value is a list) is used as default
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -538,6 +540,7 @@ def get_all_documents(
operator is provided, `"$eq"` (or `"$in"` if the comparison value is a list) is used as default
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -599,6 +602,7 @@ def get_all_documents_generator(
operator is provided, `"$eq"` (or `"$in"` if the comparison value is a list) is used as default
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -926,6 +930,7 @@ def delete_documents(
operator is provided, `"$eq"` (or `"$in"` if the comparison value is a list) is used as default
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -1029,6 +1034,7 @@ def query_by_embedding(
operator is provided, `"$eq"` (or `"$in"` if the comparison value is a list) is used as default
operation.
__Example__:
```python
filters = {
"$and": {
Expand All @@ -1055,6 +1061,7 @@ def query_by_embedding(
To use the same logical operator multiple times on the same level, logical operators take
optionally a list of dictionaries as value.
__Example__:
```python
filters = {
"$or": [
Expand Down
91 changes: 51 additions & 40 deletions haystack/document_stores/search_engine.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,7 @@ def get_metadata_values_by_key(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -572,6 +573,7 @@ def get_all_documents(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -623,6 +625,7 @@ def get_all_documents_generator(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -728,6 +731,7 @@ def query(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -756,6 +760,7 @@ def query(
optionally a list of dictionaries as value.
__Example__:
```python
filters = {
"$or": [
Expand Down Expand Up @@ -787,29 +792,29 @@ def query(
::
**An example custom_query:**
```python
| {
| "size": 10,
| "query": {
| "bool": {
| "should": [{"multi_match": {
| "query": ${query}, // mandatory query placeholder
| "type": "most_fields",
| "fields": ["content", "title"]}}],
| "filter": [ // optional custom filters
| {"terms": {"year": ${years}}},
| {"terms": {"quarter": ${quarters}}},
| {"range": {"date": {"gte": ${date}}}}
| ],
| }
| },
| }
```python
{
"size": 10,
"query": {
"bool": {
"should": [{"multi_match": {
"query": ${query}, // mandatory query placeholder
"type": "most_fields",
"fields": ["content", "title"]}}],
"filter": [ // optional custom filters
{"terms": {"year": ${years}}},
{"terms": {"quarter": ${quarters}}},
{"range": {"date": {"gte": ${date}}}}
],
}
},
}
```
**For this custom_query, a sample retrieve() could be:**
```python
| self.retrieve(query="Why did the revenue increase?",
| filters={"years": ["2019"], "quarters": ["Q1", "Q2"]})
self.retrieve(query="Why did the revenue increase?",
filters={"years": ["2019"], "quarters": ["Q1", "Q2"]})
```
Optionally, highlighting can be defined by specifying the highlight settings.
Expand All @@ -818,31 +823,31 @@ def query(
::
**Example custom_query with highlighting:**
```python
| {
| "size": 10,
| "query": {
| "bool": {
| "should": [{"multi_match": {
| "query": ${query}, // mandatory query placeholder
| "type": "most_fields",
| "fields": ["content", "title"]}}],
| }
| },
| "highlight": { // enable highlighting
| "fields": { // for fields content and title
| "content": {},
| "title": {}
| }
| },
| }
```python
{
"size": 10,
"query": {
"bool": {
"should": [{"multi_match": {
"query": ${query}, // mandatory query placeholder
"type": "most_fields",
"fields": ["content", "title"]}}],
}
},
"highlight": { // enable highlighting
"fields": { // for fields content and title
"content": {},
"title": {}
}
},
}
```
**For this custom_query, highlighting info can be accessed by:**
```python
| docs = self.retrieve(query="Why did the revenue increase?")
| highlighted_content = docs[0].meta["highlighted"]["content"]
| highlighted_title = docs[0].meta["highlighted"]["title"]
docs = self.retrieve(query="Why did the revenue increase?")
highlighted_content = docs[0].meta["highlighted"]["content"]
highlighted_title = docs[0].meta["highlighted"]["title"]
```
:param index: The name of the index in the DocumentStore from which to retrieve documents
Expand Down Expand Up @@ -915,6 +920,7 @@ def query_batch(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -943,6 +949,7 @@ def query_batch(
optionally a list of dictionaries as value.
__Example__:
```python
filters = {
"$or": [
Expand Down Expand Up @@ -1166,6 +1173,7 @@ def update_embeddings(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -1270,6 +1278,7 @@ def delete_all_documents(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -1320,6 +1329,7 @@ def delete_documents(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down Expand Up @@ -1383,6 +1393,7 @@ def delete_labels(
operation.
__Example__:
```python
filters = {
"$and": {
Expand Down
Loading

0 comments on commit 4a83b20

Please sign in to comment.