Redesign primitives

[tholor]

Proposed changes:

• Addressing Re-design primitives #1232

Tasks:

• Move Answer to schema
• Review Answer fields
• Rename "question" to "query" everywhere
• Review Label + Multilabel
• Re-use labels in API instead of ExtractiveQAFeedback
• Decide if to go for plain class, dataclass, BaseModel or TypedDict
• Review Document
• Fix REST API serialization
• Adjust feedback endpoint in REST API
• Make behavior of write_labels / indices consistent across doc stores
• Update Game-of-thrones example docker image to include new doc structure

Status (please check what you already did):

• First draft (up for discussions & feedback)
• Final code
• Added tests
• Updated documentation

Breaking changes

There are many and we should probably do a "migration guide" when we release 1.0
Here are the central ones:

Document

• Renamed Document.text -> Document.content
• Removed Document.question (was only used a while ago in FAQ search cases)
• Rename text_field -> content_field in ElasticsearchDocumentStore & Weaviate init
• Remove faq_question_field in ElasticsearchDocumentStore & Weaviate init

Label

• Rename Label.question -> Label.query
• Make Label.answer an Answer obj rather than plain str
• Remove Label.document_id (can now be accessed via Label.document.id)
• Rename Label.model_id -> Label.pipeline_id
• Remove Label.offset_start_in_doc (can now be accessed via label.answer.offsets_in_document[0].start`

Answer

The reader returns now an Answer object rather than a dict.
It follows this new structure:

Particularly the handling of offsets has changed to be more explicit and allow for other multiple spans (e.g. TableQA):

# Old -> New 

answer["offset_start_in_doc"] -> answer.offsets_in_document[0].start
answer["offset_end_in_doc"] -> answer.offsets_in_document[0].end

answer["offset_start"] -> answer.offset_in_context[0].start
answer["offset_end"] -> answer.offset_in_context[0].end

Future work

• Fix autocomplete in IDE for Answer primitive #1582 Fix autocomplete for Answer (dataclasses-json seems to break it -> probably switch to manual serialization)
• Fix serialization of pd dataframe for REST API
• Let Preprocessor return Documents instead of Dicts
• Evaluate if we want to add File
• Evaluate if we want to add Query
• Return Documents instead of Dicts from PreProcessor
• Refactor RAGenerator to use Answer
• Check if want to use relations between Label and Document in ORM (see Evaluate logic of index to group Labels and Documents more closely #1563)

[tholor]

@lalitpagaria Thanks for your comments! This is still a super early draft. We will need more discussions and iterations in the next week to achieve a nice design. It's one of the few remaining things we want to get straight before the 1.0 release :)

[lalitpagaria]

Oh great! so Haystack 1.0 is around 🎉

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[ZanSara]

I focused a bit more on small things this time because the diff is huge! I found nothing that I believe might be a bug, all comments are on details, style, and such. BTW on many of them I can help fixing if you need