Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add user feedback buttons to UI to collect domain-specifc training data #864

Closed
tholor opened this issue Feb 26, 2021 · 1 comment
Closed
Assignees
Labels
good first issue Good for newcomers stale type:feature New feature or request

Comments

@tholor
Copy link
Member

tholor commented Feb 26, 2021

Is your feature request related to a problem? Please describe.
If you work on domain corpora, collecting additional training data to improve your reader or retriever models is very helpful.
For collecting training data, there are two main options:
a) Manual Labelling
b) User feedback for "live predictions"

b) is particularly promising if you don't have enough time or resources for a) OR if you want to continuously improve your model once it is production.

We already have the API for storing user feedback and used it successfully in some deployments. In these deployments, we used proprietary user interfaces to collect feedback. Let's add those feedback buttons also to our Haystack Demo UI based on streamlit so that everybody can experiment with it more easily and understand the mechanics.

Describe the solution you'd like

  • Add 👍 / 👎 buttons in the streamlit UI that allow capturing user feedback per result
  • Optionally: When clicking 👎 we could distinguish between "wrong answer and wrong passage" and "wrong answer, but correct passage" (this distinction helps for the training of retriever models)

image

Additional context

@stale
Copy link

stale bot commented Jul 22, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 21 days if no further activity occurs.

@stale stale bot added the stale label Jul 22, 2021
@stale stale bot closed this as completed Aug 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers stale type:feature New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants