Improve document conversion

[masci]

File conversion and data extraction has proven problematic lately: we have several converters and parsers in Haystack but it’s not always clear which one (or which combination) to use; sometimes one tool is good for a task but not for another; special characters and text layout can easily make document ingestion fail.

Desired outcome

• Fix the pain points with PDF layouts: special chars, multi column and headlines
• Test on a larger dataset of examples, this will also ease comparison if we want to introduce OCR later
• Conversion and preprocessing often fail silently, review the code and find where adding a Warning message could help users

Related issues:

• Extract headings from Markdown files #3056
• Make use of Parsr's heading detection #3057
• Add page information to Document metadata when converting PDF files #2374
• https://github.com/deepset-ai/haystack-private/issues/6
• feat: Add page number to Documents coming from PDFConverters and PreProcessor #2932
• Enable PreProcessor to split files by custom regex (e.g. for markdown headlines) #2583
• Incorporate LayoutLM for information extraction from PDFs #1404
• Assess usage of LayoutLM for extracting structural elements of PDFs #3058

---

Process

• Assess LayoutLM's capabilities and whether it makes sense to integrate it
• Write a guide about when and how to use the Document converters
  • https://docs.haystack.deepset.ai/docs/file_converters#choosing-the-right-pdf-converter
• Identify and solve pain points during preprocessing
  • feat: split PreProcessor #3557
  • feat: preprocessor raises warning when doc length exceeds threshold #3837
• [OPTIONAL] Implement CSVConverter
  • feat: Add CsvTextConverter #3587