Skip to main content

OCR for more models

Feature Request

Currently, interacting with text-based LLMs requires pre-processing of visual data (images, PDFs) to extract the text. Implementing state-of-the-art OCR models directly into the chat interface would eliminate this manual step. Users could upload images or PDFs, and the system would automatically extract the text, making it available for the LLM. A key enhancement is the ability to recognise and format tabular data within these documents into Markdown tables.

Benefits:

  • Improved User Experience: Simplifies the process of using visual data with text-based LLMs.

  • Increased Efficiency: Reduces the time and effort required for text extraction.

  • Markdown Table Formatting: Automatically converts tabular data into well-formatted Markdown tables for easy readability and integration.

  • Competitive Advantage: Positions the platform as a leader in integrating visual and textual data processing.

Status: Completed5 comments

Log in to comment and vote

Comments5

  • Moccasin Croissant

    I came here to suggest adding deepseek OCR, and found this feature request. Two birds with one stone? I would love to test deepseek OCR.

    • Mark Florkowski

      Team

      Just waiting for good hosting options, it does seem like a very promising model though

  • Mark Florkowski

    Team

    This is already available on Gemini and Claude models. We are working on bringing capabilities to other models too

    • Rose Milkyway

      Sorry, I shoud’ve been clearer in my post. This is a feature that would enhance the LLMs that dont have vision capability enabling text only llms to have access to text in uploaded images and pdfs. Integrating a state of the art OCR llm as a preprocessor for the text llm would in my opinion be a killer feature 😁.

  • Rose Milkyway

    Check out

    https://mistral.ai/news/mistral-ocr (good performance and speed)

    https://olmocr.allenai.org/ (open source)