https://github.com/adithya-s-k/omniparse/assets/27956426/457d8b5b-9573-44da-8bcf-616000651a13 aaa

✅ Completely local, no external APIs
✅ Fits in a T4 GPU
✅ Supports ~20 file types
✅ Convert documents, multimedia, and web pages to high-quality structured markdown
✅ Table extraction, image extraction/captioning, audio/video transcription, web page crawling
✅ Easily deployable using Docker and Skypilot
✅ Colab friendly
✅ Interative UI powered by Gradio

It's challenging to process data as it comes in different shapes and sizes. OmniParse aims to be an ingestion/parsing platform where you can ingest any type of data, such as documents, images, audio, video, and web content, and get the most structured and actionable output that is GenAI (LLM) friendly.

git clone https://github.com/adithya-s-k/omniparse
cd omniparse

Create a Virtual Environment:

conda create -n omniparse-venv python=3.10
conda activate omniparse-venv

Install Dependencies:

uv sync
# or
pip install -e .
# or
pip install -r pyproject.toml

If using uv, activate the venv created by uv sync.

To use OmniParse with Docker, execute the following commands:

1. Pull the OmniParse API Docker image from Docker Hub:
2. Run the Docker container, exposing port 8000: 👉🏼Docker Image

docker pull savatar101/omniparse:0.1
# if you are running on a gpu 
docker run --gpus all -p 8000:8000 savatar101/omniparse:0.1
# else
docker run -p 8000:8000 savatar101/omniparse:0.1

Alternatively, if you prefer to build the Docker image locally: Then, run the Docker container as follows:

docker build -t omniparse .
# if you are running on a gpu
docker run --gpus all -p 8000:8000 omniparse
# else
docker run -p 8000:8000 omniparse

Run the Server:

python server.py --host 0.0.0.0 --port 8000 --documents --media --web

• --documents: Load in all the models that help you parse and ingest documents (Docling models and Florence-2).
• --media: Load in Whisper model to transcribe audio and video files.
• --web: Set up selenium crawler.

Download Models: If you want to download the models before starting the server

python download.py --documents --media --web

• --documents: Load in all the models that help you parse and ingest documents (Docling models and Florence-2).
• --media: Load in Whisper model to transcribe audio and video files.
• --web: Set up selenium crawler.

curl -X POST -F "file=@/path/to/document" http://localhost:8000/parse_document

curl -X POST -F "file=@/path/to/document.pdf" http://localhost:8000/parse_document/pdf

curl -X POST -F "file=@/path/to/presentation.ppt" http://localhost:8000/parse_document/ppt

curl -X POST -F "file=@/path/to/document.docx" http://localhost:8000/parse_document/docs

curl -X POST -F "file=@/path/to/image.jpg" http://localhost:8000/parse_media/image

curl -X POST -F "image=@/path/to/image.jpg" -F "task=Caption" -F "prompt=Optional prompt" http://localhost:8000/parse_media/process_image

curl -X POST -F "file=@/path/to/video.mp4" http://localhost:8000/parse_media/video

curl -X POST -F "file=@/path/to/audio.mp3" http://localhost:8000/parse_media/audio

curl -X POST -H "Content-Type: application/json" -d '{"url": "https://example.com"}' http://localhost:8000/parse_website

🦙 LlamaIndex | Langchain | Haystack integrations coming soon 📚 Batch processing data ⭐ Dynamic chunking and structured data extraction based on specified Schema
🛠️ One magic API: just feed in your file prompt what you want, and we will take care of the rest
🔧 Dynamic model selection and support for external APIs
📄 Batch processing for handling multiple files at once

Final goal: replace all the different models currently being used with a single MultiModel Model to parse any type of data and get the data you need.

There is a need for a GPU with 8~10 GB minimum VRAM as we are using deep learning models. \

Document Parsing Limitations \

• Docling is the underlying PDF parser and its limitations will apply.
• It is good at parsing english but might struggle for languages such as Chinese
• Tables are not always formatted 100% correctly; text can be in the wrong column.
• Whitespace and indentations are not always respected.
• Not all lines/spans will be joined properly.
• This works best on digital PDFs that won't require a lot of OCR. It's optimized for speed, and limited OCR is used to fix errors.
• To fit all the models in the GPU, we are using the smallest variants, which might not offer the best-in-class performance.

OmniParse is licensed under the GPL-3.0 license. See LICENSE for more information.

This project builds upon the remarkable Docling project. We express our gratitude for the inspiration and foundation provided by this project. Special thanks to Crawl4AI for their contributions.

Models being used:

• Docling IBM models
• Florence-2 base
• Whisper Small

Thank you to the authors for their contributions to these models.