DocumentLens: Rich Document Parsing with LLMs

A powerful, LLM-based tool for extracting structured data from rich documents (PDFs) with Llama models.

Overview

This tool uses Llama models to extract text, tables, images, and charts from PDFs, converting unstructured document data into structured, machine-readable formats. It supports:

Text extraction: Extract and structure main text, titles, captions, etc.
Table extraction: Convert complex tables into structured data formats
Image extraction: Extract images with contextual descriptions and captions
Chart extraction: Convert charts and graphs into structured JSON data
Multiple output formats: JSON, CSV, Excel, and SQL database storage
Vector search capabilities: Semantic search across extracted content

The tool is designed to handle complex documents with high accuracy and provides flexible configuration options to tailor extraction tasks to specific needs.

Project Structure

structured_parser/
├── src/
│   ├── structured_extraction.py  # Main entry point and extraction logic
│   ├── utils.py                  # Utility functions and classes
│   ├── typedicts.py             # Type definitions
│   ├── json_to_table.py         # Database integration functions
│   └── config.yaml              # Configuration file
├── pdfs/                        # Sample PDFs and extraction results
├── README.md                    # This file
├── CONTRIBUTING.md              # Contribution guidelines
└── requirements.txt             # Python dependencies

Installation

Prerequisites

Python 3.9+
[Optional] Local GPU for offline inference

Setup

Clone the repository

git clone https://github.com/meta-llama/llama-cookbook.git
cd llama-cookbook

Install project specific dependencies: bash cd end-to-end-use-cases/structured_parser pip install -r requirements.txt

Configure the tool (see Configuration section)

(Note: Setup API Key, Model for inferencing, etc.)

Extract text from a PDF:

python src/structured_extraction.py path/to/document.pdf text

Extract charts and tables, and save them as CSV files:

python src/structured_extraction.py path/to/document.pdf charts,tables --save_tables_as_csv

Process a directory of PDFs and export tables to Excel:

python src/structured_extraction.py path/to/pdf_directory text,tables --export_excel

Extract all artifact types and save to database and as Excel sheets:

python src/structured_extraction.py path/to/document.pdf text,tables,images,charts --save_to_db --export_excel

Configuration

The tool is configured via src/config.yaml. Key configuration options include:

Model Configuration

model:
  backend: openai-compat  # [offline-vllm, openai-compat]

  # For openai-compat
  base_url: "https://api.llama.com/compat/v1"
  api_key: "YOUR_API_KEY"
  model_id: "Llama-4-Maverick-17B-128E-Instruct-FP8"

  # For offline-vllm
  path: "/path/to/checkpoint"
  tensor_parallel_size: 4
  max_model_len: 32000
  max_num_seqs: 32

Inference Parameters

extraction_inference:
  temperature: 0.2
  top_p: 0.9
  max_completion_tokens: 32000
  seed: 42

Database Configuration

database:
  sql_db_path: "sqlite3.db"
  vector_db_path: "chroma.db"

Artifact Configuration

The tool includes configurable prompts and output schemas for each artifact type (text, tables, images, charts). These can be modified in the config.yaml file to customize extraction behavior for specific document types.

Output Formats

JSON Output

The primary output format includes all extracted artifacts in a structured JSON format with timestamps.

CSV Export

Tables and charts can be exported as individual CSV files for easy analysis in spreadsheet applications.

Excel Export

Multiple tables can be combined into a single Excel workbook with separate tabs for each table.

Database Storage

Extracted data can be stored in SQLite databases with optional vector indexing for semantic search.

API Usage

Programmatic Usage

from src.structured_extraction import ArtifactExtractor
from src.utils import PDFUtils

# Extract pages from PDF
pages = PDFUtils.extract_pages("document.pdf")

# Process specific pages
for page in pages[10:20]:  # Process pages 10-19
    artifacts = ArtifactExtractor.from_image(
        page["image_path"],
        ["text", "tables"]
    )
    # Custom processing of artifacts...

Single Image Processing

from src.structured_extraction import ArtifactExtractor

# Extract from a single image
artifacts = ArtifactExtractor.from_image(
    "path/to/image.png",
    ["text", "tables", "images"]
)

Architecture

Core Components

RequestBuilder: Builds inference requests for LLMs with image and text content
ArtifactExtractor: Extracts structured data from documents using configurable prompts
PDFUtils: Handles PDF processing and page extraction as images
InferenceUtils: Manages LLM inference with support for VLLM and OpenAI-compatible APIs
JSONUtils: Handles JSON extraction and validation from LLM responses
ImageUtils: Utility functions for image encoding and processing

Data Flow

PDFs are converted to images (one per page) using PyMuPDF
Images are processed by the LLM to extract structured data based on configured prompts
Structured data is saved in various formats (JSON, CSV, SQL, etc.)
Optional vector indexing for semantic search capabilities

Supported Artifact Types

text: Main text content, titles, captions, and other textual elements
tables: Structured tabular data with proper formatting
images: Image descriptions, captions, and metadata
charts: Chart data extraction with structured format including axes, data points, and metadata

Extending the Tool

Adding New Artifact Types

Add a new artifact type configuration in config.yaml:

artifacts:
  my_new_artifact:
    prompts:
      system: "Your system prompt here..."
      user: "Your user prompt with {schema} placeholder..."
    output_schema: {
      # Your JSON schema here
    }
    use_json_decoding: true

Customizing Extraction Logic

The extraction logic is modular and can be customized by:

Modifying prompts in the config.yaml file
Adjusting output schemas to capture different data structures
Extending the ArtifactExtractor class for specialized extraction needs

Using Different Models

The tool supports two backends:

openai-compat: Any API compatible with the OpenAI API format (including Llama API)
offline-vllm: Local inference using VLLM for self-hosted deployments

Best Practices

Model Selection: Use larger models for complex documents or when high accuracy is required
Prompt Engineering: Adjust prompts in config.yaml for your specific document types
Output Schema: Define precise schemas to guide the model's extraction process

Troubleshooting

Common Issues

Model capacity errors: Reduce max tokens or use a larger model
Extraction quality issues: Adjust prompts or output schemas
Configuration errors: Verify model paths and API credentials in config.yaml

README.md 7.3 KB History Raw