How to Build a JSON Training Dataset from PDF Documents Without Manual Annotation

Building quality training datasets is one of the most time-consuming parts of any machine learning project. For most teams, that bottleneck isn’t compute or model architecture, it’s data. More specifically, it’s the hours spent manually annotating documents before you can even start training.

PDFs are everywhere in the enterprise. Legal contracts, research papers, technical manuals, financial reports, product documentation, they contain exactly the kind of domain-specific knowledge that makes fine-tuned models valuable. The problem is that turning those PDFs into structured, ready-to-train JSON datasets has traditionally required either expensive human annotation or a lot of brittle custom scripting.

In this article, you’ll build a complete, production-ready pipeline that extracts text from PDFs, generates structured instruction-response pairs using an LLM, validates them programmatically, and outputs a clean JSON dataset — with zero manual labeling. The approach uses several different dataset formats depending on your target training framework, and we’ll compare their trade-offs directly.

By the end, you’ll have a pipeline that can turn a folder of PDFs into thousands of quality training examples overnight.

Prerequisites: Python 3.10+, familiarity with basic NLP concepts, and a working understanding of what fine-tuning is for. If you need a primer on the latter, SitePoint’s guide to fine-tuning local LLMs covers the full workflow from dataset prep through training and export.

Why Building Training Datasets from PDFs Is Harder Than It Looks

Before writing a single line of code, it’s worth understanding why PDFs create problems that other document formats don’t.

A PDF is not a document with structure; it’s a rendering instruction set. Each character has an exact pixel position on a page. There are no paragraph breaks, no semantic headings, no notion of which text block is a caption versus a title versus body copy. The same sentence on two different pages might be stored in completely different byte orders internally, depending on the software that created the document.

This creates several specific problems:

• Reading order corruption: Multi-column layouts frequently produce merged columns or reversed lines
• Table mangling: Tabular data is stored as floating text positions, so naive extraction produces garbage
• Header/footer bleed: Page numbers and headers get injected into paragraph middles
• Ligature and hyphenation issues: Common in academic PDFs, causing words to split incorrectly
• Scanned PDFs: Flat images with no extractable text at all, requiring OCR

You’ll see how to handle each of these. But first, let’s set up the environment.

Setting Up the Environment

bash

pip install pymupdf pdfplumber unstructured[pdf] \
    tiktoken openai langchain-text-splitters \
    jsonschema datasets tqdm

Note: The unstructured[pdf] install pulls in poppler-utils and tesseract as optional OCR dependencies. If you’re on macOS: brew install poppler tesseract. On Ubuntu: sudo apt-get install poppler-utils tesseract-ocr.

Not all PDF extractors are equal. The “right” one depends on your document type. The table below summarizes how the four main options compare across the dimensions that matter most for dataset building:

Benchmarks from community testing on a standard mixed-layout PDF. Your results will vary by document complexity.

For dataset building from enterprise PDFs, PyMuPDF is the best default choice. It outputs clean Markdown via pymupdf4llm, which preserves heading hierarchy and handles tables reasonably well. Use pdfplumber when your documents are heavily tabular (financial reports, spreadsheets exported to PDF). Use unstructured when you’re processing scanned documents or need proper semantic chunking with element classification out of the box.

The pipeline we’ll build uses PyMuPDF as the primary extractor, with a fallback to pdfplumber for table-dense pages.

Step 1: Extract and Clean Text from PDFs

Here’s the extraction module. It handles encoding issues, removes common PDF artifacts, and outputs clean Markdown:

python

# extractor.py
import fitz  # PyMuPDF
import pymupdf4llm
import pdfplumber
import re
from pathlib import Path
from dataclasses import dataclass
from typing import Optional

@dataclass
class ExtractedDocument:
    source: str
    pages: list[str]
    total_pages: int
    has_tables: bool
    method_used: str

def clean_text(text: str) -> str:
    """Remove common PDF extraction artifacts."""
    # Fix hyphenated line breaks
    text = re.sub(r'(\w+)-\n(\w+)', r'\1\2', text)
    # Normalize whitespace
    text = re.sub(r'\n{3,}', '\n\n', text)
    # Remove page numbers (common patterns)
    text = re.sub(r'\n\s*\d+\s*\n', '\n', text)
    # Remove headers/footers (lines under 6 words that repeat)
    lines = text.split('\n')
    cleaned = [l for l in lines if len(l.split()) > 5 or l.strip() == '']
    return '\n'.join(cleaned)

def detect_table_density(pdf_path: str) -> bool:
    """Check if document has significant tabular content."""
    with pdfplumber.open(pdf_path) as pdf:
        tables_found = sum(
            len(page.extract_tables()) for page in pdf.pages[:5]
        )
    return tables_found > 2

def extract_with_pymupdf(pdf_path: str) -> list[str]:
    """Extract using PyMuPDF with LLM-optimized Markdown output."""
    md_text = pymupdf4llm.to_markdown(pdf_path, page_chunks=True)
    pages = []
    for chunk in md_text:
        cleaned = clean_text(chunk['text'])
        if len(cleaned.strip()) > 100:  # Skip near-empty pages
            pages.append(cleaned)
    return pages

def extract_with_pdfplumber(pdf_path: str) -> list[str]:
    """Extract using pdfplumber for table-heavy documents."""
    pages = []
    with pdfplumber.open(pdf_path) as pdf:
        for page in pdf.pages:
            text_parts = []
            # Extract text with layout preservation
            text = page.extract_text(x_tolerance=2, y_tolerance=3)
            if text:
                text_parts.append(clean_text(text))
            # Extract tables separately as markdown
            for table in page.extract_tables():
                if table:
                    # Convert table to markdown format
                    rows = []
                    for i, row in enumerate(table):
                        row_clean = [str(c or '').strip() for c in row]
                        rows.append('| ' + ' | '.join(row_clean) + ' |')
                        if i == 0:
                            rows.append('|' + '---|' * len(row))
                    text_parts.append('\n'.join(rows))
            
            combined = '\n\n'.join(text_parts)
            if len(combined.strip()) > 100:
                pages.append(combined)
    return pages

def extract_document(pdf_path: str) -> ExtractedDocument:
    """Auto-select extraction method based on document characteristics."""
    path = Path(pdf_path)
    has_tables = detect_table_density(pdf_path)
    
    if has_tables:
        pages = extract_with_pdfplumber(pdf_path)
        method = 'pdfplumber'
    else:
        pages = extract_with_pymupdf(pdf_path)
        method = 'pymupdf4llm'
    
    return ExtractedDocument(
        source=path.name,
        pages=pages,
        total_pages=len(pages),
        has_tables=has_tables,
        method_used=method
    )

Step 2: Chunking Strategy; Why Naive Splitting Fails

Once you have clean text, the instinct is to split it by character count and move on. This is a mistake.

Naive character splitting breaks sentences mid-thought. An LLM asked to generate a question about “…the capital of France is Pa” will hallucinate. For training data specifically, chunk quality has a multiplicative effect on final model quality.

The right strategy depends on what you’re building:

For instruction tuning — which is what most fine-tuning projects need, recursive character splitting with a sliding window gives the best coverage-to-quality ratio. Here’s a token-aware implementation:

python

# chunker.py
from langchain_text_splitters import RecursiveCharacterTextSplitter
import tiktoken
from typing import Generator

class SmartChunker:
    def __init__(
        self,
        chunk_size: int = 800,
        chunk_overlap: int = 100,
        model: str = "gpt-4o-mini"
    ):
        self.encoder = tiktoken.encoding_for_model(model)
        self.splitter = RecursiveCharacterTextSplitter(
            chunk_size=chunk_size,
            chunk_overlap=chunk_overlap,
            length_function=self._token_count,
            separators=["\n\n", "\n", ". ", "! ", "? ", " ", ""]
        )
    
    def _token_count(self, text: str) -> int:
        return len(self.encoder.encode(text))
    
    def chunk_document(
        self, 
        pages: list[str], 
        source: str
    ) -> Generator[dict, None, None]:
        """Yield chunks with metadata."""
        for page_num, page_text in enumerate(pages):
            chunks = self.splitter.split_text(page_text)
            for chunk_idx, chunk in enumerate(chunks):
                # Skip low-quality chunks
                token_count = self._token_count(chunk)
                if token_count < 40:
                    continue
                # Skip chunks that look like extracted headers only
                lines = [l.strip() for l in chunk.split('\n') if l.strip()]
                if len(lines) <= 2 and all(len(l) < 80 for l in lines):
                    continue
                    
                yield {
                    "text": chunk.strip(),
                    "source": source,
                    "page": page_num + 1,
                    "chunk_index": chunk_idx,
                    "token_count": token_count
                }

Step 3: Designing Your JSON Schema

Before generating any examples, you need to decide which JSON format matches your training framework. The three most widely used formats are:

Alpaca Format

Used by most instruction-tuning setups. Simple and widely supported:

json

{
  "instruction": "Explain the key provisions of the Basel III capital requirements.",
  "input": "",
  "output": "Basel III introduced three main capital requirements..."
}

ShareGPT / Conversation Format

Used by Axolotl, LLaMA-Factory, and most chat model fine-tuning. Better for multi-turn tasks:

json

{
  "conversations": [
    {"from": "human", "value": "What is the definition of Tier 1 capital?"},
    {"from": "gpt", "value": "Tier 1 capital refers to the core capital..."}
  ]
}

OpenAI ChatML Format

Used by the Hugging Face trl library and OpenAI fine-tuning API. Most portable:

json

{
  "messages": [
    {"role": "system", "content": "You are a financial regulatory expert."},
    {"role": "user", "content": "Summarize the leverage ratio requirement."},
    {"role": "assistant", "content": "The Basel III leverage ratio requires..."}
  ]
}

For most projects starting from scratch, start with ChatML, it’s directly compatible with the Hugging Face fine-tuning ecosystem and the OpenAI fine-tuning API, giving you the most flexibility without reformatting.

Step 4: Automated QA Generation Using an LLM

This is the core of the pipeline. Instead of having humans write question-answer pairs, you use a capable LLM to read each chunk and generate training examples from it.

The key insight from recent research (including the AWS ML Blog guide on LLM dataset preparation) is that LLM-generated QA pairs can match or exceed human annotation quality when the prompts are carefully structured and the outputs are validated. In fact, one 2025 study found that Mistral-7b fine-tuned on LLM-generated QA pairs outperformed the same model trained on human-annotated data on BERT F1, BLEU, and ROUGE scores.

The trick is prompt engineering. Vague prompts produce vague training data:

python

# generator.py
from openai import OpenAI
import json
from typing import Optional

client = OpenAI()

SYSTEM_PROMPT = """You are a dataset generation specialist. Your task is to create 
high-quality instruction-following training examples from document chunks.

For each chunk, generate between 2 and 4 diverse question-answer pairs that:
1. Ask about specific facts, definitions, or concepts in the text
2. Vary in question type (factual, explanatory, comparative, or applied)
3. Have answers that are fully contained in the provided text
4. Avoid yes/no questions

Return ONLY valid JSON. No markdown fences, no explanation.

Output format:
[
  {
    "messages": [
      {"role": "system", "content": ""},
      {"role": "user", "content": ""},
      {"role": "assistant", "content": ""}
    ]
  }
]"""

def generate_qa_pairs(
    chunk: dict,
    domain_context: str = "a domain expert",
    model: str = "gpt-4o-mini",
    temperature: float = 0.7
) -> Optional[list[dict]]:
    """Generate QA training pairs from a text chunk."""
    
    user_message = f"""Generate training examples from this document excerpt.

Source: {chunk['source']} (page {chunk['page']})
Domain context: {domain_context}

DOCUMENT EXCERPT:
{chunk['text']}

Generate 2-4 diverse question-answer pairs as ChatML JSON."""

    try:
        response = client.chat.completions.create(
            model=model,
            messages=[
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": user_message}
            ],
            temperature=temperature,
            response_format={"type": "json_object"},
            max_tokens=1500
        )
        
        raw = response.choices[0].message.content
        parsed = json.loads(raw)
        
        # Handle both list and dict with list key
        if isinstance(parsed, list):
            return parsed
        for key in parsed:
            if isinstance(parsed[key], list):
                return parsed[key]
        return None
        
    except (json.JSONDecodeError, KeyError, IndexError) as e:
        # Log and skip rather than crash
        print(f"Generation failed for chunk {chunk['chunk_index']}: {e}")
        return None

Step 5: Validation Without Human Review

The biggest risk in automated dataset generation is hallucination, the LLM generating answers that aren’t in the source document. You need programmatic checks that catch the most common failure modes before anything goes into your training file.

python

# validator.py
import re
from difflib import SequenceMatcher

def normalize(text: str) -> str:
    """Normalize text for comparison."""
    return re.sub(r'\s+', ' ', text.lower().strip())

def overlap_score(answer: str, source_chunk: str) -> float:
    """
    Measure how much of the answer is grounded in the source.
    Uses longest common subsequence ratio.
    """
    return SequenceMatcher(
        None,
        normalize(answer),
        normalize(source_chunk)
    ).ratio()

def validate_example(example: dict, source_chunk: str) -> tuple[bool, str]:
    """
    Validate a single training example.
    Returns (is_valid, rejection_reason)
    """
    messages = example.get('messages', [])
    
    # Check structure
    if len(messages) < 2:
        return False, "fewer than 2 messages"
    
    roles = [m.get('role') for m in messages]
    if 'user' not in roles or 'assistant' not in roles:
        return False, "missing user or assistant role"
    
    user_msg = next((m['content'] for m in messages if m['role'] == 'user'), '')
    assistant_msg = next((m['content'] for m in messages if m['role'] == 'assistant'), '')
    
    # Length checks
    if len(user_msg.split()) < 4:
        return False, "question too short"
    if len(assistant_msg.split()) < 10:
        return False, "answer too short"
    if len(assistant_msg.split()) > 600:
        return False, "answer too long (possible hallucination)"
    
    # Yes/no filter
    stripped = assistant_msg.strip().lower()
    if stripped.startswith(('yes.', 'no.', 'yes,', 'no,')):
        return False, "yes/no answer"
    
    # Grounding check — answer must be substantially traceable to source
    score = overlap_score(assistant_msg[:300], source_chunk)
    if score < 0.15:
        return False, f"low grounding score ({score:.2f}) — possible hallucination"
    
    return True, "valid"

def validate_batch(
    examples: list[dict],
    source_chunk: str,
    min_valid_ratio: float = 0.5
) -> list[dict]:
    """Filter and return only valid examples from a batch."""
    valid = []
    stats = {"valid": 0, "rejected": 0, "reasons": {}}
    
    for ex in examples:
        is_valid, reason = validate_example(ex, source_chunk)
        if is_valid:
            valid.append(ex)
            stats["valid"] += 1
        else:
            stats["rejected"] += 1
            stats["reasons"][reason] = stats["reasons"].get(reason, 0) + 1
    
    return valid, stats

Step 6: Deduplication and Final Cleaning

Even with good chunking, neighboring chunks will share similar context, leading to near-duplicate training examples. A simple cosine similarity check on question embeddings catches most of this:

python

# deduplicator.py
from openai import OpenAI
import numpy as np

client = OpenAI()

def get_embeddings(texts: list[str], model: str = "text-embedding-3-small") -> list[list[float]]:
    """Batch-embed a list of texts."""
    response = client.embeddings.create(input=texts, model=model)
    return [r.embedding for r in response.data]

def cosine_similarity(a: list[float], b: list[float]) -> float:
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def deduplicate_dataset(
    examples: list[dict],
    similarity_threshold: float = 0.92
) -> list[dict]:
    """
    Remove near-duplicate examples using question embedding similarity.
    Threshold of 0.92 catches paraphrased duplicates while keeping
    legitimately similar but distinct examples.
    """
    # Extract questions for embedding
    questions = []
    for ex in examples:
        q = next((m['content'] for m in ex['messages'] if m['role'] == 'user'), '')
        questions.append(q)
    
    # Batch embed (chunk to avoid API limits)
    batch_size = 100
    all_embeddings = []
    for i in range(0, len(questions), batch_size):
        batch = questions[i:i+batch_size]
        all_embeddings.extend(get_embeddings(batch))
    
    # Greedy deduplication
    keep = [0]  # Always keep first
    for i in range(1, len(all_embeddings)):
        is_duplicate = False
        for j in keep:
            if cosine_similarity(all_embeddings[i], all_embeddings[j]) > similarity_threshold:
                is_duplicate = True
                break
        if not is_duplicate:
            keep.append(i)
    
    return [examples[i] for i in keep]

Step 7: The Complete Pipeline

With all modules in place, here’s the orchestration layer that ties everything together:

python

# pipeline.py
import json
from pathlib import Path
from tqdm import tqdm
from extractor import extract_document
from chunker import SmartChunker
from generator import generate_qa_pairs
from validator import validate_batch
from deduplicator import deduplicate_dataset

def build_dataset(
    pdf_dir: str,
    output_path: str,
    domain_context: str = "a domain expert",
    chunk_size: int = 800,
    chunk_overlap: int = 100,
    model: str = "gpt-4o-mini",
    deduplicate: bool = True
) -> dict:
    """
    Full pipeline: PDF folder -> JSON training dataset.
    
    Args:
        pdf_dir: Directory containing PDF files
        output_path: Where to write the .jsonl output file
        domain_context: Describes the domain for system prompt generation
        chunk_size: Token size per chunk
        chunk_overlap: Overlap tokens between chunks
        model: OpenAI model for generation
        deduplicate: Whether to run deduplication pass
    
    Returns:
        Pipeline statistics dictionary
    """
    pdf_files = list(Path(pdf_dir).glob("*.pdf"))
    print(f"Found {len(pdf_files)} PDF files")
    
    chunker = SmartChunker(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
    all_examples = []
    pipeline_stats = {
        "pdfs_processed": 0,
        "chunks_generated": 0,
        "examples_generated": 0,
        "examples_after_validation": 0,
        "examples_after_dedup": 0,
        "validation_rejections": {}
    }
    
    for pdf_path in tqdm(pdf_files, desc="Processing PDFs"):
        # Extract
        try:
            doc = extract_document(str(pdf_path))
        except Exception as e:
            print(f"  Skipping {pdf_path.name}: {e}")
            continue
        
        pipeline_stats["pdfs_processed"] += 1
        
        # Chunk
        chunks = list(chunker.chunk_document(doc.pages, doc.source))
        pipeline_stats["chunks_generated"] += len(chunks)
        
        # Generate QA pairs per chunk
        for chunk in chunks:
            raw_examples = generate_qa_pairs(
                chunk,
                domain_context=domain_context,
                model=model
            )
            if not raw_examples:
                continue
            
            pipeline_stats["examples_generated"] += len(raw_examples)
            
            # Validate
            valid_examples, stats = validate_batch(raw_examples, chunk['text'])
            pipeline_stats["examples_after_validation"] += len(valid_examples)
            
            for reason, count in stats["reasons"].items():
                pipeline_stats["validation_rejections"][reason] = (
                    pipeline_stats["validation_rejections"].get(reason, 0) + count
                )
            
            all_examples.extend(valid_examples)
    
    # Deduplicate
    if deduplicate and len(all_examples) > 10:
        print(f"\nDeduplicating {len(all_examples)} examples...")
        all_examples = deduplicate_dataset(all_examples)
    
    pipeline_stats["examples_after_dedup"] = len(all_examples)
    
    # Write JSONL output
    output = Path(output_path)
    output.parent.mkdir(parents=True, exist_ok=True)
    
    with open(output, 'w', encoding='utf-8') as f:
        for example in all_examples:
            f.write(json.dumps(example, ensure_ascii=False) + '\n')
    
    print(f"\nDataset written to {output}")
    print(f"Total training examples: {len(all_examples)}")
    return pipeline_stats

if __name__ == "__main__":
    stats = build_dataset(
        pdf_dir="./pdfs",
        output_path="./output/training_data.jsonl",
        domain_context="a financial regulatory compliance expert",
        chunk_size=800,
        chunk_overlap=100,
        model="gpt-4o-mini"
    )
    print("\nPipeline Statistics:")
    for k, v in stats.items():
        print(f"  {k}: {v}")

Pipeline Performance Benchmarks

To give you a realistic sense of what to expect, here are the results from running this pipeline on three different document types. All tests used gpt-4o-mini, chunk size 800 tokens, on a 2025 MacBook Pro M3.

Documents Tested

• Legal contracts: 47 PDFs, ~320 pages total
• Technical manuals: 12 PDFs, ~890 pages total
• Research papers: 83 PDFs, ~640 pages total

Results

The validation rejection rate is highest for legal documents because they contain more ambiguous text, where the LLM either over-generates long answers or defaults to yes/no responses. Technical manuals perform best. structured, factual prose generates high-quality QA pairs consistently.

Cost Comparison: Manual vs. Automated

This cost difference fundamentally changes what’s feasible. Teams that previously couldn’t afford to fine-tune domain-specific models now can.

Choosing Your Generation Model

The model you use for generation has a significant impact on dataset quality. Here’s how the main options compare for this specific task:

For most engineering teams building domain-specific training sets, gpt-4o-mini at scale beats local models because the quality difference compounds: a 10% improvement in per-example quality across 5,000 examples is 500 better training examples, which meaningfully affects final model performance.

If you’re working with confidential documents, medical records, legal matter files, internal IP, local inference is the right call. SitePoint’s complete guide to running local LLMs in 2026 walks through the full local setup.

Advanced: Generating Multi-Turn Conversation Data

The pipeline above generates single-turn instruction-response pairs. For fine-tuning chat models, multi-turn conversation data often produces better results. Here’s an extended generator for that:

python

MULTI_TURN_PROMPT = """From this document excerpt, create a realistic 3-4 turn 
conversation between a user and a domain expert assistant.

The conversation should:
- Start with a broad question and progressively get more specific
- Include at least one follow-up or clarification
- Feel natural, not like a quiz
- Stay fully grounded in the provided text

Return as ChatML JSON with "messages" array."""

def generate_multiturn(chunk: dict, domain: str) -> Optional[dict]:
    """Generate a multi-turn conversation from a chunk."""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": MULTI_TURN_PROMPT},
            {"role": "user", "content": f"Domain: {domain}\n\nText:\n{chunk['text']}"}
        ],
        response_format={"type": "json_object"},
        temperature=0.8,
        max_tokens=1200
    )
    try:
        return json.loads(response.choices[0].message.content)
    except json.JSONDecodeError:
        return None

Mix your dataset: 70% single-turn instruction pairs for breadth, 30% multi-turn conversations for depth. This is roughly the ratio used in most instruction-tuned open-source models.

When Automated Generation Isn’t Enough

The pipeline above handles the majority of dataset-building scenarios well. But there are genuine cases where automated annotation should be supplemented or replaced:

Subjective judgment tasks: If your model needs to evaluate quality, rate sentiment on nuanced scales, or make ethical judgments, automated generation introduces systematic biases that are hard to detect and harder to fix.

Regulatory and compliance applications: Medical, legal, and financial models used in decision-making may require human-verified annotation chains for audit purposes. In these domains, the quality floor isn’t just about model performance — it’s about liability.

Low-resource languages and specialized notation: Scientific formulas, legal citations with specific formatting requirements, and non-Latin scripts often produce poor results with automated generation because the base LLM itself has limited competence in these areas.

Ground truth verification for evaluation sets: Even if your training set is fully automated, your evaluation set should have some human-verified examples to ensure your metrics are meaningful. This is a common place where teams invest in expert annotation even when they don’t for training data.

Understanding where human judgment genuinely adds value — versus where it’s just friction — is the key to running efficient labeling operations at scale. The broader data labeling landscape has evolved significantly around this distinction, with modern workflows combining automated pre-annotation with targeted human review at quality gates rather than end-to-end manual annotation.

Quality Metrics: Evaluating Your Dataset Before Training

Before you send your dataset to a fine-tuning job, run these checks:

python

# quality_report.py
import json
from collections import Counter
import statistics

def generate_quality_report(jsonl_path: str) -> dict:
    examples = []
    with open(jsonl_path) as f:
        for line in f:
            examples.append(json.loads(line))
    
    questions, answers = [], []
    for ex in examples:
        msgs = ex.get('messages', [])
        for m in msgs:
            if m['role'] == 'user':
                questions.append(m['content'])
            elif m['role'] == 'assistant':
                answers.append(m['content'])
    
    q_lengths = [len(q.split()) for q in questions]
    a_lengths = [len(a.split()) for a in answers]
    
    # Question type distribution
    q_types = Counter()
    for q in questions:
        q_lower = q.lower()
        if q_lower.startswith('what'): q_types['what'] += 1
        elif q_lower.startswith('how'): q_types['how'] += 1
        elif q_lower.startswith('why'): q_types['why'] += 1
        elif q_lower.startswith('when'): q_types['when'] += 1
        elif q_lower.startswith('where'): q_types['where'] += 1
        else: q_types['other'] += 1
    
    return {
        "total_examples": len(examples),
        "question_stats": {
            "mean_length": round(statistics.mean(q_lengths), 1),
            "median_length": statistics.median(q_lengths),
            "min_length": min(q_lengths),
            "max_length": max(q_lengths)
        },
        "answer_stats": {
            "mean_length": round(statistics.mean(a_lengths), 1),
            "median_length": statistics.median(a_lengths),
            "min_length": min(a_lengths),
            "max_length": max(a_lengths)
        },
        "question_type_distribution": dict(q_types.most_common()),
        "diversity_score": len(q_types) / max(q_types.values()),  # 1.0 = perfect distribution
    }

A healthy dataset shows question type diversity (not all “What is…” questions), answer lengths between 40-150 words for instruction tuning, and no extreme outliers in either direction.

Next Steps: From Dataset to Trained Model

Once you have a clean training_data.jsonl, The next natural steps are:

1. Split your dataset: Use an 85/10/5 train/validation/test split. Keep your test set aside entirely until final evaluation.
2. Choose a base model: For most domain-specific use cases, a 7B–8B parameter model (Llama 3.1 8B, Mistral 7B, Qwen 2.5 7B) fine-tuned on your dataset will outperform a general-purpose 70B model with prompt engineering.
3. Fine-tune with QLoRA: SitePoint’s fine-tuning guide covers QLoRA training configuration in detail. For preference data, the **DPO fine-tuning guide **is the better reference.
4. Evaluate against your held-out test set: Run both your fine-tuned model and the base model on identical test queries and compare. The gap is your signal.

If you’re building toward RAG rather than fine-tuning, the complete LangChain guide on SitePoint covers how to connect indexed documents to your LLM, a natural complement to the extraction work we did here.

Conclusion

Manual annotation is not a requirement for building training datasets, it’s a legacy assumption from before capable LLMs existed. The pipeline in this article demonstrates that you can go from a folder of raw PDFs to a validated, deduplicated, production-ready JSON dataset with a few hundred lines of Python and an API key.

The costs are dramatically lower than human annotation (often 100x). The throughput is unlimited — you can scale to tens of thousands of documents overnight. And the quality, when the pipeline is properly tuned, is competitive with human-generated data for factual QA tasks.

What you can’t fully automate is judgment, and that’s the right boundary. Use the automation for scale, apply human review at quality gates, and save expert annotation time for the edge cases that genuinely need it.

The complete source code for this pipeline is available to adapt. Start with a small batch of 5-10 PDFs, review the generated examples manually, tune your validation thresholds, and then scale.