4 posts tagged with "deepseek"

This article documents an evaluation harness for a Remote EU job classifier—but the real focus is AI observability: how to design traces, spans, metadata, scoring, and run-level grouping so you can debug, compare, and govern LLM behavior over time.

The script runs a batch of curated test cases, loads the latest production prompt from Langfuse (with a safe fallback), executes a structured LLM call, scores results, and publishes metrics back into Langfuse. That gives you:

• Reproducibility (prompt versions + test set + session IDs)
• Debuggability (one trace per test case; inspect inputs/outputs)
• Comparability (run-level aggregation; trend metrics across changes)
• Operational safety (flush guarantees, CI thresholds, rate-limit control)

Why "observability-first" evals matter​

A typical eval script prints expected vs actual and calls it a day. That's not enough once you:

• iterate prompts weekly,
• swap models,
• add guardrails,
• change schemas,
• tune scoring rubrics,
• and need to explain regressions to humans.

Observability-first evals answer questions like:

• Which prompt version produced the regression?
• Is accuracy stable but confidence becoming overconfident?
• Are failures clustered by location phrasing ("EMEA", "EU timezone", "Worldwide")?
• Did we increase tokens/latency without improving correctness?
• Can we click from CI logs straight into the trace of the failing example?

Langfuse becomes your "flight recorder": the trace is the unit of truth for what happened.

End-to-end architecture​

Observability design: what gets traced and why​

Trace strategy: one trace per test case​

Principle: if you can't click into an individual example, you can't debug.

Each test case produces a Langfuse trace (think "request-level unit"), tagged with:

• sessionId: groups a full run (critical for comparisons)
• testCaseId, description: anchors the trace to your dataset
• prompt metadata: name/label/version/hash (ideal)
• model metadata: provider, model name, parameters (ideal)

This makes failures navigable and filterable.

Span strategy: one generation per model call​

Inside each trace, you create a generation span for the model call:

• captures input (prompt + job posting)
• captures output (structured object + reason)
• captures usage (token counts)
• optionally captures latency (recommended)
• optionally captures model params (temperature, top_p, etc.)

Even if the script is "just evals," treat each example like production traffic. That's how you build a reliable debugging workflow.

Prompt governance: Langfuse prompts + fallback behavior​

Your harness fetches a prompt by name and label:

• name: job-classifier
• label: production

If prompt retrieval fails or is disabled (e.g., SKIP_LANGFUSE_PROMPTS=true), it uses a local fallback prompt.

Observability tip: always record the effective prompt identity​

To compare runs, you want "which exact prompt did this use?" in trace metadata. If your prompt fetch returns versioning info, store:

• promptName
• promptLabel
• promptVersion or promptId or promptHash

If it does not return version info, you can compute a stable hash of the prompt text and store that (lightweight, extremely useful).

Structured output: Zod as an observability contract​

The classifier returns:

• isRemoteEU: boolean
• confidence: "high" | "medium" | "low"
• reason: string

Why structured output is observability, not just "parsing"​

A strict schema:

• removes ambiguity ("was that JSON-ish text or valid data?")
• enables stable scoring and aggregation
• prevents downstream drift as prompts change
• improves triage because the same fields are always present

If you ever add fields like region, countryHints, remotePolicy, do it via schema extension and keep historical compatibility in your scorer.

The full eval lifecycle as a trace model​

This is what you want stored per test case:

When a case fails, you should be able to answer in one click:

• Which prompt version?
• What input text exactly?
• What output object exactly?
• What scoring decision and why?
• Was the model "confidently wrong"?

Scoring and metrics: accuracy is necessary, insufficient​

Your harness logs two scores:

1. remote-eu-accuracy A numeric score from your scorer. This can be binary (0/1) or continuous (0..1). Continuous is often better because it supports partial credit and more informative trend analysis.

2. confidence-match A binary score (1/0) tracking whether the model's confidence matches expected confidence.

Observability tip: store scorer metadata as the comment (or trace metadata)​

A score without context is hard to debug. For incorrect cases, write comments like:

• expected vs actual isRemoteEU
• expected vs actual confidence
• a short reason ("Predicted EU-only due to 'EMEA' but posting says US time zones")

Also consider storing structured metadata (if your Langfuse SDK supports it) so you can filter/group later.

Run-level grouping: session IDs as your "eval run" primitive​

A sessionId = eval-${Date.now()} groups the whole batch. This enables:

• "show me all traces from the last run"
• comparisons across runs
• slicing by prompt version across sessions
• CI links that land you on the run dashboard

Recommendation: include additional stable tags:

• gitSha, branch, ciBuildId (if running in CI)
• model and promptVersion (for quick comparisons)

Even if you don't have them now, design the metadata schema so adding them later doesn't break anything.

Mermaid: evaluation flow, sequence, and data model (together)​

1) Flow: control plane of the batch run​

2) Sequence: what actually happens per case​

3) Data model: eval artifacts​

How to run (and make it debuggable in one click)​

Environment variables​

Required:

• LANGFUSE_SECRET_KEY
• LANGFUSE_PUBLIC_KEY
• LANGFUSE_BASE_URL
• DEEPSEEK_API_KEY

Optional:

• SKIP_LANGFUSE_PROMPTS=true (use local prompt fallback)

Run:

pnpm tsx scripts/eval-remote-eu-langfuse.ts

Local prompt fallback:

SKIP_LANGFUSE_PROMPTS=true pnpm tsx scripts/eval-remote-eu-langfuse.ts

Observability tip: print a stable "run header"​

In console output (and CI logs), it helps to print:

• sessionId
• model name
• prompt version/hash
• total test cases

That turns logs into an index into Langfuse.

Debugging workflow: from CI failure to root cause​

When accuracy drops below threshold and CI fails, you want a deterministic workflow:

1. Open the Langfuse session for the run (grouped by sessionId)

2. Filter traces where remote-eu-accuracy = 0 (or below some threshold)

3. For each failing trace:

   • check prompt version/hash
   • check job posting input text (location phrasing is often the culprit)
   • inspect structured output (especially confidence)
   • read the reason for the scorer's decision

Practical tips & gotchas (observability edition)​

1) Always flush telemetry​

If you exit early, you can lose the most important traces. Ensure flushAsync() happens even on errors (e.g., in a finally block) and only exit after flush completes.

2) Don't parallelize blindly​

Parallel execution improves speed but can:

• amplify rate limits
• introduce noisy latency
• create non-deterministic output ordering in logs

If you do parallelize, use bounded concurrency and capture per-case timing.

3) Track prompt identity, not just prompt text​

Prompt text alone is hard to compare across runs. Record version/hash so you can correlate changes with performance.

4) Separate "correctness" from "calibration"​

A model can get higher accuracy while becoming confidently wrong on edge cases. Keeping confidence-match (or richer calibration metrics later) prevents hidden regressions.

5) Add slice metrics before you add more test cases​

Instead of only "overall accuracy," compute accuracy by category:

• "EU-only"
• "Worldwide remote"
• "EMEA" phrasing
• "Hybrid" / "On-site"
• "Contractor / employer-of-record constraints"

This reveals what's actually breaking when a prompt changes.

Suggested next upgrades (high leverage)​

A) Add latency and cost proxies​

Record:

• duration per generation span (ms)
• token totals per case

Then you can chart:

• cost/latency vs accuracy
• regressions where prompt got longer but not better

B) Add a "reason quality" score (optional, small rubric)​

Create a third score like reason-quality to detect when explanations degrade (too vague, irrelevant, or missing key constraints). Keep it light—don't overfit to phrasing.

C) Prompt A/B within the same run​

Evaluate production vs candidate prompts on the same test set:

• two sessions (or two labels within one session)
• compare metrics side-by-side in Langfuse

Docusaurus note: Mermaid support​

If Mermaid isn't rendering, enable it in Docusaurus:

// docusaurus.config.js
const config = {
  markdown: { mermaid: true },
  themes: ["@docusaurus/theme-mermaid"],
};
module.exports = config;

The takeaway: observability is the eval superpower​

A well-instrumented eval harness makes improvements measurable and regressions explainable:

• traces turn examples into clickable evidence
• structured outputs stabilize scoring
• session IDs make runs comparable
• multiple metrics prevent hidden failure modes

If you treat evals like production requests—with traces, spans, and scores—you'll iterate faster and break less.

Introduction​

Remote job postings are noisy, inconsistent, and often misleading. A role is labeled “Remote”, but the actual constraints show up in one sentence buried halfway down the description:

• “Remote (US only)”
• “Must be authorized to work in the U.S. without sponsorship”
• “EU/EEA only due to payroll constraints”
• “Must overlap PST business hours”
• “Hybrid, 2 days/week in-office”

This article breaks down a LangGraph System that pre-screens job postings using DeepSeek structured extraction, then applies deterministic rules to instantly decide:

✅ Apply
❌ Reject (with reasons + quotes)

The goal is simple: filter out non-viable jobs before you spend time applying.

The Problem: “Remote” Doesn’t Mean “Work From Anywhere”​

Why Traditional Filters Fail​

Keyword filters (“remote”, “anywhere”) fail because job descriptions are written inconsistently and constraints can be phrased in dozens of ways:

1. Remote but country-restricted
2. Remote but timezone-restricted
3. Remote but payroll-limited
4. Remote but no visa sponsorship
5. Remote but actually hybrid

Instead of relying on fragile string matching, we use an LLM to read the description like a human, but output machine-usable constraints.

System Overview​

This agent evaluates job postings in two phases:

1. Analyze job text (DeepSeek + structured schema)
2. Check eligibility (deterministic rules)

What It Detects​

• Location scope
  • US-only / EU-only / Global / Specific regions / Unknown
• Remote status
  • fully-remote / remote-with-restrictions / hybrid / on-site / unknown
• Visa sponsorship
  • explicit yes/no/unknown
• Work authorization requirements
  • must be authorized in US/EU, or not specified
• Timezone restrictions
  • PST overlap / CET overlap / etc.

Tech Stack​

• LangGraph: workflow orchestration and state transitions
• DeepSeek: high-signal extraction from messy job text (deepseek-chat)
• LangChain structured output: strict schema → stable parsing
• Deterministic rules engine: eligibility enforcement without “LLM vibes”

Architecture Patterns​

1) LangGraph Workflow​

Instead of a linear script, the system is a graph-driven workflow:

This shape is production-friendly because the workflow can expand safely:

• add salary checks
• add tech stack fit scoring
• add seniority mismatch detection
• add contractor vs employee constraints

Typed State + Structured Extraction​

State Model (TypedDict)​

LangGraph becomes far more reliable when state is explicit:

class JobScreeningState(TypedDict):
    job_title: str
    company: str
    description: str
    location: str
    url: str

    # Candidate requirements
    candidate_needs_visa_sponsorship: bool
    requires_fully_remote: bool
    requires_worldwide_remote: bool
    candidate_locations: List[str]

    # Output
    is_eligible: bool
    rejection_reasons: List[str]

    # Extracted requirements
    location_requirement: Optional[str]
    specific_regions: List[str]
    excluded_regions: List[str]
    visa_sponsorship_available: Optional[bool]
    work_authorization_required: Optional[str]
    remote_status: str
    timezone_restrictions: List[str]
    confidence: str
    key_phrases: List[str]
    analysis_explanation: Optional[str]

DeepSeek Extraction: Converting Messy Text Into Policy Constraints​

Why Structured Output Is Non-Negotiable​

Freeform LLM output is fragile. A production system needs predictable extraction. This agent forces DeepSeek into a strict schema:

class JobAnalysisSchema(TypedDict):
    location_requirement: Literal["US-only", "EU-only", "Global", "Specific-regions", "Unknown"]
    specific_regions: List[str]
    excluded_regions: List[str]
    remote_status: Literal["fully-remote", "remote-with-restrictions", "hybrid", "on-site", "unknown"]
    visa_sponsorship_available: Optional[bool]
    work_authorization_required: Literal["US-only", "EU-only", "Any", "Unknown"]
    timezone_restrictions: List[str]
    confidence: Literal["high", "medium", "low"]
    key_phrases: List[str]
    explanation: str

With this contract, the agent can safely feed extracted requirements into deterministic logic.

Token Efficiency: Keep Only High-Signal Lines​

Job descriptions are long. Constraints are usually short. To reduce tokens and improve extraction precision, the system trims input to keyword-adjacent lines:

KEYWORDS = (
    "remote", "anywhere", "worldwide", "timezone", "sponsor", "visa",
    "authorized", "work authorization", "must be located", "eligible to work",
    "location", "region", "country", "overlap", "hours", "time zone"
)

def _keep_relevant(text: str, window: int = 2) -> str:
    lines = text.splitlines()
    keep = set()
    for i, ln in enumerate(lines):
        if any(k in ln.lower() for k in KEYWORDS):
            for j in range(max(0, i - window), min(len(lines), i + window + 1)):
                keep.add(j)
    return "\n".join(lines[i] for i in sorted(keep)) or text

This improves the system in four ways:

• lower inference cost
• faster runtime
• less noise
• fewer hallucination opportunities

Heuristics + DeepSeek: Hybrid Extraction That Wins​

Before invoking DeepSeek, the system runs a tiny heuristic pre-check:

• detects obvious “Remote (Worldwide)”
• detects “Remote (US only)”
• detects “Hybrid / On-site”

def _fast_heuristic_precheck(state: JobScreeningState) -> Optional[Dict[str, Any]]:
    loc = state.get("location", "") or ""
    desc = state.get("description", "") or ""
    seed: Dict[str, Any] = {}

    if _looks_worldwide(loc) or _looks_worldwide(desc):
        seed["location_requirement"] = "Global"
        seed["remote_status"] = "fully-remote"

    if (_looks_us_only(loc) or _looks_us_only(desc)) and not seed.get("location_requirement"):
        seed["location_requirement"] = "US-only"

    if _looks_hybrid_or_onsite(loc):
        seed["remote_status"] = "hybrid"

    return seed if seed else None

DeepSeek still performs the full extraction, but seeding improves resilience against incomplete metadata.

Eligibility Rules: Enforcing Worldwide Remote Strictly​

The most valuable mode is strict worldwide remote filtering:

If requires_worldwide_remote=True, the job must satisfy ALL of the following:

• remote_status == "fully-remote"
• location_requirement == "Global"
• no specific_regions
• no timezone_restrictions

if state["requires_worldwide_remote"]:
    if state["remote_status"] != "fully-remote":
        rejection_reasons.append(
            f"Not worldwide-remote: remote status is '{state['remote_status']}'"
        )
    if state["location_requirement"] != "Global":
        rejection_reasons.append(
            f"Not worldwide-remote: location requirement is '{state['location_requirement']}'"
        )
    if state["specific_regions"]:
        rejection_reasons.append(
            f"Not worldwide-remote: restricted to {state['specific_regions']}"
        )
    if state["timezone_restrictions"]:
        rejection_reasons.append(
            f"Not worldwide-remote: timezone restrictions {state['timezone_restrictions']}"
        )

This instantly rejects “remote marketing” jobs like:

• “Remote, EU only”
• “Remote, US/Canada preferred”
• “Remote, PST overlap required”

Visa Sponsorship Semantics: Correct and Safe​

Sponsorship logic is easy to get wrong. The correct behavior:

• reject only when sponsorship is explicitly not available (False)
• do not reject on unknown (None)

if state["candidate_needs_visa_sponsorship"]:
    if state["visa_sponsorship_available"] is False:
        rejection_reasons.append(
            "Job does not offer visa sponsorship, but candidate needs sponsorship"
        )

This avoids dropping jobs that simply don’t mention sponsorship.

Explainability: Rejection Reasons + Key Phrases​

Trust requires receipts. The system stores:

• rejection_reasons (deterministic outcomes)
• key_phrases (quotes that triggered the decision)
• analysis_explanation (LLM summary for debugging)

That produces outputs like:

• “Job requires US location; candidate is not in US”
• “Not worldwide-remote: timezone restrictions ['US Pacific business hours']”
• key phrases like “Must be authorized to work in the U.S. without sponsorship”

Real-World Test Scenarios​

The included test suite covers the most common job board traps:

1. US-only remote + no sponsorship
2. Remote worldwide (work from anywhere)
3. EU-only remote
4. Remote with timezone overlap requirement

This validates both extraction quality and deterministic enforcement.

Production Enhancements​

1) Add a Match Score (Not Only Pass/Fail)​

Binary decisions are clean, but scoring improves ranking:

• 100 = perfect match
• 70 = acceptable
• 30 = not worth it
• 0 = reject

2) Cache Results by URL Hash​

You already compute a stable thread_id from the job URL. Persist results keyed by:

• url_hash
• model version
• rule version

This prevents re-analyzing duplicate postings.

3) Detect Payroll Constraints Explicitly​

Add signals for:

• “We can only hire in countries where we have an entity”
• “Deel/Remote.com limited coverage”
• “W2 only / no contractors”

This is one of the highest ROI improvements for global applicants.

Conclusion​

This LangGraph System turns job descriptions into enforceable constraints:

• DeepSeek extracts remote reality, location scope, and sponsorship signals
• Structured output makes extraction stable and machine-safe
• Deterministic rules enforce candidate requirements precisely
• Worldwide-remote mode filters out fake “remote” listings instantly
• Decisions are explainable with reasons and quotes

This is how you scale job hunting without wasting time: automate rejection early, apply only where it can actually work.

References​

• LangGraph Documentation
• LangChain Structured Output
• DeepSeek Platform
• LangChain ChatOpenAI Integration

Introduction​

Generating long-form audio content—audiobooks spanning hours, educational courses, or extended podcasts—presents unique challenges: API rate limits, network failures, resource constraints, and the sheer duration of processing. This article explores a production-ready architecture for long-running TTS pipelines that can gracefully handle long-form generation tasks, resume after failures, and maintain state across distributed systems.

Built with LangGraph, the system orchestrates complex workflows involving AI content generation (DeepSeek), text-to-speech conversion (OpenAI TTS), and distributed storage (Cloudflare R2). The key innovation: PostgreSQL checkpointing enables resumable execution, making it possible to generate 5-30+ minute audio segments reliably, even when individual API calls or processing steps fail.

The Challenge: Long-Form Audio at Scale​

Why Long-Running Pipelines Are Hard​

Traditional TTS approaches fail at scale:

1. Time Constraints: A 30-minute audio narrative requires ~4,500 words, chunked into 10-15 API calls, taking 2-5 minutes to generate
2. Failure Points: Each step (text generation, chunking, TTS, storage) can fail independently
3. Memory Pressure: Holding all audio segments in memory for hours is impractical
4. Cost Management: Retrying from scratch wastes API credits and compute time
5. State Loss: Without persistence, crashes mean starting over

Our Solution: Stateful Orchestration

• LangGraph manages workflow state transitions
• PostgreSQL persists checkpoints after each successful step
• R2 provides durable storage for completed segments
• Resumable execution using thread_id for job recovery

System Overview​

The pipeline orchestrates three main workflows:

1. Research Generation: Structured content research using DeepSeek
2. Narrative Text Generation: Long-form content creation with context awareness
3. Audio Synthesis: Text-to-speech conversion with OpenAI TTS and Cloudflare R2 storage

Tech Stack​

• LangGraph: State machine orchestration with built-in checkpointing
• DeepSeek: Long-form text generation (deepseek-chat, 2500+ token outputs)
• OpenAI TTS: Streaming audio synthesis (gpt-4o-mini-tts, 4096 char limit per request)
• PostgreSQL: Durable checkpointing for long-running jobs (Neon serverless for production)
• Cloudflare R2: S3-compatible storage with zero egress fees (critical for multi-GB audio)
• FastAPI: Async REST API for non-blocking long operations
• Docker: Containerized deployment with ffmpeg for audio merging

Why This Stack for Long-Running Jobs:

• Postgres checkpointing: Resume from any point in the workflow (text generation → chunking → TTS → upload)
• Streaming TTS: Memory-efficient direct-to-disk writes (no buffering entire audio in RAM)
• R2 durability: Segments uploaded immediately, survive process crashes
• Async execution: Non-blocking background processing for hours-long jobs

Architecture Patterns​

1. Core LangGraph State Machine​

The system implements three distinct LangGraph workflows, each optimized for specific tasks.

2. Research Generation Pipeline​

The research pipeline generates structured research content using a focused LangGraph workflow.

Key Features:

• Low temperature (0.3) for factual accuracy
• Structured JSON output with validation
• Evidence level classification (A/B/C)
• Relevance scoring for topic matching

3. Long-Form Text Generation Pipeline​

The most sophisticated workflow, supporting both full generation and audio-only modes.

Conditional Routing Logic:

def should_skip_text_generation(state: TextState) -> str:
    """Route to text generation or skip to audio."""
    if state.get("existing_content") and state["existing_content"].get("text"):
        return "chunk_text"  # Audio-only mode
    return "generate_text"  # Full generation

def should_generate_audio(state: TextState) -> str:
    """Route to audio generation or end."""
    if state.get("generate_audio", True):
        return "chunk_text"
    return END  # Text-only mode

4. Audio Generation Pipeline (Standalone)​

A simplified pipeline for generic long-form narration.

Iterative Chunk Processing:

The system uses a recursive edge pattern for processing chunks:

g.add_conditional_edges(
    "tts_one_chunk",
    edge_should_continue,
    {
        "tts_one_chunk": "tts_one_chunk",  # Loop back
        "finalize": "finalize",             # Exit loop
    },
)

def edge_should_continue(state: JobState) -> str:
    if state["chunk_index"] < len(state["chunks"]):
        return "tts_one_chunk"
    return "finalize"

Deep Dive: Key Architectural Components​

State Management​

LangGraph uses typed state dictionaries for type safety and IDE support:

class TextState(TypedDict):
    # Input metadata
    content_id: int
    title: str
    content_type: str
    language: str
    target_duration_minutes: int | None
    
    # Generation data
    research_items: list[dict]
    existing_content: dict | None
    generated_text: str | None
    
    # TTS fields
    voice: str
    chunks: List[str]
    segment_urls: List[str]
    manifest_url: Optional[str]
    audio_url: Optional[str]
    
    # Control flow
    generate_audio: bool
    database_saved: bool
    error: str | None

Postgres Checkpointing: The Key to Long-Running Resilience​

For long-running jobs, checkpointing is non-negotiable. Without it, a network glitch at minute 25 of a 30-minute generation means restarting from scratch.

How Checkpointing Works:

async def run_pipeline(state: TextState, thread_id: str):
    db_url = os.getenv("DATABASE_URL")
    
    async with AsyncPostgresSaver.from_conn_string(db_url) as checkpointer:
        await checkpointer.setup()  # Creates checkpoint tables
        app = build_graph(checkpointer=checkpointer)
        config = {"configurable": {"thread_id": thread_id}}
        
        # LangGraph automatically saves state after each node execution
        final_state = await app.ainvoke(state, config=config)
        return final_state

What Gets Checkpointed:

• Complete state dictionary after each node
• Edge transitions and routing decisions
• Timestamps and execution metadata
• Partial results (generated text, uploaded segment URLs)

Recovery Example:

# Job crashes after generating 8 of 12 TTS segments
# Resume with same thread_id:
final_state = await run_pipeline(initial_state, thread_id="job-12345")

# LangGraph:
# 1. Loads last checkpoint from Postgres
# 2. Sees 8 segments already uploaded to R2
# 3. Continues from segment 9
# 4. Completes remaining 4 segments

Production Benefits:

• Cost Savings: No wasted API calls on retry
• Time Efficiency: Resume from 80% complete, not 0%
• Reliability: Transient failures (rate limits, timeouts) don't kill long-form jobs
• Observability: Query checkpoint table to monitor progress
• Parallel Execution: Multiple jobs with different thread_id values

Text Chunking Algorithm: Optimizing for Long-Form Narration​

For 30-minute audio (4,500+ words), naive chunking creates jarring transitions. Our algorithm balances API constraints with narrative flow:

Constraints:

• OpenAI TTS: 4,096 character limit per request
• Target: ~4,000 chars per chunk (safety margin)
• Goal: Natural pauses at paragraph/sentence boundaries

Strategy:

def chunk_text(text: str, max_chars: int = 4000) -> List[str]:
    """
    Multi-level chunking for long-form content:
    
    1. Split by paragraphs (\n\n) - natural topic boundaries
    2. Accumulate paragraphs until approaching 4K limit
    3. If single paragraph > 4K, split by sentences
    4. If single sentence > 4K, split mid-sentence (rare edge case)
    
    Result: 10-15 chunks for 30-min audio, each ending at natural pause
    """
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks = []
    buf = []
    
    for p in paragraphs:
        candidate = "\n\n".join(buf + [p]) if buf else p
        if len(candidate) <= max_chars:
            buf.append(p)
        else:
            if not buf:
                # Paragraph too large - split by sentences
                sentences = re.split(r"(?<=[.!?])\s+", p)
                # Accumulate sentences with same logic...
            else:
                chunks.append("\n\n".join(buf))
                buf = [p]
    
    return chunks

Why This Matters for Long-Form:

• Seamless Merging: Chunk boundaries at natural pauses prevent audio glitches
• Even Distribution: Avoids tiny final chunks (better for progress tracking)
• Memory Efficiency: Process one chunk at a time, not entire 4,500-word text
• Resumability: Each chunk is independent; can resume mid-sequence

OpenAI TTS Streaming​

Efficient audio generation using streaming responses:

async def node_tts_one_chunk(state: JobState) -> JobState:
    chunk_text = state["chunks"][state["chunk_index"]]
    segment_path = f"segment_{state['chunk_index']:04d}.mp3"
    
    client = OpenAI()
    
    # Stream directly to disk (memory efficient)
    with client.audio.speech.with_streaming_response.create(
        model="gpt-4o-mini-tts",
        voice=state["voice"],
        input=chunk_text,
        response_format="mp3",
    ) as response:
        response.stream_to_file(segment_path)
    
    # Upload to R2
    r2_url = upload_to_r2(segment_path, state["job_id"])
    
    return {
        **state,
        "segment_urls": [*state["segment_urls"], r2_url],
        "chunk_index": state["chunk_index"] + 1,
    }

Audio Merging Strategy​

The system uses ffmpeg for high-quality concatenation:

async def node_generate_audio(state: TextState) -> TextState:
    # Generate all segments...
    
    # Create concat list for ffmpeg
    file_list_path.write_text(
        "\n".join(f"file '{segment}'" for segment in segment_paths)
    )
    
    # Merge using ffmpeg (codec copy - fast and lossless)
    subprocess.run([
        "ffmpeg", "-f", "concat", "-safe", "0",
        "-i", str(file_list_path),
        "-c", "copy",  # No re-encoding
        str(merged_path)
    ])
    
    # Fallback to binary concatenation if ffmpeg unavailable
    if not merged_path.exists():
        with open(merged_path, "wb") as merged:
            for segment in segment_paths:
                merged.write(segment.read_bytes())

Cloudflare R2 Integration​

S3-compatible storage for globally distributed audio:

def get_r2_client():
    return boto3.client(
        's3',
        endpoint_url=f'https://{R2_ACCOUNT_ID}.r2.cloudflarestorage.com',
        aws_access_key_id=R2_ACCESS_KEY_ID,
        aws_secret_access_key=R2_SECRET_ACCESS_KEY,
        config=Config(signature_version='s3v4'),
    )

def upload_to_r2(file_path: Path, job_id: str) -> str:
    key = f"{job_id}/{file_path.name}"
    
    client.put_object(
        Bucket=R2_BUCKET_NAME,
        Key=key,
        Body=file_path.read_bytes(),
        ContentType='audio/mpeg',
    )
    
    return f"{R2_PUBLIC_DOMAIN}/{key}"

Structured Content Generation​

Narrative Architecture Framework​

The system implements a flexible content framework with customizable sections:

Key Components:

1. Introduction (2-3 min): Hook the listener and set expectations
2. Context: Background information and relevance
3. Core Content: Main topic introduction with clear structure
4. Examples: Concrete illustrations and case studies
5. Deep Dive: Detailed exploration of key concepts
6. Applications: Practical use cases and implementation
7. Advanced Topics: Nuanced discussion for engaged learners
8. Synthesis: Connect all concepts together
9. Takeaways: Summary of key points
10. Conclusion: Clear closing and next steps

Dynamic Content Adaptation​

def build_content_prompt(state: TextState) -> str:
    minutes = state.get("target_duration_minutes") or 5
    target_words = int(minutes * 150)  # 150 words per minute narration
    
    content_type = state.get("content_type")
    
    # Select architecture based on content type
    architecture = generate_content_architecture(content_type)
    
    return f"""
Create a {state['language']} narrative for audio:

TOPIC: {state['title']}
TYPE: {content_type}
TARGET: {target_words} words ({minutes} minutes)

{architecture}

RESEARCH CONTEXT:
{format_research_summary(state['research_items'])}

Requirements:
- Plain text only (no markdown)
- Natural paragraph breaks
- Engaging, clear tone
- Appropriate language for audio listening
"""

API Endpoints​

FastAPI Service Layer​

Endpoint Implementations:

@app.post("/api/research/generate")
async def research_endpoint(req: ResearchRequest):
    """Generate research context using LangGraph + DeepSeek."""
    return await generate_research(req)

@app.post("/api/text/generate")
async def text_endpoint(req: TextGenerationRequest):
    """Generate long-form text content (text-only mode)."""
    return await generate_text(req)

@app.post("/api/audio/generate")
async def audio_endpoint(req: TextGenerationRequest):
    """Generate audio from existing content (audio-only mode)."""
    return await generate_audio(req)

@app.post("/api/tts/generate")
async def tts_endpoint(req: TTSRequest, background_tasks: BackgroundTasks):
    """Generic TTS generation (fire-and-forget)."""
    return await generate_tts(req, background_tasks)

Deployment Architecture​

Docker Containerization​

FROM python:3.12-slim

WORKDIR /app

# Install ffmpeg for audio merging
RUN apt-get update && apt-get install -y ffmpeg && rm -rf /var/lib/apt/lists/*

# Install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY . .

# Expose port
EXPOSE 8080

# Run FastAPI server
CMD ["uvicorn", "langgraph_server:app", "--host", "0.0.0.0", "--port", "8080"]

Environment Configuration​

# AI Services
DEEPSEEK_API_KEY=sk-...
OPENAI_API_KEY=sk-...

# Database (Neon Postgres)
DATABASE_URL=postgresql://user:pass@host/db?sslmode=require

# Cloudflare R2
R2_ACCOUNT_ID=...
R2_ACCESS_KEY_ID=...
R2_SECRET_ACCESS_KEY=...
R2_BUCKET_NAME=longform-tts
R2_PUBLIC_DOMAIN=https://pub-longform-tts.r2.dev

Cloudflare Workers Deployment​

# wrangler.toml
name = "langgraph-tts"
compatibility_date = "2024-01-01"

[build]
command = "docker build -t langgraph-tts ."

[[services]]
name = "langgraph-tts"
image = "langgraph-tts:latest"

[env]
PORT = "8080"

[[r2_buckets]]
binding = "LONGFORM_TTS"
bucket_name = "longform-tts"

Production Considerations​

Performance Metrics for Long-Running Jobs​

Benchmarks (30-minute audio generation):

Long-Running Job Profile:

# Example: 2-hour audiobook chapter
text_length = 18,000 words
chunks = 45  # ~4,000 chars each
tts_time = 45 * 15s = 11.25 minutes
text_gen_time = 2-3 minutes
total_time = ~15 minutes for 2-hour audio

# Checkpoint frequency:
# - 1 after text generation
# - 45 after each TTS segment
# - 1 after merge
# Total: 47 recovery points

Failure Recovery Times:

• Crash at 80% complete → Resume in 1–2 seconds, continue from segment 36/45
• Network timeout on segment 20 → Retry only segment 20, not segments 1–19
• Database connection loss → Reconnect and load last checkpoint (<500ms)

Error Handling & Resilience​

async def node_generate_text(state: TextState) -> TextState:
    try:
        llm = ChatDeepSeek(model="deepseek-chat", temperature=0.7, max_tokens=2500)
        prompt = build_therapeutic_prompt(state)
        
        resp = await llm.ainvoke([HumanMessage(content=prompt)])
        text = clean_for_tts(resp.content)
        
        return {**state, "generated_text": text, "error": None}
    except Exception as e:
        print(f"❌ Text generation failed: {e}")
        return {**state, "error": str(e)}

Monitoring & Observability​

Key metrics to track:

1. Generation Metrics:

   • Text generation latency (DeepSeek)
   • TTS latency per chunk (OpenAI)
   • Total pipeline duration

2. Quality Metrics:

   • Text length vs target duration
   • Chunk count and size distribution
   • Audio segment file sizes

3. Infrastructure Metrics:

   • R2 upload success rate
   • Database checkpoint writes
   • ffmpeg merge success rate

4. Cost Metrics:

   • DeepSeek token usage
   • OpenAI TTS character count
   • R2 storage and bandwidth

Scaling Patterns​

Horizontal Scaling:

• FastAPI instances behind load balancer
• Stateless design (state in Postgres)
• R2 for distributed storage

Batch Processing:

async def batch_generate_audio(goal_ids: List[int]):
    """Process multiple goals in parallel."""
    tasks = [run_pipeline(build_state(id), f"batch-{id}") for id in goal_ids]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    return results

Queue-Based Processing:

• Use background tasks for long-running jobs
• Celery/Redis for distributed task queue
• Webhook callbacks for completion notifications

Performance Optimization​

Chunking Optimization​

# Optimize chunk size for TTS quality vs API limits
OPTIMAL_CHUNK_SIZE = 3000  # Sweet spot for natural pauses

# Parallel TTS generation (with rate limiting)
async def parallel_tts_generation(chunks: List[str], max_concurrent: int = 3):
    semaphore = asyncio.Semaphore(max_concurrent)
    
    async def generate_with_limit(chunk, index):
        async with semaphore:
            return await generate_tts_segment(chunk, index)
    
    tasks = [generate_with_limit(c, i) for i, c in enumerate(chunks)]
    return await asyncio.gather(*tasks)

Caching Strategy​

# Cache research results for similar goals
@lru_cache(maxsize=100)
def get_research_for_goal_type(therapeutic_type: str, age: int):
    """Cache research by type + age bracket."""
    return fetch_research(therapeutic_type, age)

# Cache text generation for re-use
async def get_or_generate_text(goal_id: int):
    existing = await db.fetch_story(goal_id)
    if existing and existing.created_at > datetime.now() - timedelta(days=7):
        return existing.text
    return await generate_new_text(goal_id)

Testing Strategy​

Unit Tests​

def test_chunk_text_respects_limit():
    long_text = "word " * 2000
    chunks = chunk_text(long_text, max_chars=4000)
    
    for chunk in chunks:
        assert len(chunk) <= 4000

def test_clean_for_tts_removes_markdown():
    text = "# Title\n\n**bold** and `code`"
    cleaned = clean_for_tts(text)
    assert "#" not in cleaned
    assert "**" not in cleaned
    assert "`" not in cleaned

Integration Tests​

@pytest.mark.asyncio
async def test_full_pipeline():
    state = {
        "goal_id": 1,
        "goal_title": "Test anxiety reduction",
        "therapeutic_goal_type": "anxiety_reduction",
        "age": 8,
        # ... other fields
    }
    
    result = await run_pipeline(state, "test-thread-1")
    
    assert result["generated_text"] is not None
    assert len(result["chunks"]) > 0
    assert result["audio_url"] is not None
    assert result["error"] is None

Lessons Learned​

1. State Design is Critical​

• Use TypedDict for type safety
• Keep state flat (avoid deep nesting)
• Include metadata for debugging (timestamps, IDs)

2. Checkpoint Strategically​

• Not all workflows need checkpointing
• Audio-only mode: disable checkpoints to avoid schema issues
• Use thread_id conventions: {workflow}-{entity_id}-{timestamp}

3. Error Recovery​

• Graceful degradation (segments work even if merge fails)
• Fallback strategies (binary concat if ffmpeg unavailable)
• Preserve partial results (individual segments in R2)

4. Cost Management​

• Monitor token usage (DeepSeek is cost-effective at $0.14/$0.28 per 1M tokens)
• OpenAI TTS: $15 per 1M characters
• R2 storage: $0.015/GB/month (much cheaper than S3)

5. Content Quality​

• Structured frameworks improve consistency
• Repetition aids retention and comprehension
• Audience-appropriate language is crucial for engagement

Future Enhancements​

1. Multi-Voice Narratives​

# Support character dialogue with different voices
voices = {
    "narrator": "cedar",
    "child_character": "nova",
    "parent_character": "marin"
}

2. Emotion-Adaptive TTS​

# Adjust voice parameters based on content emotion
def get_tts_params(text: str) -> dict:
    sentiment = analyze_sentiment(text)
    
    if sentiment == "calm":
        return {"speed": 0.9, "pitch": 0}
    elif sentiment == "energetic":
        return {"speed": 1.1, "pitch": 2}

3. Real-Time Streaming​

# Stream audio as it's generated (SSE)
async def stream_audio_generation(goal_id: int):
    async for chunk_url in generate_audio_stream(goal_id):
        yield f"data: {json.dumps({'chunk_url': chunk_url})}\n\n"

4. Multilingual Support​

• Expand beyond Romanian and English
• Voice selection per language
• Cultural adaptation of content frameworks

Conclusion​

This LangGraph-based TTS architecture demonstrates several key patterns:

1. Composable Workflows: Three distinct pipelines sharing common components
2. Conditional Routing: Smart flow control based on state
3. Durable Execution: PostgreSQL checkpointing for resilience
4. Streaming Efficiency: Direct-to-disk TTS for memory optimization
5. Distributed Storage: R2 for globally accessible audio

The system successfully processes 5-30+ minute long-form narratives (up to 7,000+ words), generating research-backed content, converting to high-quality audio, and delivering via CDN—all while maintaining resumability after failures and full observability.

Real-World Performance:

• 30-minute generation: 12-15 TTS chunks, ~3-5 minutes total processing time
• Failure recovery: Resume from any checkpoint in <1 second
• Cost efficiency: $0.02-$0.07 per 30-minute audio (DeepSeek + OpenAI TTS)
• Throughput: 10+ concurrent jobs on single instance

Key Takeaways for Long-Running Pipelines:

• LangGraph + Postgres checkpointing is essential for long-form workflows
• Streaming TTS to disk prevents memory exhaustion on long generations
• Smart chunking (4K chars) balances API limits with narrative coherence
• Immediate R2 uploads ensure partial results survive crashes
• Async architecture enables fire-and-forget long operations
• Thread-based recovery makes interrupted jobs trivial to resume

The architecture scales to long-form audio generation: audiobooks (10+ hours), comprehensive courses, documentary narration, or serialized storytelling—any use case where reliability and resumability are non-negotiable.

References​

• LangGraph Documentation
• DeepSeek API
• OpenAI TTS API
• Cloudflare R2
• PostgreSQL Async Python

This architecture powers long-form audio generation, combining LangGraph orchestration, OpenAI TTS streaming, and distributed storage for production-ready AI audio systems.

Introduction​

DeepSeek-R1 is a family of large language models (LLMs) known for advanced natural language capabilities. While hosting an LLM in the cloud can be convenient, local deployment provides greater control over latency, privacy, and resource utilization. Tools like Ollama simplify this process by handling model downloading and quantization. However, to truly scale or integrate these capabilities into other services, you often need a robust REST API layer—FastAPI is perfect for this.

This article covers the entire pipeline:

1. Installing and configuring Ollama to serve DeepSeek-R1 locally
2. Interacting with DeepSeek-R1 using the CLI, Python scripts, or a FastAPI endpoint for streaming responses
3. Demonstrating a minimal FastAPI integration, so you can easily wrap your model in a web service

By the end, you’ll see how to run DeepSeek-R1 locally while benefiting from FastAPI’s scalability, logging, and integration features—all without sending your data to external servers.

1. Why Run DeepSeek-R1 Locally?​

Running DeepSeek-R1 on your own machine has multiple advantages:

• Privacy & Security: No data is sent to third-party services
• Performance & Low Latency: Local inference avoids remote API calls
• Customization: Fine-tune or adjust inference parameters as needed
• No Rate Limits: In-house solution means no usage caps or unexpected cost spikes
• Offline Availability: Once downloaded, the model runs even without internet access

2. Setting Up DeepSeek-R1 Locally With Ollama​

2.1 Installing Ollama​

1. Download Ollama from the official website.
2. Install it on your machine, just like any application.

2.2 Download and Test DeepSeek-R1​

Ollama makes model retrieval simple:

ollama run deepseek-r1

This command automatically downloads DeepSeek-R1 (the default variant). If your hardware cannot handle the full 671B-parameter model, specify a smaller distilled version:

ollama run deepseek-r1:7b

2.3 Running DeepSeek-R1 in the Background​

To serve the model continuously (useful for external services like FastAPI):

ollama serve

By default, Ollama listens on http://localhost:11434.

3. Using DeepSeek-R1 Locally​

3.1 Command-Line (CLI) Inference​

You can chat directly with DeepSeek-R1 in your terminal:

ollama run deepseek-r1

Type a question or prompt; responses stream back in real time.

3.2 Accessing DeepSeek-R1 via API​

If you’re building an application, you can call Ollama’s REST API:

curl http://localhost:11434/api/chat -d '{
  "model": "deepseek-r1",
  "messages": [{ "role": "user", "content": "Solve: 25 * 25" }],
  "stream": false
}'

3.3 Python Integration​

Install the ollama Python package:

pip install ollama

Then use:

import ollama

response = ollama.chat(
    model="deepseek-r1",
    messages=[
        {"role": "user", "content": "Explain Newton's second law of motion"},
    ],
)
print(response["message"]["content"])

4. FastAPI Integration and Streaming Responses​

To wrap DeepSeek-R1 in a fully customizable FastAPI service, you can define streaming endpoints for advanced usage. Below is an example that sends chunked responses to the client:

import os
import json
from typing import List
from pydantic import BaseModel
from dotenv import load_dotenv
from fastapi import FastAPI, Query
from fastapi.responses import StreamingResponse
from openai import OpenAI

from .utils.prompt import ClientMessage, convert_to_openai_messages
from .utils.tools import get_current_weather  # example tool
from .utils.tools import available_tools  # hypothetical dict of tool funcs

load_dotenv(".env.local")

app = FastAPI()
client = OpenAI(api_key="ollama", base_url="http://localhost:11434/v1/")

class Request(BaseModel):
    messages: List[ClientMessage]

def stream_text(messages: List[ClientMessage], protocol: str = 'data'):
    stream = client.chat.completions.create(
        messages=messages,
        model="deepseek-r1",
        stream=True,
    )

    if protocol == 'text':
        for chunk in stream:
            for choice in chunk.choices:
                if choice.finish_reason == "stop":
                    break
                else:
                    yield "{text}".format(text=choice.delta.content)

    elif protocol == 'data':
        draft_tool_calls = []
        draft_tool_calls_index = -1

        for chunk in stream:
            for choice in chunk.choices:
                if choice.finish_reason == "stop":
                    continue
                elif choice.finish_reason == "tool_calls":
                    for tool_call in draft_tool_calls:
                        yield f'9:{{"toolCallId":"{tool_call["id"]}","toolName":"{tool_call["name"]}","args":{tool_call["arguments"]}}}\n'

                    for tool_call in draft_tool_calls:
                        tool_result = available_tools[tool_call["name"]](**json.loads(tool_call["arguments"]))
                        yield (
                            f'a:{{"toolCallId":"{tool_call["id"]}","toolName":"{tool_call["name"]}","args":{tool_call["arguments"]},'
                            f'"result":{json.dumps(tool_result)}}}\n'
                        )
                elif choice.delta.tool_calls:
                    for tool_call in choice.delta.tool_calls:
                        id = tool_call.id
                        name = tool_call.function.name
                        arguments = tool_call.function.arguments
                        if id is not None:
                            draft_tool_calls_index += 1
                            draft_tool_calls.append({"id": id, "name": name, "arguments": ""})
                        else:
                            draft_tool_calls[draft_tool_calls_index]["arguments"] += arguments
                else:
                    yield f'0:{json.dumps(choice.delta.content)}\n'

            # usage
            if chunk.choices == []:
                usage = chunk.usage
                prompt_tokens = usage.prompt_tokens
                completion_tokens = usage.completion_tokens
                yield (
                    f'd:{{"finishReason":"{"tool-calls" if len(draft_tool_calls) > 0 else "stop"}",'
                    f'"usage":{{"promptTokens":{prompt_tokens},"completionTokens":{completion_tokens}}}}}\n'
                )

@app.post("/api/chat")
async def handle_chat_data(request: Request, protocol: str = Query('data')):
    messages = request.messages
    openai_messages = convert_to_openai_messages(messages)
    response = StreamingResponse(stream_text(openai_messages, protocol))
    response.headers['x-vercel-ai-data-stream'] = 'v1'
    return response

Key Points:​

• stream=True allows the server to stream content chunk by chunk.
• The code handles optional “tool calls” logic—customizable for your own environment.
• FastAPI’s StreamingResponse ensures the client receives partial output in real time.

With this setup, you can embed DeepSeek-R1 into more complex microservices or orchestrate multi-step workflows that rely on streaming LLM responses.

6. Conclusion​

DeepSeek-R1 combined with Ollama and FastAPI gives you a powerful local LLM service. You can handle all aspects of data ingestion, retrieval, and inference in one place—without relying on third-party endpoints or paying subscription costs. Here’s a recap:

• Ollama manages downloading and serving the DeepSeek-R1 models.
• FastAPI provides a flexible web layer for streaming responses or building microservices.

Build your local AI solutions confidently and privately—DeepSeek-R1 is now at your fingertips.