Workspace-Scoped Ingestion Profiles

Goal

Keep the canonical Nexus ingestion pipeline unchanged globally, while allowing selected workspaces to opt into:

• a stronger Docling profile for heavy visual manuals
• richer stage tracking
• heartbeat refreshes during long Docling stages
• workspace-specific watchdog thresholds

This is intended for corpora such as automotive repair manuals without turning Nexus itself into a verticalized product.

Workspace Configuration

workspaces_config now supports these nullable fields:

• doc_ingestion_profile
• doc_stage_tracking_enabled
• doc_heartbeat_enabled
• ingestion_warn_after_seconds_override
• ingestion_kill_after_seconds_override

Supported profile values:

• standard
• heavy_visual_manual

If the fields are left unset, Nexus keeps its previous generic behavior.

Runtime Behavior

For heavy_visual_manual, the ingestion worker resolves a stronger effective Docling configuration at job runtime:

• OCR enabled
• table structure enabled
• picture description enabled via OpenAI-compatible remote runtime
• picture classification disabled
• formula enrichment disabled
• page image generation enabled

Picture classification intentionally stays disabled for this profile. It is not required for the automotive retrieval workflow and has shown unstable runtime behavior on heavy repair PDFs. Picture description is intentionally scoped to the heavy profile only. Standard workspaces keep the existing local Docling defaults and do not incur external vision calls.

The worker still uses the same canonical ingestion flow:

1. download
2. docling
3. artifacts upload
4. chunking
5. embeddings
6. vector upsert
7. persist

There is no second business pipeline.

Heartbeat And Stages

The worker already stored coarse stages in ingestion_jobs.metrics. The workspace profile now allows a richer mode where:

• elapsed_ms is refreshed during the job
• last_heartbeat_at_epoch is refreshed during long Docling phases
• progress_current / progress_total can be stored when available
• effective watchdog thresholds are copied into job metrics

This lets the watchdog apply workspace-specific thresholds per job instead of only using global defaults.

API Impact

GET /api/ingest/batches/{batch_id} now exposes a summary block derived from the preferred current job, including:

• current_job_id
• current_job_status
• stage
• last_heartbeat_at_epoch
• elapsed_ms
• progress_current
• progress_total
• doc_ingestion_profile

GET /api/ingest/jobs/{job_id} also exposes the same normalized metric fields at top level in addition to raw metrics.

Recommended Usage

For the automotive demo workspace:

• keep all other workspaces on standard
• set repair_information to heavy_visual_manual
• enable stage tracking
• enable heartbeat
• tune watchdog thresholds only if the corpus really needs it

This keeps Nexus generic while making long-running visual PDF ingestion observable where needed.