Document Ingestion & Embedding Pipeline

Problem

Enterprise corpora change constantly. Ingestion had to stay current without reprocessing everything, and failures needed safe retries.

Solution

Scheduled Container App Jobs pull from SharePoint, ADI, and Blob Storage, generate embeddings, and apply delta/retry logic for resilient updates.

Technology Stack

Learnings

Delta detection and retry semantics matter as much as the embedding model when the corpus is large and continuously changing.