Context

In a vertical CRM for the financial sector, sensitive data, tight operational flows, and operators who cannot afford manual field-by-field data entry coexist. Hence the need for a tool entirely in-house (without relying on third-party services) capable of reading documents — identity and tax documents such as 730 forms, pay slips, tax ID cards/codes, and similar — and automatically extracting the useful information to pre-populate CRM records:

• for assisted data entry, with the operator in front of the screen
• in bulk mode, to process hundreds or thousands of documents in a single cycle

Objectives: speed, accuracy, traceability and, crucially, privacy by design: the data remains within the company's domain.

The system has been internally validated on N clients, covering a wide variety of formats, layouts, and source qualities.

The challenge

In the document world, "standard" is the exception. Payslips seem similar until you compare them: different layouts, variable headers, migrating sections, non-uniform wording. Tax forms (730s) follow the same pattern: recognizable forms, but constant variations.

Among them are:

• Native PDFs (searchable)
• scans borderline: crooked, grainy, with stamps or annotations covering figures and fields

The engine must recognize what it's facing, choose the most suitable strategy, and handle edge cases without degrading workflows. All while maintaining latencies suitable for back-office use and with an error rate that doesn't turn automation into double work.

Approach

No shortcuts: Methodical R&D, data engineering, and modular architecture. The system is an orchestrated ensemble: multiple extractors, multiple rules, a decision-making logic that selects the best pipeline based on the document and context.

1) Understand what document it is

Before extraction, it is classified:

• a first level identifies the family (e.g., 730 form, payslip, ID document, tax code/card, etc.)
• a second level distinguishes the variant (including "creative" ones, and outdated formats)

The classification determines the subsequent workflow.

2) Choose the shortest (and most reliable) path

• If it is a searchable PDF, textual parsing + positional heuristics are used (known coordinates, established patterns).
• If it is scanned, it goes through OCR and then models for key-value detection and normalization.
• In both cases, semantic rules are applied to handle noise and ambiguity: anomalous numerical separators, equivalent wordings, O/0 in tax codes, "totals" that aren't truly totals, etc.

3) Orchestrate the "brains" (ensemble)

Each document family has multiple extractors:

• some fast and deterministic
• others slower but more robust

An orchestration logic decides which one to use and how much to trust the results.
At the field level, a confidence score is calculated and cross-consistency checks are applied, for example:

• if a tax code implies an implausible date of birth, the field is discarded or marked as uncertain
• if the figures for gross, deductions, and net don't reconcile, it is recalculated or confirmation is requested

4) Human-in-the-loop, only where needed

The operator finds the pre-filled form with the less certain fields highlighted. Corrections (when necessary) fuel a continuous improvement cycle: golden set, periodic retraining, new targeted rules. We don't "train everything all the time": improvement happens where the data justifies it.
Operational architecture

The engine is designed to reside within the CRM's perimeter:

• Internal APIs for the assisted real-time
• scheduled jobs for bulk imports

The pipeline is idempotent: the same document does not produce duplicates, but versions and audits.
Every step is logged (classification, extraction, validations, handled errors) for monitoring and diagnostics: if a new variant starts circulating, it's visible from the score drift and action is taken.

Security:

• encryption in transit and at rest
• role-based access control
• no external calls with PII
  Guiding principle: everything in-house.

Impact on daily work

The change is immediate: the operator no longer starts from scratch. Crucial fields arrive pre-filled, and anything uncertain is flagged.
Result: less data entry, fewer errors, fewer repeated checks.

In bulk, the nightmare of 'upload today, check tomorrow' disappears: extraction runs, tracks, aligns, and makes the entire process auditable.

Why an ensemble

Because documents change, quality fluctuates, and edge cases are not uncommon. A well-orchestrated ensemble allows for incremental evolution: you add an extractor, modify a decision branch, without rewriting everything. It's maintainable and scalable.

R&D: what has been done

• collection and normalization of samples (anonymized)
• definition of field taxonomies per document family
• design of specific feature extractors
• Building golden sets for regression testing
• Field-level metrics (precision/recall per data point)
• Cross-document consistency rules (multiple files must align)
• Stakeholder-focused KPIs: time saved, outstanding corrections, improvement priorities

Result

A production system that:

• recognizes the document type
• selects the best approach
• extracts data and pre-fills the CRM in assisted and bulk mode
• applies quality controls, logging, and auditing
• can be extended to new document types without re-engineering the system

It doesn't promise "zero errors". It promises a platform that improves over time, doesn't disrupt workflows, and respects data.