Find the table and its cells
PyTorch models see layout and structure while preserving the coordinates needed downstream.
TATR TD→TATR TSR v1.1
62.5Mparameters across the shipped geometry track
5,353false annotations removed by the repair tool
16,000synthetic images used to train for reality
1stplace · MACHATHON 7.0
First place for Agent P-DF.
The competition result belongs to the system shown on this page: 62.5M parameters across the shipped geometry track, hard-negative training, annotation repair, deterministic reconstruction, and the correction studio.
Studio walkthrough · 0:48
AI is approximate. The editing environment is exacting. Every chapter below follows that split.
01 · System shape
Document page → editable table
Agent P-DF separates visual geometry, text extraction, and semantic reconstruction so each part can be inspected, benchmarked, and repaired independently.
Read what each region says
Fast ONNX inference turns OCR boxes into text lines without coupling recognition to the structure model.
MobileDet→MobileRec
Rebuild a useful table
Deterministic assignment, row rebuilding, heuristics, and the studio resolve the residual ambiguity.
Cells→CSV · XLSX · HTML
02 · Model and data decisions
Teaching the detector to say no
Early experiments made the choice clear. YOLO learned visual texture instead of document boundaries and introduced an unsuitable licensing constraint. The solution was a DETR-family geometry track plus data designed around the mistakes production documents actually make.
01
YOLO
Fast baseline; boxes words and texture as if they were tables.
02
Clean data
Beautiful public tables produce confident false positives.
03
Hard negatives
LaTeX-generated near-tables force aggressive rejection.
04
Mixed training
Clean and noisy samples are weighted together to resist forgetting.
05
Deployment track
TATR geometry, ONNX OCR, and deterministic post-processing ship together.
The useful metric
Sanitized-only training reached an attractive AP 0.93 while hallucinating tables on non-tables. Adversarial mixing settled around AP 0.89 with the false-positive behavior needed in production. The model learned that “not a table” is a first-class answer.
Open the training protocol
The mixed distribution combines public table corpora with roughly 16,000 synthetic images. A weighted sampler oversamples the minority hard-negative cases, spatial crops translate coordinates so the model cannot memorize page margins, and the lower ResNet is frozen while later layers use differential learning rates. The detector loss falls from 0.61 to 0.17, with AP50 approaching 0.97 on the recorded run.
03 · Data surgery
When the labels poison the model
Structure recognition was the hardest part of the project because the dataset looked plausible until the model exposed its contradictions. A validation collapse to AP 0.477 led to an audit of more than sixty pages: the projected-row-header class had been sprayed across real content, column headers were missing, and spanning cells were inconsistent.
Failure signatureAP 0.477
Exploding decoder gradients were a symptom. The root cause was annotation geometry that taught the model to see a dense field of phantom row headers.
Repair tool
Make the data failure visible
I built an AI-assisted annotation editor with continuous scroll, span repair, row-header conversion, previews, and bulk actions. The repair pass targeted pages 82–154 and removed exactly the incorrect annotations that had been poisoning training.
5,353false annotations
removed in one repair pass
removed in one repair pass
The ablation was decisive: dropping the corrupted class produced an immediate AP jump. The fix was a change in representation, not a larger model.
04 · Structure recovery
Escaping the 125-query ceiling
A dense table can generate more competing row-header hypotheses than the TATR decoder's 125-query budget can represent. Instead of asking the model to solve every semantic label at once, I moved the ambiguous decision downstream and made geometry deliberately unambiguous.
01 · Drop the trap
Remove projected-row-header from training and map the visual case to ordinary rows.
02 · Build a dumb grid
Force a dense, stable grid that cannot spend queries on phantom structure.
03 · Restore meaning
RapidOCR and deterministic heuristics re-flag genuine full-width headers after the grid exists.
This is a recurring design pattern in the system: let learned models handle geometry, then use constrained code where the business meaning must be predictable.
05 · Cell reconstruction
From OCR boxes to actual cells
Nearest-center matching breaks as soon as a line crosses a boundary or a cell spans columns. Each OCR box is scored against every candidate cell using overlap, intersection-over-area, and an inside-center bonus, with a normalized-distance fallback and a text-anchored row rebuild when structure boundaries disagree with text.
score(b, c) = IoA(b, c) + 0.20 × IoU(b, c) + 0.35 × 1[center(b) ∈ c]
Open the assignment rules
Assignment accepts a candidate at IoA ≥ 0.35 or IoU ≥ 0.02, then falls back to a capped normalized cell-center distance. The row rebuild uses OCR line gaps to re-anchor structure, while production heuristics merge left spanners, prune conflicts, and perform weighted assignment. This keeps the output deterministic even when the model's grid is imperfect.
06 · Auxiliary research track
TableFormerV2: freeze the vision, adapt the task
TableFormerV2 was a separate experiment, not part of the shipped 62.5M-parameter geometry total. I modified its training path to combine OTSL sequence supervision with direct cell-box supervision, while freezing the lower visual representation and adapting the task-specific heads.
frozen forward path trainable adaptation supervision / loss
Visual backbonefeature extractor · squeeze-excitation · convolution mixer · feature-to-embedding prefixes
→
Adapted TableFormerV2OTSL token head + bounding-box head · 26.4M trainable of 52.244M total
→
Two signalsstructure tokens and cell geometry are checked together
Sequence supervisionteacher-forced OTSL cross-entropy with padding masked
Box supervisionL1 + GIoU on cell-bearing tags, with predicted counts checked against targets
L = Lseq, CE + 5Lbox, L1 + 2 mean(1 − GIoU)
Why this matters
Freezing is visible here as a forward-path decision: frozen modules still transform the image, but their weights do not update. The trainable heads are the part adapted to the annotation policy. This is the same architectural judgment as the detector's surgical freezing, expressed in a different model family.
07 · Product surface
The correction studio is part of the model
A pipeline that cannot expose its uncertainty pushes the cost onto the user. The studio makes the remaining work legible: select regions, inspect confidence and heatmaps, repair cells, and export an artifact people can continue using.
InspectDocument, structure grid, text, confidence, and heatmap views stay side by side.
CorrectMarquee selection, cell edits, row and span operations, and deterministic rebuilds absorb residual errors.
ExportJSON, CSV, XLSX, and HTML preserve the extracted structure for the next tool.
Keep the operator in controlHandwritten, degraded, and exotic layouts can be flagged for human review instead of receiving a false promise.
Multi-page jobsGlobal IDs, ordered pages, and ZIP-ready outputs keep long documents coherent.
Runtime disciplinePrewarming, device locks, semaphores, worker limits, and cleanup protect constrained deployments.
Health you can readWarm-up, p50/p95 timing, queue state, and stage-level failures stay visible to the operator.
Limitations are part of the interface contract: handwritten tables, heavily degraded scans, and unusual layouts still need review. The system is designed to make that review efficient and precise.
Artifact shelf
Go deeper when the question is technical
The case study is the guided path. These are the complete artifacts for a reviewer who wants to inspect the work as a deliverable.