To interpret and understand uploaded build plans, TakeoffQS uses a combination of traditional computer vision techniques, purpose-trained deep learning models, and vision language models.
These are designed to handle the variability in real-world plans, where symbols, annotations, and layouts differ significantly between projects and between drafting offices. Rather than relying on a fixed rule set, the system learns patterns directly from training data, then applies geometric and domain checks before producing anything a user sees.
This is an update to a post we first published in January. The pipeline has changed enough since then that it was worth rewriting rather than patching.
System overview
At a high level, TakeoffQS processes construction drawings in five stages:
Page classification → Detection & segmentation → Reading the drawing → Geometry & rules → Verification
Each stage addresses a different category of problem. Learned models are used where visual interpretation and ambiguity dominate — deciding what a page is, finding a wall, reading a handwritten-looking dimension string. Classical geometry and explicit rules are used where the answer is determined rather than inferred — turning a wall polygon into a length, a roof outline into an area, a set of segments into a perimeter.
The division matters. A model that is 95% right about where a wall is, followed by arithmetic that is exactly right about how long it is, produces a traceable number. A model asked to output the number directly does not.
Page classification
A typical plan set mixes architectural, structural, mechanical and reference drawings, plus title pages, schedules and specifications. Almost nothing downstream works until you know what each page is.
Classification runs first, and today it is handled by a vision language model reading the rendered page. It works on visual content rather than file metadata, sheet naming conventions or drafting-office title blocks — those vary too much between practices to rely on, and are frequently wrong or absent.
The output routes each page to the appropriate downstream processing. A foundation plan and an elevation get entirely different treatment, so getting this stage right removes a large amount of wasted computation and a larger amount of downstream failure.
We’re currently bringing a second, much smaller classifier alongside it — a linear model trained on the text extracted from each page, running locally inside our own service. It’s not more capable than the vision model; it’s faster and cheaper on the pages where the answer is obvious, and it abstains rather than guesses. Where it abstains, or where its confidence sits below a per-page-type threshold we set from measured precision, the page falls through to the vision model as before. It runs in shadow first: a page type only starts answering on its own once its agreement with the vision model has been measured on real traffic and clears the bar we set for that type.
That pattern — a cheap model that is allowed to say “I don’t know”, backed by an expensive one that always answers — is something we now use in several places.
Detection and segmentation
Once a page is classified, TakeoffQS dispatches a specific set of detectors for that page type. There is no single general-purpose model. We run thirteen detector families, each trained on its own labelled corpus for one job: external walls, internal walls, roof, foundations, foundation beams, elevations, wall heights, bracing, floor openings, mid-floor, soffits, subfloor piles, subfloor structure.
Page-type routing means a floor plan gets the wall and opening detectors, a foundation plan gets pods, beams and piles, and neither pays for the other’s inference.
The families currently run on two architectures. Five are RF-DETR-Seg, a transformer-based segmentation model; eight are YOLO variants. That split is historical rather than deliberate, and we’re consolidating it — more on that below.
For example, on a foundation plan, the segmentation layer detects structural elements including pods and beams. These are often tightly packed and visually similar, with differences in line weight, scale and annotation depending on drafting standards. The model segments individual pods as repeated structural units and identifies beams as elongated connecting elements. Geometric algorithms are then applied to derive ribs and to classify each beam type.
Detector output is an intermediate representation, not a result. Nothing a detector produces reaches a user without passing through the geometry and rules layer.
Reading the drawing
Finding shapes is only half the problem. A plan carries a large amount of information in text and symbols that a segmentation model is the wrong tool for: dimension strings, bracing codes and their schedules, roof pitch callouts, fall arrows, downpipe symbols, room names, and the drawing scale itself.
We use vision language models for these, but narrowly — cropped to the relevant region of the page at high resolution, prompted for one specific reading, and returned as structured data that is validated against the schema before it is accepted. A pitch that doesn’t parse as a pitch is discarded, not guessed at.
This is deliberately not “hand the whole plan to a language model and ask for a takeoff”. The failure mode of that approach is confident, plausible, wrong numbers with no traceable origin. Scoping each call to one small question against one small crop keeps the output checkable.
Geometry and rules
This is where detections become quantities, and it’s the least glamorous and most load-bearing part of the system.
The rules layer computes floor area from wall geometry, external wall outline lengths, cladding runs including regeneration of segments hidden behind other elements, lintel sizing and garage type, pitch factors and true roof areas, and room-level wall and paint areas. These are explicit computations over the geometry the detectors produced, at a known scale, with domain constraints applied.
They also cross-check. A quantity that can be derived two ways gets derived two ways, and a disagreement is surfaced rather than averaged.
Where it runs, and who trains it
Through the first half of this year we’ve moved our model training and hosting onto our own infrastructure, and off third-party hosted training and inference.
There were two reasons. Control over training — being able to retrain a family when we get better data, rather than when a vendor’s platform allows it. And durability — our full training corpus, every dataset version and every trained checkpoint, is now archived in our own storage, so no model we depend on can become unretrainable because a third party changed their retention policy. That archive is not hypothetical insurance; when we pulled it, we found a vendor had already deleted most of the version history for one of our projects.
All thirteen families have been ported and parity-tested against their existing behaviour on our own infrastructure. Production is mid-cutover.
Alongside that, we’ve built our own annotation platform for producing training data — the labelled plan corpus that everything above depends on. Owning the annotation step matters more than it sounds: it’s the part of the pipeline that determines the ceiling on every model downstream, and it is the part most tied up in a vendor if you don’t own it.
What we’re changing next
Two pieces of work are in progress. Both are honest works-in-progress rather than shipped features, and we’d rather describe them that way.
Consolidating onto one architecture. The eight YOLO families are being retrained onto RF-DETR-Seg, the architecture the other five already use. The evidence that this is safe comes from a direct comparison: two of our families are trained on the same dataset, one on each architecture, and they score within a point of each other. Architecture is not what’s limiting us — sample count is. Consolidating gets us to a single serving stack and a single training path, which makes every subsequent improvement cheaper to apply across the fleet.
Feeding the model the geometry it’s currently throwing away. Today our detectors read plans as photographs. We rasterise a vector PDF to an image and ask a model to recover line work from pixels, after discarding the exact line geometry the PDF gave us for free. At the resolution we work at, a wall is one to two pixels wide; as a vector primitive, it’s exact.
The obvious shortcut — read the geometry directly and skip the model — doesn’t survive contact with real plans, and we’ve tested that against our own corpus rather than assuming it. Making that geometry available to the model instead is the work we’re in the middle of.
What this doesn’t change
None of the above changes the fundamental position we’ve written about elsewhere: TakeoffQS produces a draft takeoff for supported residential scopes, which a qualified professional verifies, corrects and takes responsibility for.
Better models make the first draft closer and the verification faster. They don’t remove the verification step, and we’re not building toward a version where they do.