Computer vision papers are written for other researchers. That is a shame, because the ideas are simple and the business value is concrete — but a founder Googling "U-Net vs DeepLab" mostly finds benchmark tables. This guide takes a different route: every major model, in the order the field invented them, with the one idea that made each matter and the jobs it is actually the right tool for.
The single most useful thing to understand is that vision models are grouped by the question they answer:
Match the question to your business problem and half the model-selection work is done. "Is this product photo acceptable?" is classification. "How many cars are in the lot?" is detection. "Exactly how much of this roof is damaged?" is segmentation. Let's meet the families.
Part 1 — The Classifiers: Backbones That See "What"
AlexNet (2012) is where the modern era starts: the deep convolutional network that won the ImageNet challenge by a margin so large it converted the entire field overnight. Eight layers, trained on GPUs, and the recipe — convolutions, ReLU activations, dropout — became the grammar every later model speaks. You would never deploy AlexNet today, but every model on this page is its descendant.
VGG (2014) proved that simple depth works: stack small 3×3 convolutions sixteen or nineteen layers deep and accuracy keeps climbing. VGG is heavy by modern standards, but its clean uniform design made it the default feature extractor for years, and "VGG-style" is still shorthand for straightforward stacked convolutions.
GoogLeNet / Inception (2014) asked a smarter question: why choose one filter size? Its Inception blocks run 1×1, 3×3 and 5×5 convolutions in parallel and concatenate the results, capturing fine and coarse patterns at once with far fewer parameters than VGG.
ResNet (2015) is the most important architecture of the decade. Before it, very deep networks trained worse — the signal degraded on its way down. ResNet's fix is the skip connection: let each block learn only the change it wants to make, and pipe the input straight through alongside it. Suddenly 50-, 101-, 152-layer networks trained cleanly, and ResNet-50 remains the workhorse backbone inside detection and segmentation systems to this day.
DenseNet (2017) pushed connectivity to the limit — every layer receives the outputs of all previous layers — squeezing more accuracy per parameter. MobileNet (2017) went the other direction: depthwise-separable convolutions cut compute by an order of magnitude so vision could run on phones, shop-floor cameras and Raspberry Pis. EfficientNet (2019) then made sizing scientific, scaling depth, width and resolution together along a single dial, so you pick the accuracy/latency trade-off like choosing a t-shirt size.
Vision Transformer — ViT (2020) broke the convolution monopoly. It slices an image into 16×16 patches, treats them like words in a sentence, and runs a standard transformer over them. With enough pre-training data, global attention beats sliding filters. Swin Transformer (2021) made transformers practical for dense tasks by computing attention in shifted local windows, giving the hierarchy of a CNN with the flexibility of attention — and became a top backbone for detection and segmentation.
Any yes/no or which-one decision at volume: product photo QA before a listing goes live, room-type and amenity tagging for hotel and real-estate media libraries, damaged-vs-fine triage on returns, brand-safety screening for user uploads. Classifiers are the cheapest vision models to train and run — if your problem can be phrased as "sort these images into buckets," start here, usually with a fine-tuned ResNet or EfficientNet, or MobileNet on edge hardware.
Part 2 — The Detectors: Finding "Where"
Detection adds coordinates to the answer: not just "there's a forklift" but a box around each one. Two philosophies dominate.
The R-CNN line (2014–2015) is the careful, two-stage philosophy. R-CNN cropped ~2,000 region proposals and classified each with a CNN — accurate, painfully slow. Fast R-CNN ran the CNN once and cropped features instead of pixels. Faster R-CNN (2015) completed the idea by making the proposals themselves a tiny neural network (the Region Proposal Network) sharing the same backbone — the first end-to-end learned detector, and still the reference for accuracy-first workloads.
YOLO (2016 onward) is the fast, one-stage philosophy: You Only Look Once. Divide the image into a grid; every cell predicts boxes and classes directly, in a single forward pass. The first version traded accuracy for real-time speed; a decade of successors closed the accuracy gap while keeping the speed, which is why some flavor of YOLO now sits behind most live camera feeds in industry. SSD (2016) applied the same single-shot idea across multi-scale feature maps; RetinaNet (2017) fixed one-stage detectors' core weakness — a million background boxes drowning out rare objects — with focal loss, which simply makes easy negatives cheap and hard examples expensive.
DETR (2020) rethought detection as a translation problem: a transformer reads the image and directly outputs a set of objects, no hand-tuned anchors, no non-maximum suppression cleanup. It made detectors dramatically simpler, and its descendants power the open-vocabulary systems in Part 4.
Counting and compliance: retail shelf audits (which products, which facings, which gaps), vehicle counting for parking and drive-through analytics, PPE and safety-zone compliance on job sites, package and pallet tracking in warehouses, table-occupancy for restaurants and hotels. Rule of thumb: live video or edge hardware → YOLO-family; offline accuracy on hard scenes → a two-stage or transformer detector. This is the layer most of our vision systems and anomaly monitors are built on.
Part 3 — The Segmenters: Deciding Pixel by Pixel
Sometimes a box isn't enough — you need the exact outline. Billing depends on square footage of damage, not on "damage somewhere in this rectangle."
FCN (2015) opened the era by making classification networks fully convolutional: swap the final flat layers for convolutions and upsample back to image size, so the network outputs a class for every pixel. Coarse by today's standards, but every segmenter since inherits its skeleton.
U-Net (2015) is the beloved one. Built for biomedical images where labeled data is scarce, it pairs a shrinking encoder path with a mirror-image expanding decoder path — and copies feature maps across at every resolution, so the decoder recovers the fine edges the encoder compressed away. It is simple, trains well on small datasets with heavy augmentation, and remains the default answer for "precise masks, limited data" — from tumor boundaries to roof damage to document layout.
SegNet (2017) offered a memory-light variant — remembering only which pixel won each pooling step and unpooling accordingly. PSPNet (2017) attacked context instead: its pyramid pooling module summarizes the scene at several scales at once, so a pixel can be labeled "boat" partly because the surrounding context says "harbor."
DeepLab (2015–2018) is the other pillar of the segmentation world. Its signature move is atrous (dilated) convolution — spreading a filter's taps apart to widen its field of view without adding parameters or shrinking the map. Version by version it added Atrous Spatial Pyramid Pooling (parallel dilated filters at multiple rates, reading context at several zoom levels simultaneously) and, in DeepLabv3+ (2018), a light U-Net-style decoder for crisp edges. When you have plenty of data and varied scenes — streetscapes, aerial imagery, property exteriors — DeepLabv3+ is the industrial-strength choice.
Mask R-CNN (2017) merged the detector and segmenter worlds: take Faster R-CNN, add a small third head that paints a mask inside every detected box (plus a sub-pixel alignment fix, RoIAlign, that makes the masks clean). The result is instance segmentation — separate outlines for car #1, #2 and #3 — and it remains the go-to when you need to both count things and measure them.
Anywhere the area or outline is the money number: roof and siding damage assessment from drone photos (insurance-grade square footage, not vibes), background removal and sky replacement for listing photos at scale, floor-plan extraction, produce/defect grading by surface coverage, and medical or lab imagery. U-Net when data is scarce and masks must be precise; DeepLabv3+ when scenes are varied and data is plentiful; Mask R-CNN when instances must be counted and outlined.
Part 4 — The Foundation Era: Vision Models That Take Instructions
Everything above needs training on your labeled categories. The newest generation often doesn't.
CLIP (2021) learned vision from 400 million image–caption pairs by pulling matching images and texts together in a shared space. The payoff is zero-shot classification: describe your categories in plain English — "water damage," "mold," "cosmetic scuff" — and CLIP scores images against them with no training run at all. Accuracy won't match a fine-tuned specialist, but as a v1 classifier or a data-labeling accelerator it is absurdly cost-effective.
SAM — Segment Anything (2023) did for masks what CLIP did for labels. Trained on over a billion masks, it segments whatever you point at — a click, a box, or another model's detection — with no task-specific training. SAM 2 (2024) extended it to video, tracking masks across frames. In practice SAM is the great label-cost destroyer: teams use it to produce in hours the pixel-perfect training masks that used to take annotation farms weeks.
Grounding DINO (2023) closes the loop: an open-vocabulary detector that finds objects from a text phrase — "exposed wiring," "missing shingle," "person without hard hat" — without ever being trained on those classes. Chain it with SAM (text → boxes → masks) and you have a vision pipeline that goes from an English sentence to pixel-accurate masks with zero labeled data. For prototyping, that changes the economics of the entire field: what used to be a three-month labeling project is now a weekend proof of concept.
Speed to first value: prototype any vision idea before committing to data collection, auto-label datasets for the specialist model you'll train later, handle the long tail of rare categories no one has labels for, and build "click to select" tooling for internal review apps. This is why our prototypes can exist in days: the foundation models do the cold start, and a fine-tuned specialist takes over once the value is proven.
The Chooser: Match the Model to the Job
| You need | Start with | Step up to | Notes |
|---|---|---|---|
| Sort images into buckets | Fine-tuned ResNet-50 / EfficientNet | ViT if data is plentiful | MobileNet on edge devices; CLIP zero-shot to prototype |
| Find & count objects, live video | YOLO family | RetinaNet / DETR variants | Speed is YOLO's home turf |
| Find objects, max accuracy offline | Faster R-CNN | Transformer detectors | Two-stage still wins hard scenes |
| Exact outlines, little training data | U-Net | U-Net + modern backbone | The small-data champion since 2015 |
| Exact outlines, complex scenes | DeepLabv3+ | Transformer segmenters | ASPP handles objects at many scales |
| Count and outline instances | Mask R-CNN | — | Detection + masks in one model |
| No labels, need results this week | CLIP / Grounding DINO + SAM | Distill into a specialist later | Prototype-grade today, label factory forever |
The expensive mistake isn't picking the wrong architecture — they're all a config change apart. It's labeling ten thousand images for a problem a zero-shot model could have validated in a weekend.
How We Actually Choose, in Practice
After all the papers, model selection in a real project comes down to four questions. What's the question? — classify, detect, or segment (see the first diagram; this eliminates 80% of the menu). What's the latency budget? — live video pushes you one-stage and mobile-sized; overnight batch frees you to buy accuracy. What data exists? — nothing yet means foundation models first, a few hundred images means U-Net-style transfer learning, tens of thousands means the industrial options open up. Where does it run? — a camera box on a wall is a MobileNet/YOLO conversation; a cloud pipeline can afford transformers.
Notice what's not on that list: training anything from scratch. In 2026 virtually every production system starts from pretrained weights and fine-tunes — the architectures in this guide are ingredients you adapt, not experiments you rerun.
If you're staring at a camera feed, a photo library, or a drone folder and wondering which of these families your problem belongs to, that's literally the first thing we work out in a strategy call — and our computer vision service page shows what the resulting systems look like in production.