How AI Sees the World: Computer Vision for Beginners (Part 7)

AI Fundamentals Series · Part 7 of 10 — Previous: Part 6: Natural Language Processing — Next: Part 8: Generative AI Explained

The Surprisingly Hard Problem of Seeing

Human vision feels effortless. You glance at a crowded photograph and instantly identify every person, object, and scene element. You recognize a friend's face from across the street, in poor lighting, at an unusual angle. You understand that a blurry white shape in a dark image is probably a dog running. All of this happens in milliseconds, without conscious effort.

For decades, making computers do the same thing was considered one of the hardest problems in AI. In the 1960s, MIT professor Marvin Minsky assigned a student to “solve” computer vision over a summer. That student did not solve it, and neither did thousands of researchers working for the next fifty years. Then, in 2012, a neural network called AlexNet (introduced in Part 2) demolished the previous state of the art, and the field was transformed overnight.

This article explains how modern computer vision works and why it matters for so many real-world applications.

Images as Numbers: Pixels

Before a neural network can “see,” an image must be converted to numbers. Fortunately, images are already fundamentally numerical.

A digital image is a grid of pixels. Each pixel stores a color value. For a grayscale image, each pixel has one number from 0 (black) to 255 (white). For a color image, each pixel stores three numbers — the intensity of red, green, and blue (the RGB channels) — each ranging from 0 to 255.

A 640×480 color image therefore contains 640 × 480 × 3 = 921,600 numbers. A 4K image contains over 24 million numbers. These raw numbers are the raw inputs to a computer vision model. The challenge is to extract meaning from those millions of pixel values.

Why Simple Neural Networks Struggle with Images

You might think: we already know how neural networks work from Part 5, so why not just feed all the pixel values into a standard fully connected neural network?

The problem is scale and structure. A standard network applied to a 640×480 image would need nearly a million input neurons, and then connecting each of those to a large hidden layer would require billions of parameters — an unmanageable quantity that would overfit badly and be computationally prohibitive to train.

Worse, a standard network has no built-in notion of spatial structure. It treats pixel at position (100, 100) as completely unrelated to the pixel at (101, 100), even though those adjacent pixels are almost always closely related. Images have strong local structure: nearby pixels tend to be part of the same object or texture. A good vision model should exploit that structure.

Convolutional Neural Networks: Seeing in Patches

The solution is the Convolutional Neural Network (CNN), a neural network architecture specifically designed for image data. CNNs introduce two key ideas:

Filters (Kernels)

Instead of connecting every neuron to every input pixel, a CNN applies small grids of weights called filters (or kernels) — typically 3×3 or 5×5 pixels — that slide across the image. At each position, the filter computes a weighted sum of the pixels it covers and produces a single output value. This operation is called a convolution.

One filter might learn to detect horizontal edges. Another might detect vertical edges. Another might detect red-green color contrasts. The network learns which filters are useful by adjusting their weights during training, just like it adjusts all other weights.

The crucial insight is weight sharing: the same filter is applied across the entire image. A filter that detects a cat's ear in the top-left corner can also detect it in the bottom-right corner using exactly the same weights. This dramatically reduces the number of parameters compared to a fully connected network and makes the model translation-invariant — it can recognize a cat regardless of where in the image the cat appears.

Pooling

After each convolutional layer, a pooling layer reduces the spatial dimensions of the data. Max pooling, the most common type, divides the output into small patches and keeps only the maximum value from each patch. This makes the representation smaller and more robust to small shifts in object position.

Hierarchical Feature Learning

Stacking multiple convolutional and pooling layers creates a hierarchy of increasingly abstract features:

• Early layers: detect edges, color gradients, simple textures
• Middle layers: combine those into shapes, curves, patterns
• Deep layers: recognize object parts (wheels, eyes, windows) and eventually entire objects

This hierarchical structure closely mirrors how neuroscientists believe the human visual cortex processes images — though the analogy should not be pushed too far.

Major Computer Vision Tasks

Image Classification

The simplest vision task: given an image, assign it a single label from a fixed set. “This image contains a golden retriever.” Systems like those competing in ImageNet perform this task with superhuman accuracy on standard benchmarks. Applications include content moderation, product categorization in e-commerce, and medical image screening.

Object Detection

Image classification gives one label per image. Object detection goes further: it localizes every instance of every object, drawing a bounding box around each one and labeling it. “There is a car at coordinates (120, 45, 380, 200), a pedestrian at (50, 100, 110, 310), and a traffic light at (200, 10, 240, 80).”

Models like YOLO (You Only Look Once) and Faster R-CNN perform this in real time. Applications include autonomous vehicles, security cameras, sports analytics, and warehouse robotics.

Image Segmentation

Segmentation goes further still: instead of drawing boxes, it labels every pixel with the class of object it belongs to. Semantic segmentation labels all road pixels as “road,” all sky pixels as “sky,” etc. Instance segmentation distinguishes between individual instances — pedestrian 1, pedestrian 2, pedestrian 3 — rather than just the class “pedestrian.” Medical image segmentation can outline tumor boundaries with precision exceeding that of many radiologists.

Face Recognition

Face recognition first detects a face in an image, then represents it as a numerical embedding (similar to word embeddings in NLP, but for faces), and then compares that embedding to a database to find a match. Modern face recognition systems achieve accuracy above 99.7% on standard benchmarks — comfortably better than most humans.

This capability has powerful beneficial uses (unlocking your phone, finding missing persons) and serious risks (mass surveillance without consent). We discuss these in Part 9.

Medical Imaging

Computer vision is producing some of AI's most significant medical impacts:

• Detecting diabetic retinopathy in eye images at accuracy matching ophthalmologists
• Classifying skin lesions as benign or malignant from photographs
• Identifying pneumonia, COVID-19, and other conditions from chest X-rays
• Segmenting tumors in MRI scans for surgical planning

In high-income countries, these tools augment specialists who are already stretched thin. In low-income settings, they can provide screening capabilities that would otherwise not exist at all.

Video Understanding: Seeing Motion

Still images are only part of the visual world. Extending computer vision to video requires understanding not just what objects are present in a frame, but how they move, interact, and change over time. Video understanding adds a temporal dimension to the spatial challenges of image processing.

Key video AI tasks include:

• Action recognition: identifying what activity is happening in a video clip (walking, jumping, cooking, playing an instrument). Used in sports analytics, security monitoring, and video content tagging.
• Video object tracking: following specific objects across frames as they move, partially disappear behind other objects, and reappear. Critical for autonomous vehicles, surveillance, and augmented reality.
• Video generation: creating coherent, temporally consistent video from text prompts or still images. Systems like OpenAI's Sora demonstrated this capability publicly in early 2024, generating minute-long realistic video clips from text descriptions.

Video data is vastly larger than image data — a one-hour video at 30 frames per second contains 108,000 individual images — so video models must process data extremely efficiently while capturing temporal relationships between frames.

The Ethical Dimensions of Computer Vision

Computer vision is one of the AI modalities with the most immediate civil liberties implications, because it enables identification and tracking of people in the physical world. Several ethical issues deserve attention:

• Consent: surveillance cameras, facial recognition at events, and commercial tracking systems can operate without the knowledge or consent of the people being observed. Unlike interacting with a chatbot, you cannot opt out of being in public space.
• Accuracy disparities: multiple studies have documented that commercial facial recognition systems perform significantly worse on darker-skinned faces and on women than on lighter-skinned male faces. If these systems are used in law enforcement, the groups with the highest error rates face the highest risk of misidentification and wrongful action.
• Chilling effects: even imperfect surveillance can suppress lawful behavior. People may avoid protests, religious gatherings, or other constitutionally protected activities if they believe they are being continuously identified and tracked.
• Beneficial uses: these concerns must be weighed against genuine benefits. Computer vision tools that detect diabetic retinopathy in low-resource settings, identify missing children, or assist visually impaired people navigate their environment represent significant positive value that deserves equal weight in policy discussions.

Beyond CNNs: Vision Transformers

CNNs dominated computer vision from 2012 to around 2020. Then researchers applied the Transformer architecture (originally designed for text, as described in Part 6) to images. The result, called a Vision Transformer (ViT), divides an image into patches, treats each patch as a “token,” and applies self-attention across all patches.

Vision Transformers match or exceed CNN performance on many tasks when trained on very large datasets, and they benefit from the same scaling properties that made large language models so powerful. Most state-of-the-art vision models now use Transformer architectures or hybrids that combine CNNs and Transformers.

Computer Vision in Autonomous Vehicles

Self-driving vehicles represent one of the most demanding deployments of computer vision, requiring real-time understanding of a complex, constantly changing environment at highway speeds. A modern autonomous vehicle typically uses a combination of sensors: cameras (which provide rich color and texture information), LiDAR (which measures distance by emitting laser pulses and measuring reflections), and radar (which works in poor visibility and measures relative velocity).

Vision AI in autonomous vehicles must handle extraordinary challenges:

• Rare but critical events: a self-driving system may encounter a child running into the road, a mattress falling from a truck, or a construction worker's hand signals. These events are rare enough to be underrepresented in training data but consequential enough to demand correct handling.
• Adversarial conditions: rain, fog, glare, snow, and darkness all degrade camera performance. Research in sensor fusion — combining information from multiple sensor types — is critical for robustness.
• 3D understanding: images are 2D projections of a 3D world. Determining the actual distances and sizes of objects from camera images requires depth estimation, stereoscopic vision, or fusion with depth sensors.
• Real-time requirements: decisions about braking, steering, and acceleration must be made within milliseconds. The computational pipeline from sensor data to action command must run reliably in real time, every time.

Fully autonomous vehicles remain one of the most difficult open problems in applied AI, despite enormous investment and significant progress. The gap between “performs well 99.9% of the time” and “safe enough for general deployment in all conditions” has proven wider than early optimists anticipated.

Key Takeaways

• Images are grids of numbers (pixel values) — vision AI converts those numbers into meaning.
• CNNs use small sliding filters and weight sharing to efficiently detect spatial patterns at multiple scales.
• Stacking convolutional layers builds a hierarchy from simple edges to complex objects.
• Major tasks include image classification, object detection, segmentation, face recognition, and medical imaging.
• Vision AI has moved from CNNs toward Transformer-based architectures for state-of-the-art performance.
• Computer vision enables transformative applications in medicine, transportation, and security — alongside significant privacy concerns.

In Part 8, we bring language and vision together under the umbrella of generative AI: systems that do not just classify or detect, but create — generating realistic images, writing essays, composing code, and much more.