Computer Vision Key Tasks. Understanding and Overview

In the previous article, we looked at how a computer vision system is built internally, from a camera frame to a final decision. The next step is to understand what such a system can actually be used for. At first glance, the list of computer vision tasks may seem like a set of unrelated terms: classification, detection, segmentation, identification, tracking, texture analysis. It is not. It is a system with clear internal logic.

In this article, we go through these tasks and the basic terminology, so that later articles can build on a shared, unambiguous vocabulary.

This post isn't published yet — its content will appear here once it is.

A principle for classifying computer vision tasks

The clearest way to classify computer vision tasks is not by the algorithm used or the industry they serve, but by one simple question: what does the system actually return as output?

That perspective makes the field much easier to organize. Tasks that may look very different at first glance often share the same kind of input, whether an image or a video stream, and differ mainly in the form of the answer they produce.

This is not an exhaustive list of every computer vision task. The field continues to evolve, and hybrid formulations keep emerging at the boundaries. Even so, this table covers most of the tasks that commonly arise in real-world applications and gives you the vocabulary to state precisely what you need. Diagram below showing the basic (fundamental) tasks of computer vision.

The same table can be collapsed into a simpler, useful split: tasks that can be solved on a single frame (classification, detection, segmentation, texture analysis, identification, and verification), and tasks that inherently require a video stream or a sequence of frames (tracking, re-identification, pose estimation, and anomaly detection over time). This split matters even more once we get to the article on real-time video stream analysis, where the difference between "one frame" and "a sequence of frames" turns into an engineering problem rather than just a classification label.

It is also worth keeping a few terms distinct. Identification asks which known subject is in the frame; verification asks whether two observations come from the same subject. Tracking follows an object or person across frames within a sequence, while re-identification tries to match an object or person seen again after a gap, often in a different view or at a different time.

Later in the series we'll take each of these tasks in turn and go through the methods used to solve it, from classical algorithms to current architectures. Before moving on to methods, though, it's worth fixing the terminology itself, so nothing gets confused later.

Basic computer vision terminology

Here are three pairs of terms that sound similar but refer to fundamentally different tasks: classification vs. detection, segmentation vs. recognition, and tracking vs. matching. Being able to tell them apart is one of the most practically useful skills in computer vision work, and it's exactly where non-technical people on a project most often lose the thread.

Classification

Answer the question, “What is this frame showing overall?” with a single response for the entire frame. The system does not specify where the object appears in the image, only that it is present.

Object detection

Classification names what is in an image. Object detection goes a step further: it identifies what is there and also tells where it is, usually with a rectangular region called a bounding box. If an image contains several objects, detection can find and label each one.

Segmentation

Image segmentation is more precise because it assigns a label to every individual pixel in the image rather than to a rectangular region. This gives you the object's exact outline instead of an approximate box around it.

Practical implication

When a business requirement is phrased as "find the defect on the part," it is important to clarify up front whether the goal is simply to locate the defective area (detection), or whether you need the defect's exact outline to measure its size (segmentation). Detection is usually sufficient when the primary need is to determine whether a defect exists and its general location, especially if the next step is a manual review or a pass/fail decision. Segmentation is needed when the exact shape, boundaries, or affected area must be measured, compared, or used for further analysis. Although the wording may sound similar, these are two very different project options in terms of complexity and cost.

The image below lets you drag the divider between the raw frame and the segmentation output. Notice that the green and red regions follow the actual outline of the table and the person, not a rectangle around them.

Recognition, identification, and verification

Here the difference is of a different kind and it's about which question is actually being asked of the system.

Recognition

In the narrow sense of classification into known categories answers "which of the known, predefined classes does this object belong to." The classes are fixed in advance and finite in number.

Identification

Answers "which specific object, among a known set, is this" for example, which specific person in a database this face belongs to. This is an open-set problem: the system must be able to say "this doesn't match anyone in the database," rather than simply returning the closest match.

Verification

Is a narrower task answering "is this the same object as that other one, yes or no." Not "which one is it out of many," but a comparison between two specific instances.

This distinction matters in any task where you need a precise answer to the question of exactly which object you're looking at, and whether it's known to you. A face-based access control system is verification ("does this face match the passport photo or not"), not identification ("which of all the company's employees is currently in front of the camera"). These are tasks of different complexity, and confusing them at the start of a project regularly results in a method that simply doesn't fit the actual business problem.

Tracking and re-identification

The next typical tasks working with video and time.

Object tracking

Following a specific object from frame to frame within one continuous observation. The system keeps the object in view even as it slightly changes shape or angle, or is partially occluded by another object.

Re-identification

Solves a different problem: the object disappeared from view area, it left the frame, was fully occluded, or moved into another camera's field of view and then reappeared. The system needs to determine that this is the same object, not a new one.

The distinction is practically significant in multi-camera surveillance systems: tracking operates within one camera's field of view, while re-identification is needed to stitch an object's trajectory together as it moves between cameras. These are two different engineering problems, usually solved with different methods and different levels of accuracy.

Poses and gestures

Pose estimation asks a different question than detection or segmentation: not "where is the object" but "how is it arranged in space." The output is a set of key points like joints, landmarks, skeletal points and each with its own coordinates, rather than a single label or outline. This is what makes gesture recognition, motion analysis, and pose-driven interfaces possible: a bounding box tells you an object is present, key points tell you what it's doing.

ℹ

As example how machine sees human hand you can use widget below.

On the widget below, drag any joint and watch what happens to the dashed bounding box and the coordinates underneath it. The box barely changes in a meaningful way no matter how you pull it, just marks an area. The point coordinates change in meaningful ways, which is why gesture and pose recognition use keypoint vectors instead of bounding boxes.

3D scene analysis

Most of the tasks covered so far operate on a flat, two-dimensional image. 3D scene analysis extends that frame: it adds the depth, spatial structure, and geometric relationships that a single photo cannot show on its own. This is what a system needs when a decision depends not just on what is visible, but on how far away it is and how objects are positioned relative to each other — a requirement that becomes unavoidable once vision is used to guide movement, covered later in this series.

Anomalies and events detection

Some tasks are not about identifying a single object, but about detecting when something changes. Anomaly and event detection models track the state of a scene over time and flag deviations from expected behavior. This might include a machine operating outside its normal parameters, an object appearing where it should not, or a sequence of actions that does not match a known pattern. In these cases, the object itself may already be well understood; the critical signal is the moment its behavior ceases to be normal.

ℹ

Past Experience. Image normalization and tracking of moving objects

These are areas I worked on directly in the late 1990s and early 2000s, developing algorithms for real-time image-frame normalization in recognition tasks, tracking, and mobile robot navigation. We will return to both topics in detail in the block of articles on vision-based control systems and examine what has changed in the more than twenty years since.

It might look like all of this is a matter of vocabulary rather than engineering practice. In reality, it directly shapes the problem statement, which in turn determines everything that follows in a project: what data needs to be collected, how it should be labeled, which model architecture to choose, and which metrics to use for evaluation. If the task is stated imprecisely at the outset—say, the business needs verification but the team designs a classifier over a fixed list of classes—the result may formally "work" without solving the actual problem. This is one of the most common reasons a finished computer vision system fails to meet expectations: not because the algorithm is bad, but because it solved the wrong problem. That is why the first engineering decision in any serious computer vision project is not choosing a model, method, or algorithm, but stating the task precisely, in the terms covered in this article.

What's next?

We've mapped out computer vision tasks and sorted out the terms that get confused most often. In the next articles, we'll take these tasks one at a time and walk through the methods used to solve them, from classical algorithms to modern deep learning architectures, along with quality metrics and common implementation mistakes.

Up next: classification and object detection—how these tasks were approached historically and how they look today. Keep reading to see how the field has evolved.

Questions ?

If you need help with the implementation of Computer Vision systems, contact me via the SOFYCOD corporate website Contact Author, and I can consult you about questions related to model selection, dataset preparation, deployment, and performance optimization. the development and implementation of Computer Vision systems.

Navigation