How a Machine Sees: Principles of Computer Vision Systems

Computer vision has long ceased to be a topic of science fiction. It recognizes faces on phones, detects defects on a manufacturing line, helps a car stay in its lane, and monitors people's movement at an airport. But behind this ubiquity lies a question that rarely finds a clear answer: how does a machine actually "see"?

In this article, the first in a series on computer vision, we will review the fundamental structure of any computer vision system: the path from a camera's raw frame to the specific decision the system outputs. This is the foundation without which subsequent articles on recognition, detection, tracking, and motion control will be a collection of terms without an understanding of their origins.

Let us start from the very beginning. For a human, an image is a picture: a face, a tree, a road. For a machine, an image is a matrix of numbers. Each pixel is one or more cells in this matrix, containing brightness values for each color channel.

A standard camera outputs an image in the RGB space, which has three channels: red, green, blue. This is convenient for displaying on a screen, but not always convenient for analysis. For example, for tasks where an object's color is important regardless of lighting (say, separating a ripe fruit from an unripe one or finding a specific shade on a conveyor), engineers often convert the image to another color space, such as HSV (hue, saturation, value) or Lab. In these spaces, color and brightness are separated, and the task becomes technically simpler.

It is worth expanding on this thought right away: a camera is not the only source that provides a machine with an "image" in a broad sense. Lidar, for instance, instead of a pixel matrix, gives a point cloud in space, where each point has coordinates and a distance to the object. The format is different, but the task is the same: this data needs to be processed, features must be extracted from it, and objects must be recognized using methods that account for spatial rather than color-based representation. We will return to this in more detail in articles about motion control systems, where a camera and lidar often work in tandem.

Any computer vision system, from the simplest to an industrial one, is structured as a sequence of stages. A convenient analogy is how a person looks for a familiar face in a crowd: first, the eye catches silhouettes; then attention focuses on facial features; then the brain compares these features with memory and issues a verdict: "know" or "do not know".

This picture is an AI Generated

A computer vision system goes through a similar path, only explicitly and step by step, as shown in the diagram below.

Everything begins with a camera, and here the limitations of the entire system are already established. Resolution determines how many details are available for analysis. Frame rate determines how quickly the system can react to changes (this will become critical when we talk about video streams and real-time control systems). Shooting conditions, illumination, camera angle, and lens distortion shape what all subsequent stages will have to work with. An important practical point: if the camera is installed poorly or shoots in conditions for which the system was not designed, no algorithm in subsequent stages will completely compensate for this. A good computer vision system begins with thoughtful capture, not with a "powerful neural network".

A raw frame almost never goes directly to analysis. First, it needs to be brought into a predictable form: remove noise, correct the lighting, and normalize the object's scale and orientation. This is the stage that rarely makes it into marketing presentations, but it is often what determines whether the system will work in real conditions or only on neatly selected demo examples. A revealing example: a model trained and tested in a laboratory under uniform lighting can sharply lose accuracy in a workshop with uneven light or glare from metallic surfaces simply because the preprocessing stage was not designed for such conditions.

This is the stage where the main part of "seeing" occurs: transforming pixels into information with which one can reason: whether there is an edge, a corner, a characteristic texture, or a shape in the image. Here lies the key boundary around which the entire further series of articles will be built: 

• Classical approach. The engineer determines which features to look for edges, corners, gradient histograms, and local texture patterns and writes an algorithm to compute them. This is transparent, predictable, and does not require a large volume of data, but is limited by what a person can formulate explicitly in advance. 

• Deep learning approach. The model finds which features are important on its own, learning from a large number of examples. This removes the limitation "a person must know in advance what to look for", but in return requires data, computational resources, and reduces the transparency of the decision; namely, it becomes harder to explain why the model made that specific decision. 

Neither approach is unconditionally better; the choice depends on the task, the volume of available data, threquiredhe strictness of analysis and decision-making, and the resources for implementation. In traditional mechanical engineering and safety-critical systems, classical methods are often still justified

The features extracted in the previous step are fed into a model that issues the final verdict: which class the object belongs to, where exactly it is located in the image, and whether it is the same object as in the previous frame. This can be a simple rule, a statistical classifier, or a neural network: the principle is the same: features at the input, a decision at the output.

The result of the model is rarely a ready answer for business in its pure form. It needs to be filtered (discarding decisions with low confidence), calibrated (ensuring that the model's "95% confidence" actually corresponds to 95% accuracy in practice), and formatted for the next system or a human. Here lies another practical nuance: the quality of a computer vision system cannot be evaluated by whether it "worked nicely on a couple of examples". Metrics exist, precision, recall, noise immunity, type I and type II errors; without them, talk about system quality remains at the level of impressions.

A computer vision system is not an artifact that is set up once and forgotten. Conditions change: different equipment, another camera angle, new types of objects. A well-designed system assumes a feedback loop: collecting new data, labeling complex cases, periodic retraining or fine-tuning of the model.

This is a separate topic that deserves attention already in the first article, because this is where expectations and results most often diverge. A model that shows impressive accuracy on a test dataset can sharply lose quality when encountering real-world conditions: different lighting, other camera angles, and objects not in the training set. The difference between the data on which the model was trained and the data it encounters in real-world operation is one of the main reasons why computer vision projects show mediocre results immediately after launch, or even disappoint. In practice, it looks like this: 80% of typical cases are solved relatively quickly and predictably, while the remaining 20%, such as rare angles, nonstandard lighting, and edge cases, account for most of the project's cost and time. This is not an exception but a pattern that any serious computer vision system encounters when moving from prototype to deployment, especially when the system must operate in real-world conditions where the number of parameters that can be influenced is at its maximum.

In this article, we covered the stages that any computer vision system consists of, from frame capture to decision-making and feedback. This is the framework onto which, in subsequent articles in the series, we will overlay specific tasks: how object classification and detection are solved; how segmentation, identification, tracking, and texture analysis are performed. At the end of the series, we will reach vision-based motion control systems, including mobile robots and drones. 

Next article: Tasks of Computer Vision: we will break down the types of tasks this field addresses in general and the fundamental differences between them.