Introduction

Vision is perhaps the most effective sensor input that a robot can receive to accurately capture and process information from its environment. Computer vision is a domain that deals with algorithms that help process an image, which is essentially a grid of pixels, into actionable inputs for a robot. It has applications in tasks such as object detection and recognition, object tracking, localization and pose estimation. Computer Vision has progressed through two distinct paradigms; classical CV (deterministic algorithms such as filters and edge/feature detectors) and deep-learning CV (learned features from data). Both paradigms have their own merits: classical CV is often faster, more predictable, and good enough for structured tasks (like reading ArUco markers), while deep learning excels in dynamic, unstructured environments.

Key Concepts

• Image formation & representation — a digital image is a grid of pixels, each holding intensity (grayscale) or color (RGB/HSV/etc.) values. Understanding color spaces is important — e.g. HSV separates color from brightness, which makes color-based object detection far more robust to lighting changes than RGB thresholding.
• Classical CV pipelines — The standard toolkit for a classica CV pipeline often looks something like this: filtering/smoothing (Gaussian blur to reduce noise), edge detection (Sobel, Canny(refer to tutorial at the end) — finding sharp intensity changes that usually correspond to object boundaries), feature detection (corners/keypoints like Harris, SIFT, ORB — distinctive points that can be matched across images), and camera calibration (finding a camera's intrinsic parameters — focal length, distortion). Read these two excellent articles on various aspects of classical CV to get a better understanding.

Exploring Traditional Feature Detection and Matching Algorithms in Computer Vision

Camera Calibration Explained: Enhancing Accuracy in Computer Vision Applications

• Object detection & tracking — Object detection helps recognize and locate objects in an image and classify them under labels (usually as bounding boxes + class labels); many applications require real-time information on the location of an identified object for which object tracking algorithms are used. Modern detection is dominated by deep learning (YOLO-family, Faster R-CNN), but classical techniques (color/contour-based detection, optical flow for tracking) are still extremely useful due to their low memory overhead and latency on edge hardware (e.g. an onboard microcontroller with no GPU). Optical flow is the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and the scene. Despite being perhaps less accurate than deep-learning based methods, it is heavily applied in robotics as it provides fairly good accuracy with immensely low computational cost in comparison. It finds applications in video stabilization, object tracking, structural analysis, robot navigation, and video compression.

Optical Flow: Revolutionizing Motion Detection

The Lucas-Kanade method is the most widely used optical flow algorithm, and the most fundamental.

publications.ri.cmu.edu

• Depth estimation & stereo vision — A single camera image has no depth information by default; we either infer it (with monocular depth estimation, usually deep-learning-based today) or recover it geometrically using stereo vision: two cameras at a known baseline distance, where the disparity (pixel shift) between matching points in each image is inversely proportional to depth; this is synonymous to human vision.

Stereo Vision and Depth Estimation - GeeksforGeeks

• Visual SLAM — VSLAM is when SLAM is implemented with visual sensors. In VSLAM, a camera is used as the primary sensor to simultaneously build a map of an unknown environment and track the robot's position within it.

Introduction to Visual SLAM: Chapter 1 —Introduction to SLAM

• Deep learning for perception — Convolutional Neural Networks (CNNs) are the most fundamental architecture for vision-based applications: they learn hierarchical visual features (edges → textures → parts → objects) directly from labeled data. This is what underlies modern object detection, segmentation, and increasingly depth estimation too.

Tools & Libraries

The following are the industry-standard toolkits and libraries for computer vision applications: