Computer Vision — lecture notes

Lecture 02 #

Computer Vision Challenges #

• 2D image but 3D world
  • Motion
  • Background clutter
  • Illumination
  • Perception vs. measurement
  • Illusion

Projective Geometry #

• Instead of Euclidean coordinates, we use projective coordinates (adding a Z = 1 onto the image plane)
• We can easily find intersections

Lecture 03 #

• Pinhole camera
• Perspective transformation
• Detect the same point in different images

Lecture 04 #

Harris Detector #

Which shift gives the largest pixel differences? We can find edges and weight the center more.

• Workflow:
  1. Calculate image derivatives
  2. Sum them up and use a Gaussian filter
  3. Apply the cornerness function
  4. Find points above the threshold and the largest in an area

Laplacian #

• Detect blobs
• Find the maximum in location and scale
• Approximate the Laplacian with a difference of Gaussians

Lecture 05 #

Feature Descriptors #

Find similar patches.

• Simple descriptor (list of intensities in a patch): problems with shifts and lighting
• Histograms of image gradients
• SIFT: Scale-Invariant Feature Transform

Epipolar Geometry #

• Fundamental matrix encodes rotation and translation/scale
• If we look at an object from different perspectives, the epipolar lines should end at the same origin
• Essential matrix
  • Used if we know the focal length, etc.
  • How to estimate it

Lecture 06 #

Triangulation #

• Triangulate points
• Use SVD and decrease the rank to 2 to converge to the singularity constraint

RANSAC #

• Lines rarely intersect in 3D space, but we can project points to the epipolar plane, where they will intersect
• Multiview geometry: four constraints that can be used with up to four perspectives
• Reduce problems with linear constraints, then solve the nonlinear ones

Lecture 07 #

M-estimation #

• Use robust estimation to limit outlier influence

RANSAC #

• Algorithm
• Calculate sample size
• Adaptive sampling

Camera Calibration #

Lecture 08 #

• How COLMAP constructs space: cluster similar images, then choose similar images to calculate coordinates
• Sparsity in computer vision

Lecture 09 #

Multi-View Stereo Matching #

• Get a dense depth map
• Check disparity on the epipolar line to determine depth
• Constraints: uniqueness, ordering, disparity constraint, etc.
• Minimize energy to obtain a smoother surface using dynamic programming in multiple directions
• If points are not on the same x-coordinate, rectify the image (make epipolar lines parallel, at infinity)
• With forward motion, the lines will be radial

Lecture 10 #

Image-Based Rendering #

• Image morphing
• Mosaics (panorama)
• Light field: perspective from every angle
• NeRF (Neural Radiance Fields): represent an object as fog with a prediction function; requires high frequencies in the Fourier domain
• 3D Gaussian Splatting (3DGS): blobs of color; orthogonal projection can be used

Lecture 11 #

Deep Learning #

• MLPs (multilayer perceptrons / neural networks)
  • Cross-entropy loss: negative log-likelihood
  • Optimization (SGD)
  • Backpropagation and Jacobian chaining
• CNNs (Convolutional Neural Networks)

Lecture 12 #

• Pooling
• Normalization
• CNN examples

Lecture 13 #

Transformer Models (Images & Text) #

• Embeddings
• Attention heads
• Feed-forward layers

Traditional Segmentation Models #

• K-means
• GMM
• KDE

Lecture 14 #

• GMM (Gaussian Mixture Model): uses Gaussians to also represent probability
• KDE (Kernel Density Estimation): mean-shift density; we do not need to predefine K

Lecture 15 #

Segmentation with Deep Learning #

• Know what you have and what you do: training data, test data, evaluation

Evaluation Methods #

• Pixel accuracy
• Class pixel accuracy
• Mean pixel accuracy
• Intersection over union: intersection (true positive) / (true positive + false positive + false negative)
• Precision and recall: true positive / all true, true positive / predicted true
• Precision-recall curve and mean average precision

Lecture 16 #

• Transposed convolution
• Fully Convolutional Network
• U-Net: feed-forward connections preserve spatial information and add it to semantic information
• SegFormer: transformer block uses convolution to reduce dimensions