Image Segmentation
(Optimization Problem)
1
Labeling & Segmentation
• Labeling is a common way for modeling
various computer vision problems (e.g.
optical flow, image segmentation, stereo
matching, etc)
• The set of labels can be discrete (as in
image segmentation)
L = {l1, . . . , l m } with L = m
• Or continuous (as in optical flow)
L⊂ Rn for n ≥ 1 2
Labeling is a function
• Labels are assigned to sites (pixel locations)
• For a given image, we have 𝛺 =N c o l s .N ro w s
• Identifying a labeling function
(with segmentation) is f : 𝛺 →L
We aim at calculating a labeling function that
minimizes a given (total) error or energy
3
Energy Function
The role of an energy function in
minimization-based vision is twofold:
1. as the quantitative measure of the global
solution
2. as a guide to the search for a minimal solution.
Correct solution is embedded as the
minimum. 4
Segmentation as an Energy Minimization Problem
• Edata assigns non-negative penalties to a pixel
location p when assigning a label to this location.
• Esmooth assigns non-negative penalties by
comparing the assigned labels fp and fq at
adjacent positions p and q.
5
Segmentation as an Energy Minimization Problem
• Edata assigns non-negative penalties to a pixel
location p when assigning a label to this location.
• Esmooth assigns non-negative penalties by
comparing the assigned labels fp and fq at
adjacent positions p and q.
6
Segmentation as an Energy Minimization Problem
• Edata assigns non-negative penalties to a pixel
location p when assigning a label to this location.
• Esmooth assigns non-negative penalties by
comparing the assigned labels fp and fq at
adjacent positions p and q.
This optimization model is characterized by local
interactions along edges between adjacent pixels, and
often called MRF (Markov Random Field) model.
7
Energy Function-Details
8
Energy Function-Details
To solve this problem, transform the energy
functional into min-cut/max-flow problem and
solve it!
9
Energy Function
Unary Cost
(data term)
Discontinuity
Cost
p q 2
1
Recap: Image as a Graph
p q Edg
e Node/verte
pixe x
l
2
2
Graph Cuts for Optimal Boundary Detection
(Boykov ICCV 2001)
F
• Binary label: foreground vs.
background B
• User labels some pixels F F B
• Exploit F F B
– Statistics of known Fg & Bg F B B
– Smoothness of label
• Turn into discrete graph
optimization
– Graph cut (min cut / max flow) 2
3
Graph-Cut
Each pixel is connected to its neighbors in an undirected graph
Goal: split nodes into two sets A and B based on pixel values, and try to
13
classify Neighbors in the same way too!
Graph-Cut
Each pixel is connected to its neighbors in an undirected graph
Goal: split nodes into two sets A and B based on pixel values, and try to
14
classify Neighbors in the same way too!
Graph-Cut
• Each pixel = node
• Add two nodes F & B
• Labeling: link each pixel to either F
or B F Desired
result
B 15
Cost Function: Data term
• Put one edge between each pixel and both
F &G
• Weight of edge = minus data term
F
B 16
Cost Function: Smoothness term
• Add an edge between each neighbor
pair
• Weight = smoothness term
F
B 17
Min-Cut
• Energy optimization equivalent to graph min
cut
• Cut: remove edges to disconnect F from B
• Minimum: minimize sum of cut edge weight
F cut
B 2
9
Min-Cut
Source
2 9 Graph (V, E, C)
1 Vertices V = {v1, v2 ... vn}
v1 v2 Edges E = {(v1, v2) ....}
2 Costs C = {c(1, 2) ....}
5 4
Sink
19
Min-Cut
What is a st-cut?
An st-cut (S,T) divides the nodes
between source and sink.
Source
2 9 What is the cost of a st-cut?
1
Sum of cost of all edges going
v1 v2
from S to T
2
5 4
Sink
5 + 2 + 9 = 16 20
Min-Cut
What is a st-cut?
An st-cut (S,T) divides the nodes
between source and sink.
Source
2 9 What is the cost of a st-cut?
1
Sum of cost of all edges going
v1 v2
from S to T
2
5 4
What is the st-mincut?
Sink st-cut with the
minimum cost
5 + 2 + 9 = 16 21
How to compute min-cut?
Solve the dual maximum flow problem
Compute the maximum flow between
Source Source and Sink
Constraints
2 9
Edges: Flow < Capacity
1
v1 v2 Nodes: Flow in & Flow out
2
5 4
Min-cut\Max-flow Theorem
Sink
In every network, the maximum flow
equals the cost of the st-mincut
22
Max-Flow Algorithms
Flow = 0
Augmenting Path Based
Algorithms
Source
2 9 1. Find path from source to sink
with positive capacity
1
v1 v2
2. Push maximum possible flow
2
5 4 through this path
3. Repeat until no path can be
Sink
found
Algorithms assume non-negative capacity 23
Max-Flow Algorithms
Flow = 0
Augmenting Path Based
Algorithms
Source
2 9 1. Find path from source to sink
with positive capacity
1
v1 v2
2. Push maximum possible flow
2
5 4 through this path
3. Repeat until no path can be
Sink
found
Algorithms assume non-negative capacity 24
Max-Flow Algorithms
Flow = 0 + 2
Augmenting Path Based
Algorithms
Source
2-2 9 1. Find path from source to sink
with positive capacity
1
v1 v2
2. Push maximum possible flow
2
4 through this path
5-2
3. Repeat until no path can be
Sink
found
Algorithms assume non-negative capacity 25
Max-Flow Algorithms
Flow = 2
Augmenting Path Based
Algorithms
Source
0 9 1. Find path from source to sink
with positive capacity
1
v1 v2
2. Push maximum possible flow
2
3 4 through this path
3. Repeat until no path can be
Sink
found
Algorithms assume non-negative capacity 26
Max-Flow Algorithms
Flow = 2
Augmenting Path Based
Algorithms
Source
0 9 1. Find path from source to sink
with positive capacity
1
v1 v2
2. Push maximum possible flow
2
3 4 through this path
3. Repeat until no path can be
Sink
found
Algorithms assume non-negative capacity 27
Max-Flow Algorithms
Flow = 2
Augmenting Path Based
Algorithms
Source
0 9 1. Find path from source to sink
with positive capacity
1
v1 v2
2. Push maximum possible flow
2
3 4 through this path
3. Repeat until no path can be
Sink
found
Algorithms assume non-negative capacity 28
Max-Flow Algorithms
Flow = 2 +
Augmenting Path Based
4
Algorithms
Source
0 5 1. Find path from source to sink
with positive capacity
1
v1 v2
2. Push maximum possible flow
2
3 0 through this path
3. Repeat until no path can be
Sink
found
Algorithms assume non-negative capacity 29
Max-Flow Algorithms
Flow = 6
Augmenting Path Based
Algorithms
Source
0 5 1. Find path from source to sink
with positive capacity
1
v1 v2
2. Push maximum possible flow
2
3 0 through this path
3. Repeat until no path can be
Sink
found
Algorithms assume non-negative capacity 30
Max-Flow Algorithms
Flow = 6
Augmenting Path Based
Algorithms
Source
0 5 1. Find path from source to sink
with positive capacity
1
v1 v2
2. Push maximum possible flow
2
3 0 through this path
3. Repeat until no path can be
Sink
found
Algorithms assume non-negative capacity 31
Max-Flow Algorithms
Flow = 6 + 1
Augmenting Path Based
Algorithms
Source
0 4 1. Find path from source to sink
with positive capacity
1-1
v1 v2
2. Push maximum possible flow
2+1
2 0 through this path
3. Repeat until no path can be
Sink
found
Algorithms assume non-negative capacity 32
Max-Flow Algorithms
Flow = 7
Augmenting Path Based
Algorithms
Source
0 4 1. Find path from source to sink
with positive capacity
0
v1 v2
2. Push maximum possible flow
3
2 0 through this path
3. Repeat until no path can be
Sink
found
Algorithms assume non-negative capacity 33
Max-Flow Algorithms
Flow = 7
Augmenting Path Based
Algorithms
Source
0 4 1. Find path from source to sink
with positive capacity
0
v1 v2
2. Push maximum possible flow
3
2 0 through this path
3. Repeat until no path can be
Sink
found
Algorithms assume non-negative capacity 34
Example Segmentation in a Video
n- s = 0 st-cut
links
E(x) x*
wpq
t=1
Image Flow Global Optimum35
