-
Notifications
You must be signed in to change notification settings - Fork 0
/
boundary-jaccard.html
248 lines (240 loc) · 13.9 KB
/
boundary-jaccard.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Boundary Jaccard Metric — A Place for Asides</title>
<meta name="description" content="Title: Boundary Jaccard Metric; Date: 2020-10-25; Author: Peter Mortimer">
<meta name="author" content="Peter">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<!-- Le HTML5 shim, for IE6-8 support of HTML elements -->
<!--[if lt IE 9]>
<script src="./theme/html5.js"></script>
<![endif]-->
<link href="./theme/css/ipython.css" rel="stylesheet">
<link href="./theme/css/jquery.tocify.css" rel="stylesheet">
<link href="./theme/css/lightbox.css" rel="stylesheet">
<link href="//maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap.min.css" rel="stylesheet">
<link href="//maxcdn.bootstrapcdn.com/font-awesome/4.1.0/css/font-awesome.min.css" rel="stylesheet">
<!--
<link href="https://stackpath.bootstrapcdn.com/bootswatch/4.3.1/minty/bootstrap.min.css" rel="stylesheet">
-->
<link href="./theme/css/local.css" rel="stylesheet">
<link href="./theme/css/pygments.css" rel="stylesheet">
</head>
<body>
<div id="toc"></div><div class="container main">
<div class="page-header">
<div class="row">
<div class="col-8 col-sm-8">
<h1><a href="./">A Place for Asides</a>
<br> </div>
<div class="col-4 col-sm-4 page-header-links">
<p><a href="/pages/about.html">About</a> | <a href="/pages/publications.html">Publications</a> | <a href="/pages/previous-projects.html">Previous Projects</a> | <a href="/">Blog</a> | <a href="/pages/books.html">Book List</a> </p>
</div>
</div>
</div>
<div class="row">
<div class="col-md-8 col-md-offset-2">
<div class="article" itemscope itemtype="http://schema.org/BlogPosting">
<div class="text-center article-header">
<h1 itemprop="name headline" class="article-title">Boundary Jaccard Metric</h1>
<span itemprop="author" itemscope itemtype="http://schema.org/Person">
<h4 itemprop="name">Peter Mortimer</h4>
</span>
<time datetime="2020-10-25T09:30:00+01:00" itemprop="datePublished">October 25 2020</time>
</div>
<div itemprop="articleBody" class="article-body"><!-- Status: draft -->
<h1 style="visibility:hidden;">Boundary Jaccard (BJ Metric)</h1>
<p>The recently introduced Boundary Jaccard [<a href="#bj_iv">1</a>] metric is a new evaluation metric for semantic segmentation algorithms. The BJ lays a focus on the segmentation accuracy along the contours. Describing this mathematically can be a bit awkward. Luckly, visualizing the regions of interest of the BJ metric can provide a more intuitive understanding of its caluclation. </p>
<p>I'll quickly introduce the use case in which we will analyze the BJ metric. The Cityscapes dataset [<a href="#cityscapes">2</a>] is a large-scale dataset consisting of urban driving scenes recorded across many German cities with high quality pixel-level annotations. Cityscapes is a commonly used dataset for the development of semantic image segmentation methods. We will use Fast-SCNN [<a href="#fast_scnn">3</a>] as our semantic segmentation algorithm, since it produces fairly reasonable predictions on the Cityscapes dataset. </p>
<p><hr>
<div class="row">
<div class="col-4 col-sm-4">
<a href="/images/bj_metric/darmstadt_anon.png"
data-lightbox="cityscapes-darmstadt"
data-title="This is an input out of the training set of the Cityscapes dataset."
data-alt="This is an input out of the training set of the Cityscapes dataset."><img src="/images/bj_metric/darmstadt_anon.png"></a>
</div>
<div class="col-4 col-sm-4">
<a href="/images/bj_metric/darmstadt_gt.png"
data-lightbox="cityscapes-darmstadt"
data-title="This is a ground truth annotation out of the Cityscapes dataset. Each color represents a different semantic class."
data-alt="This is a ground truth annotation out of the Cityscapes dataset. Each color represents a different semantic class."><img src="/images/bj_metric/darmstadt_gt.png"></a>
</div>
<div class="col-4 col-sm-4">
<a href="/images/bj_metric/darmstadt_pred.png"
data-lightbox="cityscapes-darmstadt"
data-title="This is an example prediction of the Fast-SCNN network. Notice how the car hood is falsely predicted as part of the road surface. The BJ metric is a method to quantitatively evaluate the accuracy of a semantic scene segmentation."
data-alt=""><img src="/images/bj_metric/darmstadt_pred.png"></a>
</div>
</div></p>
<p><span class="caption">Here is an example scene out of the Cityscapes dataset. The second column shows the ground truth scene annotation by a human. The third column shows the output prediction of the Fast-SCNN network on the same input image. The question is how can we evaluate the model prediction in a quantitative manner?</span>
<hr></p>
<h3>Introducing the BJ Metric</h3>
<p>Metrics for the semantic scene segmentation are calculated for each class <span class="math">\(c\)</span> of a segmentation map <span class="math">\(S\)</span>. We denote the pixels of a segmentation map that belong to class <span class="math">\(c\)</span> as <span class="math">\(S^{c}\)</span>. Here we also distinguish between the predicted segmentation map <span class="math">\(S_{ps}\)</span> and the ground truth segmentation map <span class="math">\(S_{gt}\)</span>.</p>
<p>The authors of the BJ metric are inspired by the so-called BF [<a href="#bf_bmvc">4</a>] score ("Boundary F1 score"). The BF score introduces the notion of focusing on the segmentation accuracy along class borders, since this is where a good semantic segmentation can still make most of its mistakes. For this we define <span class="math">\(B^{c}\)</span> as the pixels along the border of a class <span class="math">\(c\)</span> and <span class="math">\([[\cdot]]\)</span> denotes the Iversons bracket notation, where <span class="math">\([[z]] = 1\)</span> if <span class="math">\(z=true\)</span> and <span class="math">\(0\)</span> otherwise.</p>
<div class="math">$$ P^{c} = \frac{1}{|B_{ps}^{c}|} = \sum_{x \in B_{ps}^{c}} [[d(x,B_{gt}^{c}) < \theta ]]$$</div>
<div class="math">$$ R^{c} = \frac{1}{|B_{gt}^{c}|} = \sum_{x \in B_{gt}^{c}} [[d(x,B_{ps}^{c}) < \theta ]]$$</div>
<div class="math">$$ BF^{c} = F_{1}^{c} = \frac{2 \cdot P^{c} \cdot R^{c}}{P^{c} + R^{c}}$$</div>
<p>This assumption on the higher relevance along the class borders is combined with the common Jaccard index (a.k.a. Intersection over Union). The Jaccard index is an established evaluation metric for 2D and 3D object detection tasks and can also be used to evaluate 2D semantic maps. In simple terms, the Jaccard index looks at the size of the relative overlap between the area made up by the prediction pixels and the ground truth pixels.</p>
<div class="math">$$ IoU^{c} = \frac{S_{ps}^{c} \cap S_{gt}^{c}}{S_{ps}^{c} \cup S_{gt}^{c}} $$</div>
<h3>References</h3>
<p><span id='bj_iv'>[1] Eduardo Fernandez-Moral, Renato Martins, Denis Wolf, Patrick Rives (IV 2018) <a href="https://hal.inria.fr/hal-01581525/document">A new metric for evaluating semantic segmentation: leveraging global and contour accuracy</a>.</span></p>
<p><span id='cityscapes'>[2] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld,
Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele (CVPR 2016) <a href="https://www.cityscapes-dataset.com/wordpress/wp-content/papercite-data/pdf/cordts2016cityscapes.pdf">The Cityscapes Dataset for Semantic Urban Scene Understanding</a>.</span></p>
<p><span id='fast_scnn'>[3] Rudra P. K. Poudel, Stephan Liwicki, Roberto Cipolla (BMVC 2019) <a href="https://bmvc2019.org/wp-content/uploads/papers/0959-paper.pdf">Fast-SCNN: Fast Semantic Segmentation Network</a> - I use the <a href="https://github.com/Tramac/Fast-SCNN-pytorch">PyTorch implementation of Fast-SCNN</a> by the GitHub User Tramac.</span></p>
<p><span id='bf_bmvc'>[4] Gabriela Csurka, Diane Larlus, Florent Perronnin (BMVC 2013) <a href="http://www.bmva.org/bmvc/2013/Papers/paper0032/paper0032.pdf">What is a good evaluation measure for semantic segmentation?</a></span></p>
<script type="text/javascript">if (!document.getElementById('mathjaxscript_pelican_#%@#$@#')) {
var align = "center",
indent = "0em",
linebreak = "false";
if (false) {
align = (screen.width < 768) ? "left" : align;
indent = (screen.width < 768) ? "0em" : indent;
linebreak = (screen.width < 768) ? 'true' : linebreak;
}
var mathjaxscript = document.createElement('script');
mathjaxscript.id = 'mathjaxscript_pelican_#%@#$@#';
mathjaxscript.type = 'text/javascript';
mathjaxscript.src = 'https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-MML-AM_CHTML';
var configscript = document.createElement('script');
configscript.type = 'text/x-mathjax-config';
configscript[(window.opera ? "innerHTML" : "text")] =
"MathJax.Hub.Config({" +
" config: ['MMLorHTML.js']," +
" TeX: { extensions: ['AMSmath.js','AMSsymbols.js','noErrors.js','noUndefined.js'], equationNumbers: { autoNumber: 'none' } }," +
" jax: ['input/TeX','input/MathML','output/HTML-CSS']," +
" extensions: ['tex2jax.js','mml2jax.js','MathMenu.js','MathZoom.js']," +
" displayAlign: '"+ align +"'," +
" displayIndent: '"+ indent +"'," +
" showMathMenu: true," +
" messageStyle: 'normal'," +
" tex2jax: { " +
" inlineMath: [ ['\\\\(','\\\\)'] ], " +
" displayMath: [ ['$$','$$'] ]," +
" processEscapes: true," +
" preview: 'TeX'," +
" }, " +
" 'HTML-CSS': { " +
" fonts: [['STIX', 'TeX']]," +
" styles: { '.MathJax_Display, .MathJax .mo, .MathJax .mi, .MathJax .mn': {color: 'inherit ! important'} }," +
" linebreaks: { automatic: "+ linebreak +", width: '90% container' }," +
" }, " +
"}); " +
"if ('default' !== 'default') {" +
"MathJax.Hub.Register.StartupHook('HTML-CSS Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax['HTML-CSS'].FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"MathJax.Hub.Register.StartupHook('SVG Jax Ready',function () {" +
"var VARIANT = MathJax.OutputJax.SVG.FONTDATA.VARIANT;" +
"VARIANT['normal'].fonts.unshift('MathJax_default');" +
"VARIANT['bold'].fonts.unshift('MathJax_default-bold');" +
"VARIANT['italic'].fonts.unshift('MathJax_default-italic');" +
"VARIANT['-tex-mathit'].fonts.unshift('MathJax_default-italic');" +
"});" +
"}";
(document.body || document.getElementsByTagName('head')[0]).appendChild(configscript);
(document.body || document.getElementsByTagName('head')[0]).appendChild(mathjaxscript);
}
</script></div>
<hr>
<div>
Category:
<span itemprop="articleSection">
<a href="./category/python.html" rel="category">Python</a>
</span>
</div>
<div>
Tags:
<span itemprop="keywords">
<a href="./tag/metrics.html" rel="tag">metrics</a>
</span>
<span itemprop="keywords">
<a href="./tag/image-processing.html" rel="tag">image processing</a>
</span>
</div>
</div>
</div>
</div> <!-- <hr> -->
</div> <!-- /container -->
<footer class="aw-footer bg-danger">
<div class="container"> <!-- footer -->
<div class="row">
<div class="col-md-10 col-md-offset-1">
<div class="row">
<div class="col-md-3">
<h4>Navigation</h4>
<ul class="list-unstyled my-list-style">
<li><a href=".">A Place for Asides</a></li>
<li><a href="./pages/about.html"><i class="fa fa-About "></i> About</a></li>
<li><a href="./pages/blogroll.html"><i class="fa fa-Blogroll "></i> Blogroll</a></li>
<li><a href="./pages/books.html"><i class="fa fa-Books I've Read "></i> Books I've Read</a></li>
<li><a href="./pages/previous-projects.html"><i class="fa fa-Previous Projects "></i> Previous Projects</a></li>
<li><a href="./pages/publications.html"><i class="fa fa-Publications "></i> Publications</a></li>
</ul>
</div>
<div class="col-md-3">
<h4>Author</h4>
<ul class="list-unstyled my-list-style">
<li><a href="https://github.com/tonyromarock">GitHub</a></li>
</ul>
</div>
<div class="col-md-3">
<h4>Categories</h4>
<ul class="list-unstyled my-list-style">
<li><a href="./category/linux.html">Linux (1)</a></li>
<li><a href="./category/nonfiction.html">Nonfiction (1)</a></li>
<li><a href="./category/python.html">Python (4)</a></li>
</ul>
</div>
</div>
</div>
</div>
</div>
</footer>
<div class="container bottom">
<div class="row">
<div class="col-md-12 text-center center-block aw-bottom">
<p>© Peter 2020</p>
<p>Powered by Pelican</p>
</div>
</div>
</div>
<!-- JavaScript -->
<script src="https://code.jquery.com/jquery-2.1.1.min.js"></script>
<script src="./theme/js/jquery-ui-1.9.1.custom.min.js"></script>
<script src="./theme/js/jquery.tocify.min.js"></script>
<script src="./theme/js/lightbox.js"></script>
<script src="//maxcdn.bootstrapcdn.com/bootstrap/3.2.0/js/bootstrap.min.js"></script>
<script type="text/javascript">
jQuery(document).ready(function($) {
$("div.collapseheader").click(function () {
$header = $(this).children("span").first();
$codearea = $(this).children(".input_area");
$codearea.slideToggle(500, function () {
$header.text(function () {
return $codearea.is(":visible") ? "Collapse Code" : "Expand Code";
});
});
});
});
$(function() {
var toc = $("#toc").tocify({
context:"div.article-body",
selectors:"h1,h3",
showAndHide:"false",
extendPage:"false",
history:"true",
scrollHistory:"true"
}).data("toc-tocify");
});
lightbox.option({
'wrapAround': true
});
</script>
</body>
</html>