Subset selection for visualization of relevant image fractions for deep learning based semantic image segmentation

For a trained CNN, we formulate an optimization problem to extract relevant image fractions for semantic segmentation. We try to identify a subset of pixels that contain the relevant information for the segmentation of one selected object class. In experiments on the Cityscapes dataset, we show that this is an easy way to gain valuable insight into a CNN trained for semantic segmentation. Looking at the relevant image fractions, we can identify possible limits of a trained network and draw conclusions about possible improvements.