Computer Vision News - December 2018

Every month, Computer Vision News reviews a research paper from our field. This month we have chosen CNN in MRF: Video Object Segmentation via Inference in a CNN-Based Higher-Order Spatio-Temporal MRF . We are indebted to the authors ( Linchao Bao , Baoyuan Wu and Wei Liu ), for allowing us to use their images to illustrate our review. Their article is here . Background, motivation and novelty: 1. Aim & Motivation / Challenge: This paper deals with segmentation of video, where the input includes (in addition to the sequence of images) a preliminary object mask -- this is the object whose segmentation we will want to refine. The authors propose a spatio-temporal algorithm , which uses a Markov Random Field (MRF) to the pixel-space of each-image (the spatial part) and between pixels in consecutive frames of the video (the temporal part). Its innovation on regular MRF models is the modeling of the spatial dependence between pixels for MRF using convolutional neural networks (CNN) , the temporal dependence between pixels is modelled by optical flow. The resulting MRF model combines spatial and temporal clues for video object segmentation. Performing inference directly from this MRF model is a very difficult computation problem due to the model’s high interdependence between pixels. Therefore, the authors propose performing approximate inference using embedded CNN -- this algorithm alternates between a temporal fusion step and a feed-forward CNN step. Initialized with one-shot video object segmentation CNN, the proposed algorithm achieved the top performance in the DAVIS 2017 public benchmark. 2. The main ideas: An innovative spatio-temporal MRF model for video object segmentation. The authors’ algorithm performs approximate inference from the MRF model by alternating between a temporal fusion operation and a mask refinement feed- forward CNN, incrementally inferring video object segmentation. 3. The main contributions are: (a) The Authors’ proposed spatio-temporal MRF model for video object segmentation encodes spatial dependency by CNNs trained for objects of 4 Research - CNN in MRF: Video Object Segmentation Research by Assaf Spanier Computer Vision News An attempt to mine the higher-order potentials of combining MRF/CRF with CNNs, by embedding a feed-forward pass of a CNN inside the inference of an MRF model