Computer Vision News - November 2021

3 Summary 25 Dominik Rivoir Best of ICCV 2021 itional rendering pipelines from computer graphics and try to make them differentiable and incorporate neural networks. We take the 3D information that we have from our simulated scene and try to incorporate that into the learning process. ” Obtaining the data needed for this was, as ever, a challenge in the surgical domain, as was finding the corresponding real data and simulated data . The team had to design a simulated scene that somehow resembled the realistic data. For that, they had to build the scenes first and then extract all the 3D information from that and design the model on top. “ We use something called a neural texture, ” Dominik explains. “ We have a 3D representation of the entire scene’s texture. In traditional computer vision, at each texture location in the scene you would store something like the color of the object that you want to render, then how it reflects, something like that. But we want to learn these features and not explicitly define them, so at each spatial location on the texture we have a learnable feature vector , and we project those feature vectors from 3D into 2D space when we want to capture an image. Then we use a neural network to translate these abstract features into an image. Since this whole pipeline including the projection into 3D space is differentiable, we can just learn the network combined with the neural texture together. ” “Neural rendering is where we take traditional rendering pipelines from computer graphics and try to make them differentiable and incorporate neural networks…”