Computer Vision News - November 2018

a machine to colorize objects in a video and basically it discovers the motion of objects. There’s a bunch of things like this which I think are impressive. I have a question that one of my engineers asked me to ask you. It’s a funny question, but I cannot understand it because I am not an engineer myself. He asks, which one is your favorite between ReLU and batch normalization? Oh okay… [ he laughs ] So, it is funny? It’s funny! It’s a pretty easy answer, but it asks many other questions. I would say ReLU, because it’s a simple idea that everybody uses and it’s the idea that basically allows us to train relatively deep networks. There’s another idea, the ResNet residual connection idea from Kaiming He, that allows us to train even deeper networks. Batch normalization in the mind of many people, including me, is a necessary evil. In the sense that nobody likes it, but it kind of works, so everybody uses it, but everybody is trying to replace it with something else because everybody hates it. There’s something about it that is not entirely satisfying. We’re all under the impression that there’s got to be something better than it. Also, people don’t understand why it works and how it works. There are intuitions that we’ve had about how neural nets converge and learn, and the way batch normalization works doesn’t fit in that picture, so there’s a lot of work to do there to understand why it works and to try to replace it with something else. Kaiming He also came up with this thing called group normalization, which is meant to replace batch normalization and apparently works slightly better. Looking ahead, how do you think academia and industry might work better together in our computer vision and artificial intelligence community? I have spent half my time in academia and half my time in industry in my career. I was initially at Bell Labs – that became AT&T Labs – then I spent 18 months at the NEC Research Institute. I became a professor and now I share my time between industry and academia. I think this idea that you can share your time between industry and academia is good. I actually wrote a piece about it. I have read it. About the double affiliation . That’s right. There is a very important point that I think some people have missed in this piece, which is that this model of dual affiliation only works if - in the industry in which you work - the industry lab is a research lab, not a development lab, and if it practices open research and the company that runs it is not too possessive about Pauline Luc 9 Computer Vision News “This model of dual affiliation only works if - in the industry in which you work - the industry lab is a research lab, not a development lab, and if it practices open research and the company that runs it is not too possessive about intellectual property!” Yann LeCun Guest