Computer Vision News - June 2021

15 From reviewing recent papers in the field of surgical robotics and computer vision, the authors also analyse and get inspired by the following points: 1. From SOA in Supervised Surgical gesture recognition, they borrow the notion that optical flow is an important source of information for learning to classify surgical gestures. This knowledge is fundamental for the decision to extract optical flow from the data as a source of domain-independent visual information. Moreover, from reviewing papers in this field, they also derive the use of integrating multimodal information to improve supervised gesture recognition results. 2. From papers in Unsupervised Surgical gesture recognition, they analyse the possibility to capture the latent information for surgical gesture recognition using RNNs in an unsupervised manner, and the proof that these embeddings naturally cluster corresponding to distinct higher-level activities. This turns out to be a very important factor in this research as well, even if RNNs are used instead of CNNs. 3. The field of Video activity recognition helps them shaping their final architecture, based on an encoder-decoder network. This is after reviewing different approaches with parallel CNN-based video and optical flow streams or visual attention-based models for surgical activity recognition. 4. From a Multimodal self-supervised learning study, they also get another example of corresponding embeddings for audio and video which cluster close to each other. To combine this knowledge and develop the novel algorithm discussed here, the main data is taken from the JIGSAWS dataset and combines a series of annotated clips of surgical activity (further subdivided in the three categories reported below) and kinematics in the shape of a 76-dimensional vector that includes the x,y,z coordinates of the left and right tool tips, the corresponding linear and angular velocities, the rotation matrix and the gripper angle velocities. Automatic Gesture Recognition in Surgical Robotics Figure 3: suturing Figure 2: needle passing Figure 3: suturing