arXiv subjects

Nassir Navab

Publications and source records attributed to Nassir Navab.

At least 559 records · Page 31Linked to original sources

arXiv2017

Multi-layer Visualization for Medical Mixed Reality

Medical Mixed Reality helps surgeons to contextualize intraoperative data with video of the surgical scene. Nonetheless, the surgical scene and anatomical target are often occluded by surgical instruments and surgeon hands. In this paper and to our knowledge, we propose a multi-layer visualization in Medical Mixed Reality solution which subtly improves a surgeon's visualization by making transparent the occluding objects. As an example scenario, we use an augmented reality C-arm fluoroscope device. A video image is created using a volumetric-based image synthesization technique and stereo-RGBD cameras mounted on the C-arm. From this synthesized view, the background which is occluded by the surgical instruments and surgeon hands is recovered by modifying the volumetric-based image synthesization technique. The occluding objects can, therefore, become transparent over the surgical scene. Experimentation with different augmented reality scenarios yield results demonstrating that the background of the surgical scenes can be recovered with accuracy between 45%-99%. In conclusion, we presented a solution that a Mixed Reality solution for medicine, providing transparency to objects occluding the surgical scene. This work is also the first application of volumetric field for Diminished Reality/ Mixed Reality.

cs.CV↗

arXiv2017

6D Object Pose Estimation with Depth Images: A Seamless Approach for Robotic Interaction and Augmented Reality

To determine the 3D orientation and 3D location of objects in the surroundings of a camera mounted on a robot or mobile device, we developed two powerful algorithms in object detection and temporal tracking that are combined seamlessly for robotic perception and interaction as well as Augmented Reality (AR). A separate evaluation of, respectively, the object detection and the temporal tracker demonstrates the important stride in research as well as the impact on industrial robotic applications and AR. When evaluated on a standard dataset, the detector produced the highest f1-score with a large margin while the tracker generated the best accuracy at a very low latency of approximately 2 ms per frame with one CPU core: both algorithms outperforming the state of the art. When combined, we achieve a powerful framework that is robust to handle multiple instances of the same object under occlusion and clutter while attaining real-time performance. Aiming at stepping beyond the simple scenarios used by current systems, often constrained by having a single object in absence of clutter, averting to touch the object to prevent close-range partial occlusion, selecting brightly colored objects to easily segment them individually or assuming that the object has simple geometric structure, we demonstrate the capacity to handle challenging cases under clutter, partial occlusion and varying lighting conditions with objects of different shapes and sizes.

cs.CV↗

arXiv2017

The TUM LapChole dataset for the M2CAI 2016 workflow challenge

In this technical report we present our collected dataset of laparoscopic cholecystectomies (LapChole). Laparoscopic videos of a total of 20 surgeries were recorded and annotated with surgical phase labels, of which 15 were randomly pre-determined as training data, while the remaining 5 videos are selected as test data. This dataset was later included as part of the M2CAI 2016 workflow detection challenge during MICCAI 2016 in Athens.

cs.CV↗

arXiv2017

Camera Pose Filtering with Local Regression Geodesics on the Riemannian Manifold of Dual Quaternions

Time-varying, smooth trajectory estimation is of great interest to the vision community for accurate and well behaving 3D systems. In this paper, we propose a novel principal component local regression filter acting directly on the Riemannian manifold of unit dual quaternions $\mathbb{D} \mathbb{H}_1$. We use a numerically stable Lie algebra of the dual quaternions together with $\exp$ and $\log$ operators to locally linearize the 6D pose space. Unlike state of the art path smoothing methods which either operate on $SO\left(3\right)$ of rotation matrices or the hypersphere $\mathbb{H}_1$ of quaternions, we treat the orientation and translation jointly on the dual quaternion quadric in the 7-dimensional real projective space $\mathbb{R}\mathbb{P}^7$. We provide an outlier-robust IRLS algorithm for generic pose filtering exploiting this manifold structure. Besides our theoretical analysis, our experiments on synthetic and real data show the practical advantages of the manifold aware filtering on pose tracking and smoothing.

cs.CV↗

arXiv2017

Learning in an Uncertain World: Representing Ambiguity Through Multiple Hypotheses

Many prediction tasks contain uncertainty. In some cases, uncertainty is inherent in the task itself. In future prediction, for example, many distinct outcomes are equally valid. In other cases, uncertainty arises from the way data is labeled. For example, in object detection, many objects of interest often go unlabeled, and in human pose estimation, occluded joints are often labeled with ambiguous values. In this work we focus on a principled approach for handling such scenarios. In particular, we propose a framework for reformulating existing single-prediction models as multiple hypothesis prediction (MHP) models and an associated meta loss and optimization procedure to train them. To demonstrate our approach, we consider four diverse applications: human pose estimation, future prediction, image classification and segmentation. We find that MHP models outperform their single-hypothesis counterparts in all cases, and that MHP models simultaneously expose valuable insights into the variability of predictions.

cs.CV↗

arXiv2017

Long Short-Term Memory Kalman Filters:Recurrent Neural Estimators for Pose Regularization

One-shot pose estimation for tasks such as body joint localization, camera pose estimation, and object tracking are generally noisy, and temporal filters have been extensively used for regularization. One of the most widely-used methods is the Kalman filter, which is both extremely simple and general. However, Kalman filters require a motion model and measurement model to be specified a priori, which burdens the modeler and simultaneously demands that we use explicit models that are often only crude approximations of reality. For example, in the pose-estimation tasks mentioned above, it is common to use motion models that assume constant velocity or constant acceleration, and we believe that these simplified representations are severely inhibitive. In this work, we propose to instead learn rich, dynamic representations of the motion and noise models. In particular, we propose learning these models from data using long short term memory, which allows representations that depend on all previous observations and all previous states. We evaluate our method using three of the most popular pose estimation tasks in computer vision, and in all cases we obtain state-of-the-art performance.

cs.CV↗

arXiv2017

Concurrent Segmentation and Localization for Tracking of Surgical Instruments

Real-time instrument tracking is a crucial requirement for various computer-assisted interventions. In order to overcome problems such as specular reflections and motion blur, we propose a novel method that takes advantage of the interdependency between localization and segmentation of the surgical tool. In particular, we reformulate the 2D instrument pose estimation as heatmap regression and thereby enable a concurrent, robust and near real-time regression of both tasks via deep learning. As demonstrated by our experimental results, this modeling leads to a significantly improved performance than directly regressing the tool position and allows our method to outperform the state of the art on a Retinal Microsurgery benchmark and the MICCAI EndoVis Challenge 2015.

cs.CV↗

arXiv2017

Semi-Supervised Deep Learning for Fully Convolutional Networks

Deep learning usually requires large amounts of labeled training data, but annotating data is costly and tedious. The framework of semi-supervised learning provides the means to use both labeled data and arbitrary amounts of unlabeled data for training. Recently, semi-supervised deep learning has been intensively studied for standard CNN architectures. However, Fully Convolutional Networks (FCNs) set the state-of-the-art for many image segmentation tasks. To the best of our knowledge, there is no existing semi-supervised learning method for such FCNs yet. We lift the concept of auxiliary manifold embedding for semi-supervised learning to FCNs with the help of Random Feature Embedding. In our experiments on the challenging task of MS Lesion Segmentation, we leverage the proposed framework for the purpose of domain adaptation and report substantial improvements over the baseline model.

cs.CV↗

arXiv2017

ReLayNet: Retinal Layer and Fluid Segmentation of Macular Optical Coherence Tomography using Fully Convolutional Network

Optical coherence tomography (OCT) is used for non-invasive diagnosis of diabetic macular edema assessing the retinal layers. In this paper, we propose a new fully convolutional deep architecture, termed ReLayNet, for end-to-end segmentation of retinal layers and fluid masses in eye OCT scans. ReLayNet uses a contracting path of convolutional blocks (encoders) to learn a hierarchy of contextual features, followed by an expansive path of convolutional blocks (decoders) for semantic segmentation. ReLayNet is trained to optimize a joint loss function comprising of weighted logistic regression and Dice overlap loss. The framework is validated on a publicly available benchmark dataset with comparisons against five state-of-the-art segmentation methods including two deep learning based approaches to substantiate its effectiveness.

cs.CV↗

arXiv2017

Error Corrective Boosting for Learning Fully Convolutional Networks with Limited Data

Training deep fully convolutional neural networks (F-CNNs) for semantic image segmentation requires access to abundant labeled data. While large datasets of unlabeled image data are available in medical applications, access to manually labeled data is very limited. We propose to automatically create auxiliary labels on initially unlabeled data with existing tools and to use them for pre-training. For the subsequent fine-tuning of the network with manually labeled data, we introduce error corrective boosting (ECB), which emphasizes parameter updates on classes with lower accuracy. Furthermore, we introduce SkipDeconv-Net (SD-Net), a new F-CNN architecture for brain segmentation that combines skip connections with the unpooling strategy for upsampling. The SD-Net addresses challenges of severe class imbalance and errors along boundaries. With application to whole-brain MRI T1 scan segmentation, we generate auxiliary labels on a large dataset with FreeSurfer and fine-tune on two datasets with manual annotations. Our results show that the inclusion of auxiliary labels and ECB yields significant improvements. SD-Net segments a 3D scan in 7 secs in comparison to 30 hours for the closest multi-atlas segmentation method, while reaching similar performance. It also outperforms the latest state-of-the-art F-CNN models.

cs.CV↗

arXiv2017

Segmentation of Intracranial Arterial Calcification with Deeply Supervised Residual Dropout Networks

Intracranial carotid artery calcification (ICAC) is a major risk factor for stroke, and might contribute to dementia and cognitive decline. Reliance on time-consuming manual annotation of ICAC hampers much demanded further research into the relationship between ICAC and neurological diseases. Automation of ICAC segmentation is therefore highly desirable, but difficult due to the proximity of the lesions to bony structures with a similar attenuation coefficient. In this paper, we propose a method for automatic segmentation of ICAC; the first to our knowledge. Our method is based on a 3D fully convolutional neural network that we extend with two regularization techniques. Firstly, we use deep supervision (hidden layers supervision) to encourage discriminative features in the hidden layers. Secondly, we augment the network with skip connections, as in the recently developed ResNet, and dropout layers, inserted in a way that skip connections circumvent them. We investigate the effect of skip connections and dropout. In addition, we propose a simple problem-specific modification of the network objective function that restricts the focus to the most important image regions and simplifies the optimization. We train and validate our model using 882 CT scans and test on 1,000. Our regularization techniques and objective improve the average Dice score by 7.1%, yielding an average Dice of 76.2% and 97.7% correlation between predicted ICAC volumes and manual annotations.

cs.CV↗

arXiv2017

Learning-based Surgical Workflow Detection from Intra-Operative Signals

A modern operating room (OR) provides a plethora of advanced medical devices. In order to better facilitate the information offered by them, they need to automatically react to the intra-operative context. To this end, the progress of the surgical workflow must be detected and interpreted, so that the current status can be given in machine-readable form. In this work, Random Forests (RF) and Hidden Markov Models (HMM) are compared and combined to detect the surgical workflow phase of a laparoscopic cholecystectomy. Various combinations of data were tested, from using only raw sensor data to filtered and augmented datasets. Achieved accuracies ranged from 64% to 72% for the RF approach, and from 80% to 82% for the combination of RF and HMM.

cs.LG↗

arXiv2017

CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction

Given the recent advances in depth prediction from Convolutional Neural Networks (CNNs), this paper investigates how predicted depth maps from a deep neural network can be deployed for accurate and dense monocular reconstruction. We propose a method where CNN-predicted dense depth maps are naturally fused together with depth measurements obtained from direct monocular SLAM. Our fusion scheme privileges depth prediction in image locations where monocular SLAM approaches tend to fail, e.g. along low-textured regions, and vice-versa. We demonstrate the use of depth prediction for estimating the absolute scale of the reconstruction, hence overcoming one of the major limitations of monocular SLAM. Finally, we propose a framework to efficiently fuse semantic labels, obtained from a single frame, with dense SLAM, yielding semantically coherent scene reconstruction from a single view. Evaluation results on two benchmark datasets show the robustness and accuracy of our approach.

cs.CV↗

arXiv2017

Deep Residual Learning for Instrument Segmentation in Robotic Surgery

Detection, tracking, and pose estimation of surgical instruments are crucial tasks for computer assistance during minimally invasive robotic surgery. In the majority of cases, the first step is the automatic segmentation of surgical tools. Prior work has focused on binary segmentation, where the objective is to label every pixel in an image as tool or background. We improve upon previous work in two major ways. First, we leverage recent techniques such as deep residual learning and dilated convolutions to advance binary-segmentation performance. Second, we extend the approach to multi-class segmentation, which lets us segment different parts of the tool, in addition to background. We demonstrate the performance of this method on the MICCAI Endoscopic Vision Challenge Robotic Instruments dataset.

cs.CV↗

arXiv2017

X-ray In-Depth Decomposition: Revealing The Latent Structures

X-ray radiography is the most readily available imaging modality and has a broad range of applications that spans from diagnosis to intra-operative guidance in cardiac, orthopedics, and trauma procedures. Proper interpretation of the hidden and obscured anatomy in X-ray images remains a challenge and often requires high radiation dose and imaging from several perspectives. In this work, we aim at decomposing the conventional X-ray image into d X-ray components of independent, non-overlapped, clipped sub-volumes using deep learning approach. Despite the challenging aspects of modeling such a highly ill-posed problem, exciting and encouraging results are obtained paving the path for further contributions in this direction.

cs.CV↗

arXiv2017

Learning Robust Hash Codes for Multiple Instance Image Retrieval

In this paper, for the first time, we introduce a multiple instance (MI) deep hashing technique for learning discriminative hash codes with weak bag-level supervision suited for large-scale retrieval. We learn such hash codes by aggregating deeply learnt hierarchical representations across bag members through a dedicated MI pool layer. For better trainability and retrieval quality, we propose a two-pronged approach that includes robust optimization and training with an auxiliary single instance hashing arm which is down-regulated gradually. We pose retrieval for tumor assessment as an MI problem because tumors often coexist with benign masses and could exhibit complementary signatures when scanned from different anatomical views. Experimental validations on benchmark mammography and histology datasets demonstrate improved retrieval performance over the state-of-the-art methods.

cs.CV↗

arXiv2017

Quantitative Characterization of Components of Computer Assisted Interventions

Purpose: We propose a mathematical framework for quantitative analysis weighting the impact of heterogeneous components of a surgery. While multi-level approaches, surgical process modeling and other workflow analysis methods exist, this is to our knowledge the first quantitative approach. Methods: Inspired by the group decision making problem from the field of operational research, we define event impact factors, which combine independent and very diverse low-level functions. This allows us to rate surgical events by their importance. Results: We conducted surveys with 4 surgeons to determine the importance of roles, phases and their combinations within a laparoscopic cholecystectomy. Applying this data on a recorded surgery, we showed that it is possible to define a quantitative measure for deciding on acception or rejection of calls to different roles and at different phases of surgery. Conclusions: This methodology allows us to use components such as expertise and role of the surgical staff and other aspects of a given surgery in order to quantitatively analyze and evaluate events, actions, user interfaces or procedures.

cs.OH↗

arXiv2017

Surgical Data Science: Enabling Next-Generation Surgery

This paper introduces Surgical Data Science as an emerging scientific discipline. Key perspectives are based on discussions during an intensive two-day international interactive workshop that brought together leading researchers working in the related field of computer and robot assisted interventions. Our consensus opinion is that increasing access to large amounts of complex data, at scale, throughout the patient care process, complemented by advances in data science and machine learning techniques, has set the stage for a new generation of analytics that will support decision-making and quality improvement in interventional medicine. In this article, we provide a consensus definition for Surgical Data Science, identify associated challenges and opportunities and provide a roadmap for advancing the field.

cs.CY↗