arXiv subjects

Sarah Latus

Publications and source records attributed to Sarah Latus.

At least 19 recordsLinked to original sources

arXiv2026

Point Tracking in Surgery--The 2025 Surgical Tattoos in Infrared Challenge (STIRC2025)

Point tracking in surgery is crucial to enable applications in downstream tasks such as segmentation, 3D reconstruction, virtual tissue landmarking, autonomous probe-based scanning, and subtask autonomy. This paper introduces the 2025 iteration of a point tracking challenge to address this, wherein participants submit their algorithms for quantification. Their algorithms are evaluated using a dataset named surgical tattoos in infrared (STIR), with the challenge named the STIR Challenge 2025 (STIRC2025). The STIR Challenge 2025 comprises two quantitative components: accuracy and efficiency. The accuracy component tests the accuracy of algorithms on in vivo and ex vivo sequences. The efficiency component tests algorithm inference latency. The challenge was conducted as a part of MICCAI EndoVis 2025, and seven teams participated in this challenge. In this paper we summarize the challenge results and participant methods. The challenge dataset is available at: https://zenodo.org/records/20191078, and the code for baseline models and metrics calculation is available here: https://github.com/athaddius/STIRMetrics

cs.CV↗

arXiv2025

Automated Estimation of Anatomical Risk Metrics for Endoscopic Sinus Surgery Using Deep Learning

Endoscopic sinus surgery requires careful preoperative assessment of the skull base anatomy to minimize risks such as cerebrospinal fluid leakage. Anatomical risk scores like the Keros, Gera and Thailand-Malaysia-Singapore score offer a standardized approach but require time-consuming manual measurements on coronal CT or CBCT scans. We propose an automated deep learning pipeline that estimates these risk scores by localizing key anatomical landmarks via heatmap regression. We compare a direct approach to a specialized global-to-local learning strategy and find mean absolute errors on the relevant anatomical measurements of 0.506mm for the Keros, 4.516{\deg} for the Gera and 0.802mm / 0.777mm for the TMS classification.

cs.CV↗

arXiv2025

Deep Learning for High Speed Optical Coherence Elastography with a Fiber Scanning Endoscope

Tissue stiffness is related to soft tissue pathologies and can be assessed through palpation or via clinical imaging systems, e.g., ultrasound or magnetic resonance imaging. Typically, the image based approaches are not suitable during interventions, particularly for minimally invasive surgery. To this end, we present a miniaturized fiber scanning endoscope for fast and localized elastography. Moreover, we propose a deep learning based signal processing pipeline to account for the intricate data and the need for real-time estimates. Our elasticity estimation approach is based on imaging complex and diffuse wave fields that encompass multiple wave frequencies and propagate in various directions. We optimize the probe design to enable different scan patterns. To maximize temporal sampling while maintaining three-dimensional information we define a scan pattern in a conical shape with a temporal frequency of 5.05 kHz. To efficiently process the image sequences of complex wave fields we consider a spatio-temporal deep learning network. We train the network in an end-to-end fashion on measurements from phantoms representing multiple elasticities. The network is used to obtain localized and robust elasticity estimates, allowing to create elasticity maps in real-time. For 2D scanning, our approach results in a mean absolute error of 6.31+-5.76 kPa compared to 11.33+-12.78 kPa for conventional phase tracking. For scanning without estimating the wave direction, the novel 3D method reduces the error to 4.48+-3.63 kPa compared to 19.75+-21.82 kPa for the conventional 2D method. Finally, we demonstrate feasibility of elasticity estimates in ex-vivo porcine tissue.

eess.SP↗

arXiv2025

Tracking Any Point Methods for Markerless 3D Tissue Tracking in Endoscopic Stereo Images

Minimally invasive surgery presents challenges such as dynamic tissue motion and a limited field of view. Accurate tissue tracking has the potential to support surgical guidance, improve safety by helping avoid damage to sensitive structures, and enable context-aware robotic assistance during complex procedures. In this work, we propose a novel method for markerless 3D tissue tracking by leveraging 2D Tracking Any Point (TAP) networks. Our method combines two CoTracker models, one for temporal tracking and one for stereo matching, to estimate 3D motion from stereo endoscopic images. We evaluate the system using a clinical laparoscopic setup and a robotic arm simulating tissue motion, with experiments conducted on a synthetic 3D-printed phantom and a chicken tissue phantom. Tracking on the chicken tissue phantom yielded more reliable results, with Euclidean distance errors as low as 1.1 mm at a velocity of 10 mm/s. These findings highlight the potential of TAP-based models for accurate, markerless 3D tracking in challenging surgical scenarios.

cs.CV↗

arXiv2025

Do We Need Pre-Processing for Deep Learning Based Ultrasound Shear Wave Elastography?

Estimating the elasticity of soft tissue can provide useful information for various diagnostic applications. Ultrasound shear wave elastography offers a non-invasive approach. However, its generalizability and standardization across different systems and processing pipelines remain limited. Considering the influence of image processing on ultrasound based diagnostics, recent literature has discussed the impact of different image processing steps on reliable and reproducible elasticity analysis. In this work, we investigate the need of ultrasound pre-processing steps for deep learning-based ultrasound shear wave elastography. We evaluate the performance of a 3D convolutional neural network in predicting shear wave velocities from spatio-temporal ultrasound images, studying different degrees of pre-processing on the input images, ranging from fully beamformed and filtered ultrasound images to raw radiofrequency data. We compare the predictions from our deep learning approach to a conventional time-of-flight method across four gelatin phantoms with different elasticity levels. Our results demonstrate statistically significant differences in the predicted shear wave velocity among all elasticity groups, regardless of the degree of pre-processing. Although pre-processing slightly improves performance metrics, our results show that the deep learning approach can reliably differentiate between elasticity groups using raw, unprocessed radiofrequency data. These results show that deep learning-based approaches could reduce the need for and the bias of traditional ultrasound pre-processing steps in ultrasound shear wave elastography, enabling faster and more reliable clinical elasticity assessments.

eess.IV↗

arXiv2025

A Mobile Robotic Approach to Autonomous Surface Scanning in Legal Medicine

Purpose: Comprehensive legal medicine documentation includes both an internal but also an external examination of the corpse. Typically, this documentation is conducted manually during conventional autopsy. A systematic digital documentation would be desirable, especially for the external examination of wounds, which is becoming more relevant for legal medicine analysis. For this purpose, RGB surface scanning has been introduced. While a manual full surface scan using a handheld camera is timeconsuming and operator dependent, floor or ceiling mounted robotic systems require substantial space and a dedicated room. Hence, we consider whether a mobile robotic system can be used for external documentation. Methods: We develop a mobile robotic system that enables full-body RGB-D surface scanning. Our work includes a detailed configuration space analysis to identify the environmental parameters that need to be considered to successfully perform a surface scan. We validate our findings through an experimental study in the lab and demonstrate the system's application in a legal medicine environment. Results: Our configuration space analysis shows that a good trade-off between coverage and time is reached with three robot base positions, leading to a coverage of 94.96 %. Experiments validate the effectiveness of the system in accurately capturing body surface geometry with an average surface coverage of 96.90 +- 3.16 % and 92.45 +- 1.43 % for a body phantom and actual corpses, respectively. Conclusion: This work demonstrates the potential of a mobile robotic system to automate RGB-D surface scanning in legal medicine, complementing the use of post-mortem CT scans for inner documentation. Our results indicate that the proposed system can contribute to more efficient and autonomous legal medicine documentation, reducing the need for manual intervention.

cs.RO↗

arXiv2024

A Calibration Approach for Elasticity Estimation with Medical Tools

Soft tissue elasticity is directly related to different stages of diseases and can be used for tissue identification during minimally invasive procedures. By palpating a tissue with a robot in a minimally invasive fashion force-displacement curves can be acquired. However, force-displacement curves strongly depend on the tool geometry which is often complex in the case of medical tools. Hence, a tool calibration procedure is desired to directly map force-displacement curves to the corresponding tissue elasticity.We present an experimental setup for calibrating medical tools with a robot. First, we propose to estimate the elasticity of gelatin phantoms by spherical indentation with a state-of-the-art contact model. We estimate force-displacement curves for different gelatin elasticities and temperatures. Our experiments demonstrate that gelatin elasticity is highly dependent on temperature, which can lead to an elasticity offset if not considered. Second, we propose to use a more complex material model, e.g., a neural network, that can be trained with the determined elasticities. Considering the temperature of the gelatin sample we can represent different elasticities per phantom and thereby increase our training data.We report elasticity values ranging from 10 to 40 kPa for a 10% gelatin phantom, depending on temperature.

q-bio.TO↗

arXiv2024

A Modified da Vinci Surgical Instrument for OCE based Elasticity Estimation with Deep Learning

Robot-assisted surgery has advantages compared to conventional laparoscopic procedures, e.g., precise movement of the surgical instruments, improved dexterity, and high-resolution visualization of the surgical field. However, mechanical tissue properties may provide additional information, e.g., on the location of lesions or vessels. While elastographic imaging has been proposed, it is not readily available as an online modality during robot-assisted surgery. We propose modifying a da~Vinci surgical instrument to realize optical coherence elastography (OCE) for quantitative elasticity estimation. The modified da~Vinci instrument is equipped with piezoelectric elements for shear wave excitation and we employ fast optical coherence tomography (OCT) imaging to track propagating wave fields, which are directly related to biomechanical tissue properties. All high-voltage components are mounted at the proximal end outside the patient. We demonstrate that external excitation at the instrument shaft can effectively stimulate shear waves, even when considering damping. Comparing conventional and deep learning-based signal processing, resulting in mean absolute errors of 19.27 kPa and 6.29 kPa, respectively. These results illustrate that precise quantitative elasticity estimates can be obtained. We also demonstrate quantitative elasticity estimation on ex-vivo tissue samples of heart, liver and stomach, and show that the measurements can be used to distinguish soft and stiff tissue types.

eess.IV↗

arXiv2023

Collaborative Robotic Biopsy with Trajectory Guidance and Needle Tip Force Feedback

The diagnostic value of biopsies is highly dependent on the placement of needles. Robotic trajectory guidance has been shown to improve needle positioning, but feedback for real-time navigation is limited. Haptic display of needle tip forces can provide rich feedback for needle navigation by enabling localization of tissue structures along the insertion path. We present a collaborative robotic biopsy system that combines trajectory guidance with kinesthetic feedback to assist the physician in needle placement. The robot aligns the needle while the insertion is performed in collaboration with a medical expert who controls the needle position on site. We present a needle design that senses forces at the needle tip based on optical coherence tomography and machine learning for real-time data processing. Our robotic setup allows operators to sense deep tissue interfaces independent of frictional forces to improve needle placement relative to a desired target structure. We first evaluate needle tip force sensing in ex-vivo tissue in a phantom study. We characterize the tip forces during insertions with constant velocity and demonstrate the ability to detect tissue interfaces in a collaborative user study. Participants are able to detect 91% of ex-vivo tissue interfaces based on needle tip force feedback alone. Finally, we demonstrate that even smaller, deep target structures can be accurately sampled by performing post-mortem in situ biopsies of the pancreas.

cs.RO↗

arXiv2023

Tissue Classification During Needle Insertion Using Self-Supervised Contrastive Learning and Optical Coherence Tomography

Needle positioning is essential for various medical applications such as epidural anaesthesia. Physicians rely on their instincts while navigating the needle in epidural spaces. Thereby, identifying the tissue structures may be helpful to the physician as they can provide additional feedback in the needle insertion process. To this end, we propose a deep neural network that classifies the tissues from the phase and intensity data of complex OCT signals acquired at the needle tip. We investigate the performance of the deep neural network in a limited labelled dataset scenario and propose a novel contrastive pretraining strategy that learns invariant representation for phase and intensity data. We show that with 10% of the training set, our proposed pretraining strategy helps the model achieve an F1 score of 0.84 whereas the model achieves an F1 score of 0.60 without it. Further, we analyse the importance of phase and intensity individually towards tissue classification.

eess.IV↗

arXiv2022

Ultrasound Shear Wave Elasticity Imaging with Spatio-Temporal Deep Learning

Ultrasound shear wave elasticity imaging is a valuable tool for quantifying the elastic properties of tissue. Typically, the shear wave velocity is derived and mapped to an elasticity value, which neglects information such as the shape of the propagating shear wave or push sequence characteristics. We present 3D spatio-temporal CNNs for fast local elasticity estimation from ultrasound data. This approach is based on retrieving elastic properties from shear wave propagation within small local regions. A large training data set is acquired with a robot from homogeneous gelatin phantoms ranging from 17.42 kPa to 126.05 kPa with various push locations. The results show that our approach can estimate elastic properties on a pixelwise basis with a mean absolute error of 5.01+-4.37 kPa. Furthermore, we estimate local elasticity independent of the push location and can even perform accurate estimates inside the push region. For phantoms with embedded inclusions, we report a 53.93% lower MAE (7.50 kPa) and on the background of 85.24% (1.64 kPa) compared to a conventional shear wave method. Overall, our method offers fast local estimations of elastic properties with small spatio-temporal window sizes.

eess.IV↗

arXiv2021

A novel optical needle probe for deep learning-based tissue elasticity characterization

The distinction between malignant and benign tumors is essential to the treatment of cancer. The tissue's elasticity can be used as an indicator for the required tissue characterization. Optical coherence elastography (OCE) probes have been proposed for needle insertions but have so far lacked the necessary load sensing capabilities. We present a novel OCE needle probe that provides simultaneous optical coherence tomography (OCT) imaging and load sensing at the needle tip. We demonstrate the application of the needle probe in indentation experiments on gelatin phantoms with varying gelatin concentrations. We further implement two deep learning methods for the end-to-end sample characterization from the acquired OCT data. We report the estimation of gelatin sample concentrations in unseen samples with a mean error of $1.21 \pm 0.91$ wt\%. Both evaluated deep learning models successfully provide sample characterization with different advantages regarding the accuracy and inference time.

eess.IV↗

arXiv2020

Deep Learning for High Speed Optical Coherence Elastography

Mechanical properties of tissue provide valuable information for identifying lesions. One approach to obtain quantitative estimates of elastic properties is shear wave elastography with optical coherence elastography (OCE). However, given the shear wave velocity, it is still difficult to estimate elastic properties. Hence, we propose deep learning to directly predict elastic tissue properties from OCE data. We acquire 2D images with a frame rate of 30 kHz and use convolutional neural networks to predict gelatin concentration, which we use as a surrogate for tissue elasticity. We compare our deep learning approach to predictions from conventional regression models, using the shear wave velocity as a feature. Mean absolut prediction errors for the conventional approaches range from 1.32$\pm$0.98 p.p. to 1.57$\pm$1.30 p.p. whereas we report an error of 0.90$\pm$0.84 p.p for the convolutional neural network with 3D spatio-temporal input. Our results indicate that deep learning on spatio-temporal data outperforms elastography based on explicit shear wave velocity estimation.

eess.IV↗

arXiv2019

In-Vitro MPI-Guided IVOCT Catheter Tracking in Real Time for Motion Artifact Compensation

Purpose: Using 4D magnetic particle imaging (MPI), intravascular optical coherence tomography (IVOCT) catheters are tracked in real time in order to compensate for image artifacts related to relative motion. Our approach demonstrates the feasibility for bimodal IVOCT and MPI in-vitro experiments. Material and Methods: During IVOCT imaging of a stenosis phantom the catheter is tracked using MPI. A 4D trajectory of the catheter tip is determined from the MPI data using center of mass sub-voxel strategies. A custom built IVOCT imaging adapter is used to perform different catheter motion profiles: no motion artifacts, motion artifacts due to catheter bending, and heart beat motion artifacts. Two IVOCT volume reconstruction methods are compared qualitatively and quantitatively using the DICE metric and the known stenosis length. Results: The MPI-tracked trajectory of the IVOCT catheter is validated in multiple repeated measurements calculating the absolute mean error and standard deviation. Both volume reconstruction methods are compared and analyzed whether they are capable of compensating the motion artifacts. The novel approach of MPI-guided catheter tracking corrects motion artifacts leading to a DICE coefficient with a minimum of 86% in comparison to 58% for a standard reconstruction approach. Conclusions: IVOCT catheter tracking with MPI in real time is an auspicious method for radiation free MPI-guided IVOCT interventions. The combination of MPI and IVOCT can help to reduce motion artifacts due to catheter bending and heart beat for optimized IVOCT volume reconstructions.

physics.med-ph↗

arXiv2019

Bimodal intravascular volumetric imaging combining OCT and MPI

Intravascular optical coherence tomography (IVOCT) is a catheter based image modality allowing for high resolution imaging of vessels. It is based on a fast sequential acquisition of A-scans with an axial spatial resolution in the range of 5 to 10 {\mu}m, i.e., one order of magnitude higher than in conventional methods like intravascular ultrasound or computed tomography angiography. However, position and orientation of the catheter in patient coordinates cannot be obtained from the IVOCT measurements alone. Hence, the pose of the catheter needs to be established to correctly reconstruct the three-dimensional vessel shape. Magnetic particle imaging (MPI) is a three-dimensional tomographic, tracer-based and radiation-free image modality providing high temporal resolution with unlimited penetration depth. Volumetric MPI images are angiographic and hence suitable to complement IVOCT as a co-modality. We study simultaneous bimodal IVOCT MPI imaging with the goal of estimating the IVOCT pullback path based on the 3D MPI data. We present a setup to study and evaluate simultaneous IVOCT and MPI image acquisition of differently shaped vessel phantoms. First, the infuence of the MPI tracer concentration on the optical properties required for IVOCT is analyzed. Second, using a concentration allowing for simultaneous imaging, IVOCT and MPI image data is acquired sequentially and simultaneously. Third, the luminal centerline is established from the MPI image volumes and used to estimate the catheter pullback trajectory for IVOCT image reconstruction. The image volumes are compared to the known shape of the phantoms. We were able to identify a suitable MPI tracer concentration of 2.5 mmol/L with negligible influence on the IVOCT signal. The pullback trajectory estimated from MPI agrees well with the centerline of the phantoms. (...)

physics.med-ph↗

arXiv2019

Towards Automatic Lesion Classification in the Upper Aerodigestive Tract Using OCT and Deep Transfer Learning Methods

Early detection of cancer is crucial for treatment and overall patient survival. In the upper aerodigestive tract (UADT) the gold standard for identification of malignant tissue is an invasive biopsy. Recently, non-invasive imaging techniques such as confocal laser microscopy and optical coherence tomography (OCT) have been used for tissue assessment. In particular, in a recent study experts classified lesions in the UADT with respect to their invasiveness using OCT images only. As the results were promising, automatic classification of lesions might be feasible which could assist experts in their decision making. Therefore, we address the problem of automatic lesion classification from OCT images. This task is very challenging as the available dataset is extremely small and the data quality is limited. However, as similar issues are typical in many clinical scenarios we study to what extent deep learning approaches can still be trained and used for decision support.

cs.CV↗

arXiv2019

Endoscopic vs. volumetric OCT imaging of mastoid bone structure for pose estimation in minimally invasive cochlear implant surgery

Purpose: The facial recess is a delicate structure that must be protected in minimally invasive cochlear implant surgery. Current research estimates the drill trajectory by using endoscopy of the unique mastoid patterns. However, missing depth information limits available features for a registration to preoperative CT data. Therefore, this paper evaluates OCT for enhanced imaging of drill holes in mastoid bone and compares OCT data to original endoscopic images. Methods: A catheter-based OCT probe is inserted into a drill trajectory of a mastoid phantom in a translation-rotation manner to acquire the inner surface state. The images are undistorted and stitched to create volumentric data of the drill hole. The mastoid cell pattern is segmented automatically and compared to ground truth. Results: The mastoid pattern segmented on images acquired with OCT show a similarity of J = 73.6 % to ground truth based on endoscopic images and measured with the Jaccard metric. Leveraged by additional depth information, automated segmentation tends to be more robust and fail-safe compared to endoscopic images. Conclusion: The feasibility of using a clinically approved OCT probe for imaging the drill hole in cochlear implantation is shown. The resulting volumentric images provide additional information on the shape of caveties in the bone structure, which will be useful for image-to-patient registration and to estimate the drill trajectory. This will be another step towards safe minimally invasive cochlear implantation.

cs.CV↗

arXiv2018

Bioresorbable Scaffold Visualization in IVOCT Images Using CNNs and Weakly Supervised Localization

Bioresorbable scaffolds have become a popular choice for treatment of coronary heart disease, replacing traditional metal stents. Often, intravascular optical coherence tomography is used to assess potential malapposition after implantation and for follow-up examinations later on. Typically, the scaffold is manually reviewed by an expert, analyzing each of the hundreds of image slices. As this is time consuming, automatic stent detection and visualization approaches have been proposed, mostly for metal stent detection based on classic image processing. As bioresorbable scaffolds are harder to detect, recent approaches have used feature extraction and machine learning methods for automatic detection. However, these methods require detailed, pixel-level labels in each image slice and extensive feature engineering for the particular stent type which might limit the approaches' generalization capabilities. Therefore, we propose a deep learning-based method for bioresorbable scaffold visualization using only image-level labels. A convolutional neural network is trained to predict whether an image slice contains a metal stent, a bioresorbable scaffold, or no device. Then, we derive local stent strut information by employing weakly supervised localization using saliency maps with guided backpropagation. As saliency maps are generally diffuse and noisy, we propose a novel patch-based method with image shifting which allows for high resolution stent visualization. Our convolutional neural network model achieves a classification accuracy of 99.0 % for image-level stent classification which can be used for both high quality in-slice stent visualization and 3D rendering of the stent structure.

cs.CV↗