arXiv subjects

Michael Fischer

Publications and source records attributed to Michael Fischer.

At least 19 recordsLinked to original sources

arXiv2026

Overpainting: Localized Context-aware Diffusion Image Editing

We present "overpainting", an image editing operation which offers both control over the location of the edit and awareness of the previous content in that location. The overpainted area is given by a trimap, where white-annotated pixels must be edited, gray-annotated pixels may be edited, and black-annotated pixels must not be edited. This enables both precise and loose control, depending on user intent. We implement overpainting by adapting a pretrained image editing diffusion model using a combination of joint attention and low-rank adaption across input images with attention-dropout to balance the information flow between noise, source and mask images. We present a novel, automated, training data generation pipeline that (1) generates a set of candidate image pairs leveraging existing language-based editing models, (2) carefully curates those pairs, and (3) extracts a trimap from each usable pair. We demonstrate the versatility of our overpainting model on a wide range of editing tasks.

cs.CV↗

arXiv2026

Feature-Guided Diffusion for Non-Differentiable Inverse Rendering

Inverse rendering is traditionally solved via differentiable renderers and gradient descent, which requires substantial problem-specific engineering and is prone to getting stuck in local minima due to ambiguities. Derivative-free approaches alleviate engineering requirements, but often heavily depend on a good problem initialization. In this work, we propose Feature-Informed Diffusion Evolution (FIDE), a fully black-box framework that requires no gradients or specific initialization: the renderer is treated as an opaque function whose only requirement is to produce images. Our key insight is feature guiding: rather than reducing each candidate rendering to a scalar loss value, we use a Vision Transformer (ViT) to extract dense visual features from it. We subsequently use these features to train a diffusion-based candidate proposal model, allowing the network to use visual cues to predict parameters that would match the target image. The candidate solutions proposed by this diffusion model are then refined in a closed loop with a CMA evolution strategy, continuously narrowing the proposal region as optimization progresses. We validate across diverse inverse problems from path tracing, vector splines, Voronoi shaders, and robotics, and demonstrate that feature-guiding substantially improves convergence over scalar-loss baselines and reliably escapes local minima where gradient-based methods stall.

cs.GR↗

arXiv2026

ResEdit: Residual embeddings for precise generative image editing

Conditional diffusion image generators can be repurposed for editing through inversion, without the need for large-scale paired fine-tuning data. However, producing high-quality, targeted edits while maintaining image identity and global consistency remains challenging, as weakly conditioned inversion often embeds conflicting image features into the noise. We demonstrate that incorporating a residual image encoding as additional conditioning enables both improved identity preservation and better editability. We optimize this residual encoding to provide a strong conditioning signal for reconstruction, thereby reducing the reliance on inversion and susceptibility to its aforementioned pitfalls. To ensure this residual does not interfere with desired edits, we incorporate a gradient reversal-based optimization strategy that disentangles the residual from the edited condition. We illustrate our method's ability to produce high-fidelity results across precise intrinsic-based editing and relighting, and show proof-of-concept text-guided manipulation.

cs.CV↗

arXiv2026

MAOAM: Unified Object and Material Selection with Vision-Language Models

Selection is a core operation in interactive image editing. To be practical, a user should be able to specify and disambiguate the desired selection region through either text or click-based interactions, and the system should support selecting not only objects but also other criteria, such as materials. Material-based selection is valuable for tasks like re-texturing surfaces or editing instances of a specific material. However, existing vision-language-model (VLM) based selection methods are object-centric and typically support a single interaction modality, limiting their applicability. In this work, we thus present Mask Any Object And Material (MAOAM), a unified selection framework that enables precise object and material-level selection across both text- and click-based interactions. MAOAM leverages a VLM with a segmentation head to produce pixel-accurate masks from user prompts: the VLM interprets the user's selection intent (object or material-level) and encodes visual entities, attributes, and spatial relations, while the segmentation head decodes the output token into a mask. A key challenge is the lack of material selection datasets with text annotations. We propose a scalable data generation pipeline: we collect real and synthetic images with material masks, and leverage VLMs to generate material descriptions with rich visual-semantics. We train MAOAM with a multi-task objective over click and text-based selection, along with an auxiliary VQA task derived from the material descriptions to facilitate deeper material understanding. Despite being trained with uni-modal prompts, our model exhibits an emergent improvement in selection when combining text and clicks at inference, enabling flexible image editing workflows. Experiments demonstrate accurate and coherent selections across diverse objects, materials, and interaction scenarios, highlighting robustness in practice.

cs.CV↗

arXiv2025

UniLight: A Unified Representation for Lighting

Lighting has a strong influence on visual appearance, yet understanding and representing lighting in images remains notoriously difficult. Various lighting representations exist, such as environment maps, irradiance, spherical harmonics, or text, but they are incompatible, which limits cross-modal transfer. We thus propose UniLight, a joint latent space as lighting representation, that unifies multiple modalities within a shared embedding. Modality-specific encoders for text, images, irradiance, and environment maps are trained contrastively to align their representations, with an auxiliary spherical-harmonics prediction task reinforcing directional understanding. Our multi-modal data pipeline enables large-scale training and evaluation across three tasks: lighting-based retrieval, environment-map generation, and lighting control in diffusion-based image synthesis. Experiments show that our representation captures consistent and transferable lighting features, enabling flexible manipulation across modalities.

cs.CV↗

arXiv2025

Fine-Grained Spatially Varying Material Selection in Images

Selection is the first step in many image editing processes, enabling faster and simpler modifications of all pixels sharing a common modality. In this work, we present a method for material selection in images, robust to lighting and reflectance variations, which can be used for downstream editing tasks. We rely on vision transformer (ViT) models and leverage their features for selection, proposing a multi-resolution processing strategy that yields finer and more stable selection results than prior methods. Furthermore, we enable selection at two levels: texture and subtexture, leveraging a new two-level material selection (DuMaS) dataset which includes dense annotations for over 800,000 synthetic images, both on the texture and subtexture levels.

cs.GR↗

arXiv2025

Bootstrapping Corner Cases: High-Resolution Inpainting for Safety Critical Detect and Avoid for Automated Flying

Modern machine learning techniques have shown tremendous potential, especially for object detection on camera images. For this reason, they are also used to enable safety-critical automated processes such as autonomous drone flights. We present a study on object detection for Detect and Avoid, a safety critical function for drones that detects air traffic during automated flights for safety reasons. An ill-posed problem is the generation of good and especially large data sets, since detection itself is the corner case. Most models suffer from limited ground truth in raw data, \eg recorded air traffic or frontal flight with a small aircraft. It often leads to poor and critical detection rates. We overcome this problem by using inpainting methods to bootstrap the dataset such that it explicitly contains the corner cases of the raw data. We provide an overview of inpainting methods and generative models and present an example pipeline given a small annotated dataset. We validate our method by generating a high-resolution dataset, which we make publicly available and present it to an independent object detector that was fully trained on real data.

cs.CV↗

arXiv2024

Stochastic Gradient Estimation for Higher-order Differentiable Rendering

We derive methods to compute higher order differentials (Hessians and Hessian-vector products) of the rendering operator. Our approach is based on importance sampling of a convolution that represents the differentials of rendering parameters and shows to be applicable to both rasterization and path tracing. We further suggest an aggregate sampling strategy to importance-sample multiple dimensions of one convolution kernel simultaneously. We demonstrate that this information improves convergence when used in higher-order optimizers such as Newton or Conjugate Gradient relative to a gradient descent baseline in several inverse rendering tasks.

cs.GR↗

arXiv2024

SAMa: Material-aware 3D Selection and Segmentation

Decomposing 3D assets into material parts is a common task for artists, yet remains a highly manual process. In this work, we introduce Select Any Material (SAMa), a material selection approach for in-the-wild objects in arbitrary 3D representations. Building on SAM2's video prior, we construct a material-centric video dataset that extends it to the material domain. We propose an efficient way to lift the model's 2D predictions to 3D by projecting each view into an intermediary 3D point cloud using depth. Nearest-neighbor lookups between any 3D representation and this similarity point cloud allow us to efficiently reconstruct accurate selection masks over objects' surfaces that can be inspected from any view. Our method is multiview-consistent by design, alleviating the need for costly per-asset optimization, and performs optimization-free selection in seconds. SAMa outperforms several strong baselines in selection accuracy and multiview consistency and enables various compelling applications, such as replacing the diffuse-textured materials on a text-to-3D output with PBR materials or selecting and editing materials on NeRFs and 3DGS captures.

cs.CV↗

arXiv2024

$^{27}$Al NMR spectroscopic and DFT computational study of the quadrupole coupling of aluminium in two polymorphs of the complex aluminium hydride CsAlH4

The quadrupole coupling constant $C_{\text{Q}}$ and the asymmetry parameter $\eta$ of the aluminium nuclei in two polymorphs of the complex aluminium hydride CsAlH4 are determined from both $^{27}$Al MAS NMR spectra and $^{27}$Al NMR spectra recorded for stationary samples by using the Solomon echo sequence. The accuracy with which these parameters can be determined from the static spectra (CsAlH4(o): $C_{\text{Q}}=(1.42\pm0.01)$ MHz, $\eta=(0.62\pm0.01)$ and CsAlH4(t): $C_{\text{Q}}=(1.43\pm0.02)$ MHz, $\eta<0.03$) seems to be slightly higher than via the MAS approach. The experimentally determined parameters ($\delta_{\text{iso}}$, $C_{\text{Q}}$ and $\eta$) are compared with those obtained from DFT-GIPAW (density functional theory - gauge-including projected augmented wave) calculations. When using DFT-optimized structures, the magnitude of the quadrupole coupling constant is overestimated by about 45% for both polymorphs. Further calculations in which the geometry of the AlH4 tetrahedra was varied show a high sensitivity of $C_{\text{Q}}$ on the H--Al--H angles in particular. Modest changes in the angles on the order of one to three degrees are sufficient to achieve near-perfect agreement between GIPAW calculations and experiment. The deviations found for the DFT-optimized structures are explained with the neglect of thermal motion, which typically leads to a reduction of distortions of the AlH4 tetrahedra. From a broader perspective, the uncertainty in the positions of the hydrogen atoms renders the accurate reproduction or prediction of quadrupole coupling constants for aluminium hydrides challenging.

cond-mat.mtrl-sci↗

arXiv2024

NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs

A Neural Radiance Field (NeRF) encodes the specific relation of 3D geometry and appearance of a scene. We here ask the question whether we can transfer the appearance from a source NeRF onto a target 3D geometry in a semantically meaningful way, such that the resulting new NeRF retains the target geometry but has an appearance that is an analogy to the source NeRF. To this end, we generalize classic image analogies from 2D images to NeRFs. We leverage correspondence transfer along semantic affinity that is driven by semantic features from large, pre-trained 2D image models to achieve multi-view consistent appearance transfer. Our method allows exploring the mix-and-match product space of 3D geometry and appearance. We show that our method outperforms traditional stylization-based methods and that a large majority of users prefer our method over several typical baselines.

cs.CV↗

arXiv2023

Neural Bounding

Bounding volumes are an established concept in computer graphics and vision tasks but have seen little change since their early inception. In this work, we study the use of neural networks as bounding volumes. Our key observation is that bounding, which so far has primarily been considered a problem of computational geometry, can be redefined as a problem of learning to classify space into free or occupied. This learning-based approach is particularly advantageous in high-dimensional spaces, such as animated scenes with complex queries, where neural networks are known to excel. However, unlocking neural bounding requires a twist: allowing -- but also limiting -- false positives, while ensuring that the number of false negatives is strictly zero. We enable such tight and conservative results using a dynamically-weighted asymmetric loss function. Our results show that our neural bounding produces up to an order of magnitude fewer false positives than traditional methods. In addition, we propose an extension of our bounding method using early exits that accelerates query speeds by 25%. We also demonstrate that our approach is applicable to non-deep learning models that train within seconds. Our project page is at: https://wenxin-liu.github.io/neural_bounding/.

cs.GR↗

arXiv2023

Zero Grads: Learning Local Surrogate Losses for Non-Differentiable Graphics

Gradient-based optimization is now ubiquitous across graphics, but unfortunately can not be applied to problems with undefined or zero gradients. To circumvent this issue, the loss function can be manually replaced by a ``surrogate'' that has similar minima but is differentiable. Our proposed framework, ZeroGrads, automates this process by learning a neural approximation of the objective function, which in turn can be used to differentiate through arbitrary black-box graphics pipelines. We train the surrogate on an actively smoothed version of the objective and encourage locality, focusing the surrogate's capacity on what matters at the current training episode. The fitting is performed online, alongside the parameter optimization, and self-supervised, without pre-computed data or pre-trained models. As sampling the objective is expensive (it requires a full rendering or simulator run), we devise an efficient sampling scheme that allows for tractable run-times and competitive performance at little overhead. We demonstrate optimizing diverse non-convex, non-differentiable black-box problems in graphics, such as visibility in rendering, discrete parameter spaces in procedural modelling or optimal control in physics-driven animation. In contrast to other derivative-free algorithms, our approach scales well to higher dimensions, which we demonstrate on problems with up to 35k interlinked variables.

cs.CV↗

arXiv2023

One-dimensional short-range nearest-neighbor interaction and its nonlinear diffusion limit

Repulsion between individuals within a finite radius is encountered in numerous applications, including cell exclusion, i.e. avoidance of overlapping cells, bird flocks, or microscopic pedestrian models. We define such individual based particle dynamics in one spatial dimension with minimal assumptions of the repulsion force f as well as their external velocity v and prove their characteristic properties. Moreover, we are able to perform a rigorous limit from the microscopic to the macroscopic scale, where we could recover the finite interaction radius as a density threshold. Specific choices for the repulsion force f lead to well known nonlinear diffusion equations on the macroscopic scale, as e.g. the porous medium equation. At both scaling levels numerical simulations are presented and compared to underline the analytical results.

math.AP↗

arXiv2022

Plateau-reduced Differentiable Path Tracing

Current differentiable renderers provide light transport gradients with respect to arbitrary scene parameters. However, the mere existence of these gradients does not guarantee useful update steps in an optimization. Instead, inverse rendering might not converge due to inherent plateaus, i.e., regions of zero gradient, in the objective function. We propose to alleviate this by convolving the high-dimensional rendering function that maps scene parameters to images with an additional kernel that blurs the parameter space. We describe two Monte Carlo estimators to compute plateau-free gradients efficiently, i.e., with low variance, and show that these translate into net-gains in optimization error and runtime performance. Our approach is a straightforward extension to both black-box and differentiable renderers and enables optimization of problems with intricate light transport, such as caustics or global illumination, that existing differentiable renderers do not converge on.

cs.CV↗

arXiv2022

On the use of Solomon echoes in 27Al NMR studies of complex aluminium hydrides

The quadrupole coupling constant $C_Q$ and the asymmetry parameter $\eta$ have been determined for two complex aluminium hydrides from $^{27}$Al NMR spectra recorded for stationary samples by using the Solomon echo sequence. The thus obtained data for KAlH4 ($C_Q=(1.30\pm0.02)$ MHz, $\eta=(0.64\pm0.02)$) and NaAlH4 ($C_Q=(3.11\pm0.02)$ MHz, $\eta<0.01$) agree very well with data previously determined from MAS NMR spectra. The accuracy with which these parameters can be determined from static spectra turned out to be at least as good as via the MAS approach. The experimentally determined parameters ($\delta_{iso}$, $C_Q$ and $\eta$) are compared with those obtained from DFT-GIPAW (density functional theory - gauge-including projected augmented wave) calculations. Except for the quadrupole coupling constant for KAlH4, which is overestimated in the GIPAW calculations by about 30%, the agreement is excellent. Advantages of the application of the Solomon echo sequence for the measurement of less stable materials or for in-situ studies are discussed.

cond-mat.mtrl-sci↗

arXiv2022

Learning to Learn and Sample BRDFs

We propose a method to accelerate the joint process of physically acquiring and learning neural Bi-directional Reflectance Distribution Function (BRDF) models. While BRDF learning alone can be accelerated by meta-learning, acquisition remains slow as it relies on a mechanical process. We show that meta-learning can be extended to optimize the physical sampling pattern, too. After our method has been meta-trained for a set of fully-sampled BRDFs, it is able to quickly train on new BRDFs with up to five orders of magnitude fewer physical acquisition samples at similar quality. Our approach also extends to other linear and non-linear BRDF models, which we show in an extensive evaluation.

cs.GR↗

arXiv2022

Quantum behavior of a superconducting Duffing oscillator at the dissipative phase transition

Understanding the non-deterministic behavior of deterministic nonlinear systems has been an implicit dream since Lorenz named it the "butterfly effect". A prominent example is the hysteresis and bistability of the Duffing oscillator, which in the classical description is attributed to the coexistence of two steady states in a double-well potential. However, this interpretation fails in the quantum-mechanical perspective, where a single unique steady state is allowed in the whole parameter space. Here, we measure the non-equilibrium dynamics of a superconducting Duffing oscillator and reconcile the classical and quantum descriptions in a unified picture of quantum metastability. We demonstrate that the two classically regarded steady states are in fact metastable states. They have a remarkably long lifetime in the classical hysteresis regime but must eventually relax into a single unique steady state allowed by quantum mechanics. By engineering the lifetime of the metastable states sufficiently large, we observe a first-order dissipative phase transition, which mimics a sudden change of the mean field in a 11-site Bose-Hubbard lattice. We also reveal the two distinct phases of the transition by quantum state tomography, namely a coherent-state phase and a squeezed-state phase separated by a critical point. Our results reveal a smooth quantum state evolution behind a sudden dissipative phase transition, and they form an essential step towards understanding hysteresis and instability in non-equilibrium systems.

quant-ph↗