arXiv subjects

Benjamin Lee

Publications and source records attributed to Benjamin Lee.

At least 19 recordsLinked to original sources

arXiv2026

Transition Techniques for Externally-Guided Multi-Scale Viewpoint Changes

Extended reality (XR) is increasingly used to help users understand complex virtual environments through multiple viewpoints across different immersion levels, positions, and scales. While numerous techniques address viewpoint transitions for self-guided exploration, many scenarios require externally-guided transitions where a system or presenter controls the user's viewpoint, leaving the user with limited spatial knowledge and control over the transition process, which can increase susceptibility to disorientation and discomfort. We present three transition techniques for externally-guided multi-scale XR viewpoint changes and evaluate them against a fade-to-black baseline in a within-subjects study (N=20). Participants transitioned between world-in-miniature, street-level, and indoor destination views. We combined spatial recall measures, standardized questionnaires, and semi-structured interviews to assess orientation, workload, comfort, and continuity. Results showed that in our setup, techniques externalizing reference frames and the user's pose improved multi-scale spatial recall relative to a fade, while same-scale recall was insensitive to transition technique. We conclude with design implications for multi-scale XR viewpoint transitions.

cs.HC↗

arXiv2026

CrossAtlas: Evaluating Projection Techniques for Spatial Referencing in Cross-Reality Collaboration

Cross-reality collaboration increasingly connects immersive and desktop users within synchronized workspaces, yet little is known about how bidirectional projection techniques between immersive 3D layouts and desktop 2D views influence communication. Spatial referencing depends on shared spatial understanding, but different mappings preserve and distort geometric relationships in different ways, altering perceived adjacency, orientation, and coverage across collaborators' views. We present CrossAtlas, a synchronized PC-VR collaboration platform that integrates multiple bidirectional projection techniques, including three planar projection variants and equirectangular, a spherical projection variant, across layouts of varying curvature. In a controlled study with 24 dyads, collaborators completed spatial referencing tasks under different projection-layout conditions while we collected performance and subjective measures. Our results show that projection choice strongly shaped collaboration, with the spherical variant often outperforming planar projections and remaining robust across object layouts.

cs.HC↗

arXiv2026

Situated Brushing and Linking in Virtual and Augmented Reality

In traditional visual analysis, brushing and linking is commonly used to visually connect multiple views using highlighting techniques. However, brushing and linking has rarely been used in situated analytics, which uses visualizations to analyze data in the context of physical referents. In situated analytics, data representations must be visually linked to real-world objects. Previous work has assessed situated brushing and linking in a virtual reality simulation of a supermarket scenario. Here, we replicate and extend the previous approach by studying brushing and linking in an actual physical space with augmented reality, while further improving the highlighting techniques. Using a video see-through display, we compare augmented reality with virtual reality. Results suggest that AR performs better in time and accuracy, but the effectiveness of the techniques varies by condition. These results provide a new framing of how the real-world stimuli matter in situated analytics.

cs.HC↗

arXiv2025

Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading

KV cache offloading enables long-context LLM inference by storing caches in CPU DRAM, but PCIe bandwidth limitations create severe bottlenecks. In this paper, we develops an analytical framework that derives $\kappa_{\text{crit}}$, the critical cached-to-prefill token ratio where execution becomes memory-bound and show typical workloads exceed this threshold by orders of magnitude. Empirical characterization reveals 99\% of latency spent on transfers and serving offloaded requests results in GPU's consuming only 28\% of their rated TDP, motivating our proposed optimizations for hardware interconnects, model architectures, and scheduling algorithms.

cs.AR↗

arXiv2025

Hybrid User Interfaces: Past, Present, and Future of Complementary Cross-Device Interaction in Mixed Reality

We investigate hybrid user interfaces (HUIs), aiming to establish a cohesive understanding and adopt consistent terminology for this nascent research area. HUIs combine heterogeneous devices in complementary roles, leveraging the distinct benefits of each. Our work focuses on cross-device interaction between 2D devices and mixed reality environments, which are particularly compelling, leveraging the familiarity of traditional 2D platforms while providing spatial awareness and immersion. Although such HUIs have been prominently explored in the context of mixed reality by prior work, we still lack a cohesive understanding of the unique design possibilities and challenges of such combinations, resulting in a fragmented research landscape. We conducted a systematic survey and present a taxonomy of HUIs that combine conventional display technology and mixed reality environments. Based on this, we discuss past and current challenges, the evolution of definitions, and prospective opportunities to tie together the past 30 years of research with our vision of future HUIs.

cs.HC↗

arXiv2025

A Design Space for Visualization Transitions of 3D Spatial Data in Hybrid AR-Desktop Environments

We present a design space for animated transitions of the appearance of 3D spatial datasets in a hybrid Augmented Reality (AR)-desktop context. Such hybrid interfaces combine both traditional and immersive displays to facilitate the exploration of 2D and 3D data representations in the environment in which they are best displayed. One key aspect is to introduce transitional animations that change between the different dimensionalities to illustrate the connection between the different representations and to reduce the potential cognitive load on the user. The specific transitions to be used depend on the type of data, the needs of the application domain, and other factors. We summarize these as a transition design space to simplify the decision-making process and provide inspiration for future designs. First, we discuss 3D visualizations from a spatial perspective: a spatial encoding pipeline, where 3D data sampled from the physical world goes through various transformations, being mapped to visual representations, and then being integrated into a hybrid AR-desktop environment. The transition design then focuses on interpolating between two spatial encoding pipelines to provide a smooth experience. To illustrate the use of our design space, we apply it to three case studies that focus on applications in astronomy, radiology, and chemistry; we then discuss lessons learned from these applications.

cs.GR↗

arXiv2025

Traversing Dual Realities: Investigating Techniques for Transitioning 3D Objects between Desktop and Augmented Reality Environments

Desktop environments can integrate augmented reality (AR) head-worn devices to support 3D representations, visualizations, and interactions in a novel yet familiar setting. As users navigate across the dual realities -- desktop and AR -- a way to move 3D objects between them is needed. We devise three baseline transition techniques based on common approaches in the literature and evaluate their usability and practicality in an initial user study (N=18). After refining both our transition techniques and the surrounding technical setup, we validate the applicability of the overall concept for real-world activities in an expert user study (N=6). In it, computational chemists followed their usual desktop workflows to build, manipulate, and analyze 3D molecular structures, but now aided with the addition of AR and our transition techniques. Based on our findings from both user studies, we provide lessons learned and takeaways for the design of 3D object transition techniques in desktop + AR environments.

cs.HC↗

arXiv2025

Examining the Effects of Immersive and Non-Immersive Presenter Modalities on Engagement and Social Interaction in Co-located Augmented Presentations

Head-worn augmented reality (AR) allows audiences to be immersed and engaged in stories told by live presenters. While presenters may also be in AR to have the same level of immersion and awareness as their audience, this symmetric presentation style may diminish important social cues such as eye contact. In this work, we examine the effects this (a)symmetry has on engagement, group awareness, and social interaction in co-located one-on-one augmented presentations. We developed a presentation system incorporating 2D/3D content that audiences can view and interact with in AR, with presenters controlling and delivering the presentation in either a symmetric style in AR, or an asymmetric style with a handheld tablet. We conducted a within- and between-subjects evaluation with 12 participant pairs to examine the differences between these symmetric and asymmetric presentation modalities. From our findings, we extracted four themes and derived strategies and guidelines for designers interested in augmented presentations.

cs.HC↗

arXiv2024

Visual Highlighting for Situated Brushing and Linking

Brushing and linking is widely used for visual analytics in desktop environments. However, using this approach to link many data items between situated (e.g., a virtual screen with data) and embedded views (e.g., highlighted objects in the physical environment) is largely unexplored. To this end, we study the effectiveness of visual highlighting techniques in helping users identify and link physical referents to brushed data marks in a situated scatterplot. In an exploratory virtual reality user study (N=20), we evaluated four highlighting techniques under different physical layouts and tasks. We discuss the effectiveness of these techniques, as well as implications for the design of brushing and linking operations in situated analytics.

cs.HC↗

arXiv2024

Putting Our Minds Together: Iterative Exploration for Collaborative Mind Mapping

We delineate the development of a mind-mapping system designed concurrently for both VR and desktop platforms. Employing an iterative methodology with groups of users, we systematically examined and improved various facets of our system, including interactions, communication mechanisms and gamification elements, to streamline the mind-mapping process while augmenting situational awareness and promoting active engagement among collaborators. We also report our observational findings on these facets from this iterative design process.

cs.HC↗

arXiv2024

Design Space of Visual Feedforward And Corrective Feedback in XR-Based Motion Guidance Systems

Extended reality (XR) technologies are highly suited in assisting individuals in learning motor skills and movements -- referred to as motion guidance. In motion guidance, the "feedforward" provides instructional cues of the motions that are to be performed, whereas the "feedback" provides cues which help correct mistakes and minimize errors. Designing synergistic feedforward and feedback is vital to providing an effective learning experience, but this interplay between the two has not yet been adequately explored. Based on a survey of the literature, we propose design space for both motion feedforward and corrective feedback in XR, and describe the interaction effects between them. We identify common design approaches of XR-based motion guidance found in our literature corpus, and discuss them through the lens of our design dimensions. We then discuss additional contextual factors and considerations that influence this design, together with future research opportunities for motion guidance in XR.

cs.HC↗

arXiv2023

Gemini: A Family of Highly Capable Multimodal Models

This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models responsibly to users through services including Gemini, Gemini Advanced, Google AI Studio, and Cloud Vertex AI.

cs.CL↗

arXiv2023

LSDvis: Hallucinatory Data Visualisations in Real World Environments

We propose the concept of "LSDvis": the (highly exaggerated) visual blending of situated visualisations and the real-world environment to produce data representations that resemble hallucinations. Such hallucinatory visualisations incorporate elements of the physical environment, twisting and morphing their appearance such that they become part of the visualisation itself. We demonstrate LSDvis in a ``proof of proof of concept'', where we use Stable Diffusion to modify images of real environments with abstract data visualisations as input. We conclude by discussing considerations of LSDvis. We hope that our work promotes visualisation designs which deprioritise saliency in favour of quirkiness and ambience.

cs.HC↗

arXiv2023

Designing Situated Dashboards: Challenges and Opportunities

Situated Visualization is an emerging field that unites several areas - visualization, augmented reality, human-computer interaction, and internet-of-things, to support human data activities within the ubiquitous world. Likewise, dashboards are broadly used to simplify complex data through multiple views. However, dashboards are only adapted for desktop settings, and requires visual strategies to support situatedness. We propose the concept of AR-based situated dashboards and present design considerations and challenges developed over interviews with experts. These challenges aim to propose directions and opportunities for facilitating the effective designing and authoring of situated dashboards.

cs.HC↗

arXiv2023

Data-driven Storytelling in Hybrid Immersive Display Environments

Data-driven stories seek to inform and persuade audiences through the use of data visualisations and engaging narratives. These stories have now been highly optimised to be viewed on desktop and mobile computers. In contrast, while immersive virtual and augmented reality (VR/AR) technologies have been shown to be more persuasive, no clear standard has yet emerged for such immersive stories. With this in mind, we propose that a hybrid data-driven storytelling approach can leverage the familiarity of 2D display devices with the immersiveness and presence afforded by VR/AR headsets. In this position paper, we characterise hybrid data-driven stories by describing its design opportunities, considerations, and challenges. In particular, we describe how both 2D and 3D display environments can play either complementary or symbiotic roles with each other for the purposes of storytelling. We hope that this work inspires researchers to investigate how hybrid user interfaces may be used for storytelling.

cs.HC↗

arXiv2023

Design Patterns for Situated Visualization in Augmented Reality

Situated visualization has become an increasingly popular research area in the visualization community, fueled by advancements in augmented reality (AR) technology and immersive analytics. Visualizing data in spatial proximity to their physical referents affords new design opportunities and considerations not present in traditional visualization, which researchers are now beginning to explore. However, the AR research community has an extensive history of designing graphics that are displayed in highly physical contexts. In this work, we leverage the richness of AR research and apply it to situated visualization. We derive design patterns which summarize common approaches of visualizing data in situ. The design patterns are based on a survey of 293 papers published in the AR and visualization communities, as well as our own expertise. We discuss design dimensions that help to describe both our patterns and previous work in the literature. This discussion is accompanied by several guidelines which explain how to apply the patterns given the constraints imposed by the real world. We conclude by discussing future research directions that will help establish a complete understanding of the design of situated visualization, including the role of interactivity, tasks, and workflows.

cs.HC↗

arXiv2023

PaLM 2 Technical Report

We introduce PaLM 2, a new state-of-the-art language model that has better multilingual and reasoning capabilities and is more compute-efficient than its predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture of objectives. Through extensive evaluations on English and multilingual language, and reasoning tasks, we demonstrate that PaLM 2 has significantly improved quality on downstream tasks across different model sizes, while simultaneously exhibiting faster and more efficient inference compared to PaLM. This improved efficiency enables broader deployment while also allowing the model to respond faster, for a more natural pace of interaction. PaLM 2 demonstrates robust reasoning capabilities exemplified by large improvements over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable performance on a suite of responsible AI evaluations, and enables inference-time control over toxicity without additional overhead or impact on other capabilities. Overall, PaLM 2 achieves state-of-the-art performance across a diverse set of tasks and capabilities. When discussing the PaLM 2 family, it is important to distinguish between pre-trained models (of various sizes), fine-tuned variants of these models, and the user-facing products that use these models. In particular, user-facing products typically include additional pre- and post-processing steps. Additionally, the underlying models may evolve over time. Therefore, one should not expect the performance of user-facing products to exactly match the results reported in this report.

cs.CL↗

arXiv2023

Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference

Mixture-of-Experts (MoE) models have gained popularity in achieving state-of-the-art performance in a wide range of tasks in computer vision and natural language processing. They effectively expand the model capacity while incurring a minimal increase in computation cost during training. However, deploying such models for inference is difficult due to their large size and complex communication pattern. In this work, we provide a characterization of two MoE workloads, namely Language Modeling (LM) and Machine Translation (MT) and identify their sources of inefficiencies at deployment. We propose three optimization techniques to mitigate sources of inefficiencies, namely (1) Dynamic gating, (2) Expert Buffering, and (3) Expert load balancing. We show that dynamic gating improves maximum throughput by 6.21-11.23$\times$ for LM, 5.75-10.98$\times$ for MT Encoder and 2.58-5.71$\times$ for MT Decoder. It also reduces memory usage by up to 1.36$\times$ for LM and up to 1.1$\times$ for MT. We further propose Expert Buffering, a new caching mechanism that only keeps hot, active experts in GPU memory while buffering the rest in CPU memory. This reduces static memory allocation by up to 1.47$\times$. We finally propose a load balancing methodology that provides additional scalability to the workload.

cs.DC↗