arXiv subjects

Shuai Zhang

Publications and source records attributed to Shuai Zhang.

At least 217 records · Page 12Linked to original sources

arXiv2024

Optimize the event selection strategy to study the anomalous quartic gauge couplings at muon colliders using the support vector machine and quantum support vector machine

The search of the new physics~(NP) beyond the Standard Model is one of the most important topics in current high energy physics. With the increasing luminosities at the colliders, the search for NP signals requires the analysis of more and more data, and the efficiency in data processing becomes particularly important. As a machine learning algorithm, support vector machine~(SVM) is expected to to be useful in the search of NP. Meanwhile, the quantum computing has the potential to offer huge advantages when dealing with large amounts of data, which suggests that quantum SVM~(QSVM) is a potential tool in future phenomenological studies of the NP. How to use SVM and QSVM to optimize event selection strategies to search for NP signals are studied in this paper. Taking the tri-photon process at a muon collider as an example, it can be shown that the event selection strategies optimized by the SVM and QSVM are effective in the search of the dimension-8 operators contributing to the anomalous quartic gauge couplings.

hep-ph↗

arXiv2024

Gaussian Pancakes: Geometrically-Regularized 3D Gaussian Splatting for Realistic Endoscopic Reconstruction

Within colorectal cancer diagnostics, conventional colonoscopy techniques face critical limitations, including a limited field of view and a lack of depth information, which can impede the detection of precancerous lesions. Current methods struggle to provide comprehensive and accurate 3D reconstructions of the colonic surface which can help minimize the missing regions and reinspection for pre-cancerous polyps. Addressing this, we introduce 'Gaussian Pancakes', a method that leverages 3D Gaussian Splatting (3D GS) combined with a Recurrent Neural Network-based Simultaneous Localization and Mapping (RNNSLAM) system. By introducing geometric and depth regularization into the 3D GS framework, our approach ensures more accurate alignment of Gaussians with the colon surface, resulting in smoother 3D reconstructions with novel viewing of detailed textures and structures. Evaluations across three diverse datasets show that Gaussian Pancakes enhances novel view synthesis quality, surpassing current leading methods with a 18% boost in PSNR and a 16% improvement in SSIM. It also delivers over 100X faster rendering and more than 10X shorter training times, making it a practical tool for real-time applications. Hence, this holds promise for achieving clinical translation for better detection and diagnosis of colorectal cancer.

cs.CV↗

arXiv2024

Measuring eye-tracking accuracy and its impact on usability in apple vision pro

With built-in eye-tracking cameras, the recently released Apple Vision Pro (AVP) mixed reality (MR) headset features gaze-based interaction, eye image rendering on external screens, and iris recognition for device unlocking. One of the technological advancements of the AVP is its heavy reliance on gaze- and gesture-based interaction. However, limited information is available regarding the technological specifications of the eye-tracking capability of the AVP, and raw gaze data is inaccessible to developers. This study evaluates the eye-tracking accuracy of the AVP with two sets of tests spanning both MR and virtual reality (VR) applications. This study also examines how eye-tracking accuracy relates to user-reported usability. The results revealed an overall eye-tracking accuracy of 1.11° and 0.93° in two testing setups, within a field of view (FOV) of approximately 34° x 18°. The usability and learnability scores of the AVP, measured using the standard System Usability Scale (SUS), were 75.24 and 68.26, respectively. Importantly, no statistically reliable correlation was found between eye-tracking accuracy and usability scores. These results suggest that eye-tracking accuracy is critical for gaze-based interaction, but it is not the sole determinant of user experience in VR/AR.

cs.HC↗

arXiv2024

CDFGNN: a Systematic Design of Cache-based Distributed Full-Batch Graph Neural Network Training with Communication Reduction

Graph neural network training is mainly categorized into mini-batch and full-batch training methods. The mini-batch training method samples subgraphs from the original graph in each iteration. This sampling operation introduces extra computation overhead and reduces the training accuracy. Meanwhile, the full-batch training method calculates the features and corresponding gradients of all vertices in each iteration, and therefore has higher convergence accuracy. However, in the distributed cluster, frequent remote accesses of vertex features and gradients lead to huge communication overhead, thus restricting the overall training efficiency. In this paper, we introduce the cached-based distributed full-batch graph neural network training framework (CDFGNN). We propose the adaptive cache mechanism to reduce the remote vertex access by caching the historical features and gradients of neighbor vertices. Besides, we further optimize the communication overhead by quantifying the messages and designing the graph partition algorithm for the hierarchical communication architecture. Experiments show that the adaptive cache mechanism reduces remote vertex accesses by 63.14% on average. Combined with communication quantization and hierarchical GP algorithm, CDFGNN outperforms the state-of-the-art distributed full-batch training frameworks by 30.39% in our experiments. Our results indicate that CDFGNN has great potential in accelerating distributed full-batch GNN training tasks.

cs.DC↗

arXiv2024

Electronic interactions in Dirac fluids visualized by nano-terahertz spacetime interference of electron-photon quasiparticles

Ultraclean graphene at charge neutrality hosts a quantum critical Dirac fluid of interacting electrons and holes. Interactions profoundly affect the charge dynamics of graphene, which is encoded in the properties of its electron-photon collective modes: surface plasmon polaritons (SPPs). Here we show that polaritonic interference patterns are particularly well suited to unveil the interactions in Dirac fluids by tracking polaritonic interference in time at temporal scales commensurate with the electronic scattering. Spacetime SPP interference patterns recorded in tera-hertz (THz) frequency range provided unobstructed readouts of the group velocity and lifetime of polariton that can be directly mapped onto the electronic spectral weight and the relaxation rate. Our data uncovered prominent departures of the electron dynamics from the predictions of the conventional Fermi-liquid theory. The deviations are particularly strong when the densities of electrons and holes are approximately equal. The proposed spacetime imaging methodology can be broadly applied to probe the electrodynamics of quantum materials.

cond-mat.str-el↗

arXiv2024

ASRRL-TTS: Agile Speaker Representation Reinforcement Learning for Text-to-Speech Speaker Adaptation

Speaker adaptation, which involves cloning voices from unseen speakers in the Text-to-Speech task, has garnered significant interest due to its numerous applications in multi-media fields. Despite recent advancements, existing methods often struggle with inadequate speaker representation accuracy and overfitting, particularly in limited reference speeches scenarios. To address these challenges, we propose an Agile Speaker Representation Reinforcement Learning strategy to enhance speaker similarity in speaker adaptation tasks. ASRRL is the first work to apply reinforcement learning to improve the modeling accuracy of speaker embeddings in speaker adaptation, addressing the challenge of decoupling voice content and timbre. Our approach introduces two action strategies tailored to different reference speeches scenarios. In the single-sentence scenario, a knowledge-oriented optimal routine searching RL method is employed to expedite the exploration and retrieval of refinement information on the fringe of speaker representations. In the few-sentence scenario, we utilize a dynamic RL method to adaptively fuse reference speeches, enhancing the robustness and accuracy of speaker modeling. To achieve optimal results in the target domain, a multi-scale fusion scoring mechanism based reward model that evaluates speaker similarity, speech quality, and intelligibility across three dimensions is proposed, ensuring that improvements in speaker similarity do not compromise speech quality or intelligibility. The experimental results on the LibriTTS and VCTK datasets within mainstream TTS frameworks demonstrate the extensibility and generalization capabilities of the proposed ASRRL method. The results indicate that the ASRRL method significantly outperforms traditional fine-tuning approaches, achieving higher speaker similarity and better overall speech quality with limited reference speeches.

eess.AS↗

arXiv2024

ADD 2022: the First Audio Deep Synthesis Detection Challenge

Audio deepfake detection is an emerging topic, which was included in the ASVspoof 2021. However, the recent shared tasks have not covered many real-life and challenging scenarios. The first Audio Deep synthesis Detection challenge (ADD) was motivated to fill in the gap. The ADD 2022 includes three tracks: low-quality fake audio detection (LF), partially fake audio detection (PF) and audio fake game (FG). The LF track focuses on dealing with bona fide and fully fake utterances with various real-world noises etc. The PF track aims to distinguish the partially fake audio from the real. The FG track is a rivalry game, which includes two tasks: an audio generation task and an audio fake detection task. In this paper, we describe the datasets, evaluation metrics, and protocols. We also report major findings that reflect the recent advances in audio deepfake detection tasks.

cs.SD↗

arXiv2024

Fake News Detection and Manipulation Reasoning via Large Vision-Language Models

Fake news becomes a growing threat to information security and public opinion with the rapid sprawl of media manipulation. Therefore, fake news detection attracts widespread attention from academic community. Traditional fake news detection models demonstrate remarkable performance on authenticity binary classification but their ability to reason detailed faked traces based on the news content remains under-explored. Furthermore, due to the lack of external knowledge, the performance of existing methods on fact-related news is questionable, leaving their practical implementation unclear. In this paper, we propose a new multi-media research topic, namely manipulation reasoning. Manipulation reasoning aims to reason manipulations based on news content. To support the research, we introduce a benchmark for fake news detection and manipulation reasoning, referred to as Human-centric and Fact-related Fake News (HFFN). The benchmark highlights the centrality of human and the high factual relevance, with detailed manual annotations. HFFN encompasses four realistic domains with fake news samples generated through three manipulation approaches. Moreover, a Multi-modal news Detection and Reasoning langUage Model (M-DRUM) is presented not only to judge on the authenticity of multi-modal news, but also raise analytical reasoning about potential manipulations. On the feature extraction level, a cross-attention mechanism is employed to extract fine-grained fusion features from multi-modal inputs. On the reasoning level, a large vision-language model (LVLM) serves as the backbone to facilitate fact-related reasoning. A two-stage training framework is deployed to better activate the capacity of identification and reasoning. Comprehensive experiments demonstrate that our model outperforms state-of-the-art (SOTA) fake news detection models and powerful LVLMs like GPT-4 and LLaVA.

cs.CL↗

arXiv2024

Orthogonal Constrained Neural Networks for Solving Structured Inverse Eigenvalue Problems

This paper introduces a novel neural network for efficiently solving Structured Inverse Eigenvalue Problems (SIEPs). The main contributions lie in two aspects: firstly, a unified framework is proposed that can handle various SIEPs instances. Particularly, an innovative method for handling nonnegativity constraints is devised using the ReLU function. Secondly, a novel neural network based on multilayer perceptrons, utilizing the Stiefel layer, is designed to efficiently solve SIEP. By incorporating the Stiefel layer through matrix orthogonal decomposition, the orthogonality of similarity transformations is ensured, leading to accurate solutions for SIEPs. Hence, we name this new network Stiefel Multilayer Perceptron (SMLP). Furthermore, SMLP is an unsupervised learning approach with a lightweight structure that is easy to train. Several numerical tests from literature and engineering domains demonstrate the efficiency of SMLP.

math.NA↗

arXiv2024

Plasmonic polarization sensing of electrostatic superlattice potentials

Plasmon polaritons are formed by coupling light with delocalized electrons. The half-light and half-matter nature of plasmon polaritons endows them with unparalleled tunability via a range of parameters, such as dielectric environments and carrier density. Therefore, plasmon polaritons are expected to be tuned when in proximity to polar materials since the carrier density is tuned by an electrostatic potential; conversely, the plasmon polariton response might enable the sensing of polarization. Here, we use infrared nano-imaging and nano-photocurrent measurements to investigate heterostructures composed of graphene and twisted hexagonal boron nitride (t-BN), with alternating polarization in a triangular network of moiré stacking domains. We observe that the carrier density and the corresponding plasmonic response of graphene are modulated by polar domains in t-BN. In addition, we demonstrate that the nanometer-wide domain walls of graphene moirés superlattices, created by the polar domains of t-BN, provide momenta to assist the plasmonic excitations. Furthermore, our studies establish that the plasmon of graphene could function as a delicate sensor for polarization textures. The evolution of polarization textures in t-BN under uniform electric fields is tomographically examined via plasmonic imaging. Strikingly, no noticeable polarization switching is observed under applied electric fields up to 0.23 V/nm, at variance with transport reports. Our nano-images unambiguously reveal that t-BN with triangular domains acts like a ferrielectric, rather than ferroelectric claimed by many previous studies.

cond-mat.mes-hall↗

arXiv2024

Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis

Efficient training and inference algorithms, such as low-rank adaption and model pruning, have shown impressive performance for learning Transformer-based large foundation models. However, due to the technical challenges of the non-convex optimization caused by the complicated architecture of Transformers, the theoretical study of why these methods can be applied to learn Transformers is mostly elusive. To the best of our knowledge, this paper shows the first theoretical analysis of the property of low-rank and sparsity of one-layer Transformers by characterizing the trained model after convergence using stochastic gradient descent. By focusing on a data model based on label-relevant and label-irrelevant patterns, we quantify that the gradient updates of trainable parameters are low-rank, which depends on the number of label-relevant patterns. We also analyze how model pruning affects the generalization while improving computation efficiency and conclude that proper magnitude-based pruning has a slight effect on the testing performance. We implement numerical experiments to support our findings.

cs.LG↗

arXiv2024

Unveiling the Impact of Multi-Modal Interactions on User Engagement: A Comprehensive Evaluation in AI-driven Conversations

Large Language Models (LLMs) have significantly advanced user-bot interactions, enabling more complex and coherent dialogues. However, the prevalent text-only modality might not fully exploit the potential for effective user engagement. This paper explores the impact of multi-modal interactions, which incorporate images and audio alongside text, on user engagement in chatbot conversations. We conduct a comprehensive analysis using a diverse set of chatbots and real-user interaction data, employing metrics such as retention rate and conversation length to evaluate user engagement. Our findings reveal a significant enhancement in user engagement with multi-modal interactions compared to text-only dialogues. Notably, the incorporation of a third modality significantly amplifies engagement beyond the benefits observed with just two modalities. These results suggest that multi-modal interactions optimize cognitive processing and facilitate richer information comprehension. This study underscores the importance of multi-modality in chatbot design, offering valuable insights for creating more engaging and immersive AI communication experiences and informing the broader AI community about the benefits of multi-modal interactions in enhancing user engagement.

cs.CL↗

arXiv2024

CaMML: Context-Aware Multimodal Learner for Large Models

In this work, we introduce Context-Aware MultiModal Learner (CaMML), for tuning large multimodal models (LMMs). CaMML, a lightweight module, is crafted to seamlessly integrate multimodal contextual samples into large models, thereby empowering the model to derive knowledge from analogous, domain-specific, up-to-date information and make grounded inferences. Importantly, CaMML is highly scalable and can efficiently handle lengthy multimodal context examples owing to its hierarchical design. Based on CaMML, we have developed two multimodal models, CaMML-7B and CaMML-13B, that have shown exceptional performance across an array of benchmark datasets for multimodal tasks. Remarkably, CaMML-13B achieves the state-of-the-art performance on over ten widely recognized multimodal benchmark datasets, surpassing LLaVA-1.5 (13B) with a noticeable margin, without integration of any external resources. Moreover, we have conducted extensive ablative studies to inspect the inner workings of CaMML and performed qualitative analyses to showcase its effectiveness in handling real-world challenging cases. Code and models are available at: https://github.com/amazon-science/camml.

cs.CV↗

arXiv2024

Secure Combination of Untrusted Time information Based on Optimized Dempster-Shafer Theory

Secure precision time synchronization is important for applications of Cyber-Physical Systems. However, several attacks, especially the Time Delay Attack (TDA), deteriorates the performance of time synchronization system seriously. Multiple paths scheme is thought as an effective security countermeasure to decrease the influence of TDA. However, the effective secure combination algorithm is still missed for precision time synchronization. In this paper, a secure combination algorithm based on Dempster-Shafer theory is proposed for multiple paths method. Special optimizations are done for the combination algorithm to solve the potential problems due to untrusted evidence. Theoretical simulation shows that the proposed algorithm works much better than Fault Tolerant Algorithm (FTA) and the attack detection method based on single path. And experimental demonstration proves the feasibility and superiority of the proposed algorithm, where the time stability with 27.97 ps, 1.57 ps, and 1.12 ps at average time 1s, 10s, 100s is achieved under TDA and local clock jump. The proposed algorithm can be used to improve the security and resilience of many importance synchronization protocol, such as NTP, PTP, and TWFTT.

cs.CR↗

arXiv2024

Low-Energy Electronic Structure in the Unconventional Charge-Ordered State of ScV$_6$Sn$_6$

Kagome vanadates {\it A}V$_3$Sb$_5$ display unusual low-temperature electronic properties including charge density waves (CDW), whose microscopic origin remains unsettled. Recently, CDW order has been discovered in a new material ScV$_6$Sn$_6$, providing an opportunity to explore whether the onset of CDW leads to unusual electronic properties. Here, we study this question using angle-resolved photoemission spectroscopy (ARPES) and scanning tunneling microscopy (STM). The ARPES measurements show minimal changes to the electronic structure after the onset of CDW. However, STM quasiparticle interference (QPI) measurements show strong dispersing features related to the CDW ordering vectors. A plausible explanation is the presence of a strong momentum-dependent scattering potential peaked at the CDW wavevector, associated with the existence of competing CDW instabilities. Our STM results further indicate that the bands most affected by the CDW are near vHS, analogous to the case of {\it A}V$_3$Sb$_5$ despite very different CDW wavevectors.

cond-mat.str-el↗

arXiv2024

ConceptPsy:A Benchmark Suite with Conceptual Comprehensiveness in Psychology

The critical field of psychology necessitates a comprehensive benchmark to enhance the evaluation and development of domain-specific Large Language Models (LLMs). Existing MMLU-type benchmarks, such as C-EVAL and CMMLU, include psychology-related subjects, but their limited number of questions and lack of systematic concept sampling strategies mean they cannot cover the concepts required in psychology. Consequently, despite their broad subject coverage, these benchmarks lack the necessary depth in the psychology domain, making them inadequate as psychology-specific evaluation suite. To address this issue, this paper presents ConceptPsy, designed to evaluate Chinese complex reasoning and knowledge abilities in psychology. ConceptPsy includes 12 core subjects and 1383 manually collected concepts. Specifically, we prompt GPT-4 to generate questions for each concept using carefully designed diverse prompts and hire professional psychologists to review these questions. To help to understand the fine-grained performances and enhance the weaknesses, we annotate each question with a chapter label and provide chapter-wise accuracy. Based on ConceptPsy, we evaluate a broad range of LLMs. We observe that, although some LLMs achieve similar accuracies on overall performances, they exhibit significant performance variations across different psychology concepts, even when they are models from the same series. We hope our work can facilitate the development of LLMs in the field of psychology.

cs.CL↗

arXiv2024

MINT: a Multi-modal Image and Narrative Text Dubbing Dataset for Foley Audio Content Planning and Generation

Foley audio, critical for enhancing the immersive experience in multimedia content, faces significant challenges in the AI-generated content (AIGC) landscape. Despite advancements in AIGC technologies for text and image generation, the foley audio dubbing remains rudimentary due to difficulties in cross-modal scene matching and content correlation. Current text-to-audio technology, which relies on detailed and acoustically relevant textual descriptions, falls short in practical video dubbing applications. Existing datasets like AudioSet, AudioCaps, Clotho, Sound-of-Story, and WavCaps do not fully meet the requirements for real-world foley audio dubbing task. To address this, we introduce the Multi-modal Image and Narrative Text Dubbing Dataset (MINT), designed to enhance mainstream dubbing tasks such as literary story audiobooks dubbing, image/silent video dubbing. Besides, to address the limitations of existing TTA technology in understanding and planning complex prompts, a Foley Audio Content Planning, Generation, and Alignment (CPGA) framework is proposed, which includes a content planning module leveraging large language models for complex multi-modal prompts comprehension. Additionally, the training process is optimized using Proximal Policy Optimization based reinforcement learning, significantly improving the alignment and auditory realism of generated foley audio. Experimental results demonstrate that our approach significantly advances the field of foley audio dubbing, providing robust solutions for the challenges of multi-modal dubbing. Even when utilizing the relatively lightweight GPT-2 model, our framework outperforms open-source multimodal large models such as LLaVA, DeepSeek-VL, and Moondream2. The dataset is available at https://github.com/borisfrb/MINT .

eess.AS↗

arXiv2024

Transferring Knowledge from Large Foundation Models to Small Downstream Models

How do we transfer the relevant knowledge from ever larger foundation models into small, task-specific downstream models that can run at much lower costs? Standard transfer learning using pre-trained weights as the initialization transfers limited information and commits us to often massive pre-trained architectures. This procedure also precludes combining multiple pre-trained models that learn complementary information. To address these shortcomings, we introduce Adaptive Feature Transfer (AFT). Instead of transferring weights, AFT operates purely on features, thereby decoupling the choice of the pre-trained model from the smaller downstream model. Rather than indiscriminately compressing all pre-trained features, AFT adaptively transfers pre-trained features that are most useful for performing the downstream task, using a simple regularization that adds minimal overhead. Across multiple vision, language, and multi-modal datasets, AFT achieves significantly better downstream performance compared to alternatives with a similar computational cost. Furthermore, AFT reliably translates improvement in pre-trained models into improvement in downstream performance, even if the downstream model is over $50\times$ smaller, and can effectively transfer complementary information learned by multiple pre-trained models.

cs.LG↗