arXiv subjects

Yue Zhang

Publications and source records attributed to Yue Zhang.

At least 415 records · Page 23Linked to original sources

arXiv2024

RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation

Despite Retrieval-Augmented Generation (RAG) showing promising capability in leveraging external knowledge, a comprehensive evaluation of RAG systems is still challenging due to the modular nature of RAG, evaluation of long-form responses and reliability of measurements. In this paper, we propose a fine-grained evaluation framework, RAGChecker, that incorporates a suite of diagnostic metrics for both the retrieval and generation modules. Meta evaluation verifies that RAGChecker has significantly better correlations with human judgments than other evaluation metrics. Using RAGChecker, we evaluate 8 RAG systems and conduct an in-depth analysis of their performance, revealing insightful patterns and trade-offs in the design choices of RAG architectures. The metrics of RAGChecker can guide researchers and practitioners in developing more effective RAG systems. This work has been open sourced at https://github.com/amazon-science/RAGChecker.

cs.CL↗

arXiv2024

Disorder-dependent superconducting pairing symmetry in doped graphene

Disorder and doping have profound effects on the intrinsic physical mechanisms of superconductivity. In this paper, we employed the determinant quantum Monte Carlo method to investigate the symmetry-allowed superconducting orders on the two-dimensional honeycomb lattice within the Hubbard model, using doped graphene as the carrier, focusing their response to bond disorder. Specifically, we calculated the pairing susceptibility and effective pairing interactions for the $d+id$ wave and extended $s$-wave pairings for different electron densities and disorder strengths. Our calculations show that at high electron densities, increased disorder strength may lead to a transform from $d+id$ wave dominance to extended $s$ wave dominance. However, at lower electron densities, neither of the two superconducting pairings appears under larger disorder strength. Our calculations may contribute to a further understanding of the superconducting behavior in doped materials affected by disorder.

cond-mat.supr-con↗

arXiv2024

Chain-of-Symbol Prompting Elicits Planning in Large Langauge Models

In this paper, we take the initiative to investigate the performance of LLMs on complex planning tasks that require LLMs to understand a virtual spatial environment simulated via natural language and act correspondingly in text. We propose a benchmark named Natural Language Planning and Action (Natala) composed of a set of novel tasks: Brick World, NLVR-based Manipulations, and Natural Language Navigation. We found that current popular LLMs such as ChatGPT still lack abilities in complex planning. This arises a question -- do the LLMs have a good understanding of the environments described in natural language, or maybe other alternatives such as symbolic representations are neater and hence better to be understood by LLMs? To this end, we propose a novel method called CoS (Chain-of-Symbol Prompting) that represents the complex environments with condensed symbolic spatial representations during the chained intermediate thinking steps. CoS is easy to use and does not need additional training on LLMs. Extensive experiments indicate that CoS clearly surpasses the performance of the Chain-of-Thought (CoT) Prompting in all three planning tasks with even fewer tokens used in the inputs compared with CoT on ChatGPT and InstructGPT. The performance gain is strong, by up to 60.8% accuracy (from 31.8% to 92.6%) on Brick World for ChatGPT. CoS also reduces the number of tokens in the prompt obviously, by up to 65.8% of the tokens (from 407 to 139) for the intermediate steps from demonstrations on Brick World. Code and data available at: https://github.com/hanxuhu/chain-of-symbol-planning

cs.CL↗

arXiv2024

Nonlinearity-induced dynamical self-organized twisted-bilayer lattices in Bose-Einstein condensates

Creating crystal bilayers twisted with respect to each other would lead to large periodic supercell structures, which can support a wide range of novel electron correlated phenomena, where the full understanding is still under debate. Here, we propose a new scheme to realize a nonlinearity-induced dynamical self-organized twisted-bilayer lattice in an atomic Bose-Einstein condensate (BEC). The key idea here is to utilize the nonlinear effect from the intrinsic atomic interactions to couple different layers and induce a dynamical self-organized supercell structure, dramatically distinct from the conventional wisdom to achieve the static twisted-bilayer lattices. To illustrate that, we study the dynamics of a two-component BEC and show that the nonlinear interaction effect naturally emerged in the Gross-Pitaevskii equation of interacting bosonic ultracold atoms can dynamically induce both periodic (commensurable) and aperiodic (incommensurable) moiré structures. One of the interesting moiré phenomena, i.e., the flat-band physics, is shown through investigating the dynamics of the wave packet of BEC. Our proposal can be implemented using available state-of-the-art experimental techniques and reveal a profound connection between the nonlinearity and twistronics in cold atom quantum simulators.

cond-mat.quant-gas↗

arXiv2024

Structure-Aware Simplification for Hypergraph Visualization

Hypergraphs provide a natural way to represent polyadic relationships in network data. For large hypergraphs, it is often difficult to visually detect structures within the data. Recently, a scalable polygon-based visualization approach was developed allowing hypergraphs with thousands of hyperedges to be simplified and examined at different levels of detail. However, this approach is not guaranteed to eliminate all of the visual clutter caused by unavoidable overlaps. Furthermore, meaningful structures can be lost at simplified scales, making their interpretation unreliable. In this paper, we define hypergraph structures using the bipartite graph representation, allowing us to decompose the hypergraph into a union of structures including topological blocks, bridges, and branches, and to identify exactly where unavoidable overlaps must occur. We also introduce a set of topology preserving and topology altering atomic operations, enabling the preservation of important structures while reducing unavoidable overlaps to improve visual clarity and interpretability in simplified scales. We demonstrate our approach in several real-world applications.

cs.GR↗

arXiv2024

Anomalous symmetry protected blockade of skin effect in one-dimensional non-Hermitian lattice systems

The non-Hermitian skin effect (NHSE), an anomalous localization behavior of the bulk states, is an inherently non-Hermitian phenomenon, which can not find a counterpart in Hermitian systems. However, the fragility of NHSE has been revealed recently, such as the boundary sensitivity, and it stimulates a lot of studies on discussing the fate of that. Here we present a theorem which shows that the combined spatial reflection symmetry can be considered as a criterion in one-dimensional non-Hermitian systems to determine whether the NHSE can exist or not. Distinct from previous studies, our proposed criterion only relies on analyzing the symmetry of the system, freeing out other requirements, such as the information of the energy spectrum. Furthermore, by taking the non-Hermitian Kitaev chain as an example, we verify our theorem through both a mathematical proof via the non-Bloch band theory and the exact diagonalization numerical studies. Our results reveal a profound connection between the symmetry and the fate of NHSE.

quant-ph↗

arXiv2024

Non-chiral non-Bloch invariants and topological phase diagram in non-unitary quantum dynamics without chiral symmetry

The non-Bloch topology leads to the emergence of various counter-intuitive phenomena in non-Hermitian systems under the open boundary condition (OBC), which can not find a counterpart in Hermitian systems. However, in the non-Hermitian system without chiral symmetry, being ubiquitous in nature, exploring its non-Bloch topology has so far eluded experimental effort. Here by introducing the concept of non-chiral non-Bloch invariants, we theoretically predict and experimentally identify the non-Bloch topological phase diagram of a one-dimensional (1D) non-Hermitian system without chiral symmetry in discrete-time non-unitary quantum walks of single photons. Interestingly, we find that such topological invariants not only can distinguish topologically distinct gapped phases, but also faithfully capture the corresponding gap closing in open-boundary spectrum at the phase boundary. Different topological regions are experimentally identified by measuring the featured discontinuities of the higher moments of the walker's displacement, which amazingly match excellently with our defined non-Bloch invariants. Our work provides a useful platform to study the interplay among topology, symmetries and the non-Hermiticity.

quant-ph↗

arXiv2024

Topology Optimization of Random Memristors for Input-Aware Dynamic SNN

There is unprecedented development in machine learning, exemplified by recent large language models and world simulators, which are artificial neural networks running on digital computers. However, they still cannot parallel human brains in terms of energy efficiency and the streamlined adaptability to inputs of different difficulties, due to differences in signal representation, optimization, run-time reconfigurability, and hardware architecture. To address these fundamental challenges, we introduce pruning optimization for input-aware dynamic memristive spiking neural network (PRIME). Signal representation-wise, PRIME employs leaky integrate-and-fire neurons to emulate the brain's inherent spiking mechanism. Drawing inspiration from the brain's structural plasticity, PRIME optimizes the topology of a random memristive spiking neural network without expensive memristor conductance fine-tuning. For runtime reconfigurability, inspired by the brain's dynamic adjustment of computational depth, PRIME employs an input-aware dynamic early stop policy to minimize latency during inference, thereby boosting energy efficiency without compromising performance. Architecture-wise, PRIME leverages memristive in-memory computing, mirroring the brain and mitigating the von Neumann bottleneck. We validated our system using a 40 nm 256 Kb memristor-based in-memory computing macro on neuromorphic image classification and image inpainting. Our results demonstrate the classification accuracy and Inception Score are comparable to the software baseline, while achieving maximal 62.50-fold improvements in energy efficiency, and maximal 77.0% computational load savings. The system also exhibits robustness against stochastic synaptic noise of analogue memristors. Our software-hardware co-designed model paves the way to future brain-inspired neuromorphic computing with brain-like energy efficiency and adaptivity.

cs.ET↗

arXiv2024

RefuteBench: Evaluating Refuting Instruction-Following for Large Language Models

The application scope of large language models (LLMs) is increasingly expanding. In practical use, users might provide feedback based on the model's output, hoping for a responsive model that can complete responses according to their feedback. Whether the model can appropriately respond to users' refuting feedback and consistently follow through with execution has not been thoroughly analyzed. In light of this, this paper proposes a comprehensive benchmark, RefuteBench, covering tasks such as question answering, machine translation, and email writing. The evaluation aims to assess whether models can positively accept feedback in form of refuting instructions and whether they can consistently adhere to user demands throughout the conversation. We conduct evaluations on numerous LLMs and find that LLMs are stubborn, i.e. exhibit inclination to their internal knowledge, often failing to comply with user feedback. Additionally, as the length of the conversation increases, models gradually forget the user's stated feedback and roll back to their own responses. We further propose a recall-and-repeat prompts as a simple and effective way to enhance the model's responsiveness to feedback.

cs.CL↗

arXiv2024

EUFormer: Learning Driven 3D Spine Deformity Assessment with Orthogonal Optical Images

In clinical settings, the screening, diagnosis, and monitoring of adolescent idiopathic scoliosis (AIS) typically involve physical or radiographic examinations. However, physical examinations are subjective, while radiographic examinations expose patients to harmful radiation. Consequently, we propose a pipeline that can accurately determine scoliosis severity. This pipeline utilizes posteroanterior (PA) and lateral (LAT) RGB images as input to generate spine curve maps, which are then used to reconstruct the three-dimensional (3D) spine curve for AIS severity grading. To generate the 2D spine curves accurately and efficiently, we further propose an Efficient U-shape transFormer (EUFormer) as the generator. It can efficiently utilize the learned feature across channels, therefore producing consecutive spine curves from both PA and LAT views. Experimental results demonstrate superior performance of EUFormer on spine curve generation against other classical U-shape models. This finding demonstrates that the proposed method for grading the severity of AIS, based on a 3D spine curve, is more accurate when compared to using a 2D spine curve.

cs.GR↗

arXiv2024

Common Sense Reasoning for Deepfake Detection

State-of-the-art deepfake detection approaches rely on image-based features extracted via neural networks. While these approaches trained in a supervised manner extract likely fake features, they may fall short in representing unnatural `non-physical' semantic facial attributes -- blurry hairlines, double eyebrows, rigid eye pupils, or unnatural skin shading. However, such facial attributes are easily perceived by humans and used to discern the authenticity of an image based on human common sense. Furthermore, image-based feature extraction methods that provide visual explanations via saliency maps can be hard to interpret for humans. To address these challenges, we frame deepfake detection as a Deepfake Detection VQA (DD-VQA) task and model human intuition by providing textual explanations that describe common sense reasons for labeling an image as real or fake. We introduce a new annotated dataset and propose a Vision and Language Transformer-based framework for the DD-VQA task. We also incorporate text and image-aware feature alignment formulation to enhance multi-modal representation learning. As a result, we improve upon existing deepfake detection models by integrating our learned vision representations, which reason over common sense knowledge from the DD-VQA task. We provide extensive empirical results demonstrating that our method enhances detection performance, generalization ability, and language-based interpretability in the deepfake detection task.

cs.CV↗

arXiv2024

PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts

The increasing reliance on Large Language Models (LLMs) across academia and industry necessitates a comprehensive understanding of their robustness to prompts. In response to this vital need, we introduce PromptRobust, a robustness benchmark designed to measure LLMs' resilience to adversarial prompts. This study uses a plethora of adversarial textual attacks targeting prompts across multiple levels: character, word, sentence, and semantic. The adversarial prompts, crafted to mimic plausible user errors like typos or synonyms, aim to evaluate how slight deviations can affect LLM outcomes while maintaining semantic integrity. These prompts are then employed in diverse tasks including sentiment analysis, natural language inference, reading comprehension, machine translation, and math problem-solving. Our study generates 4,788 adversarial prompts, meticulously evaluated over 8 tasks and 13 datasets. Our findings demonstrate that contemporary LLMs are not robust to adversarial prompts. Furthermore, we present a comprehensive analysis to understand the mystery behind prompt robustness and its transferability. We then offer insightful robustness analysis and pragmatic recommendations for prompt composition, beneficial to both researchers and everyday users.

cs.CL↗

arXiv2024

Dynamic neural network with memristive CIM and CAM for 2D and 3D vision

The brain is dynamic, associative and efficient. It reconfigures by associating the inputs with past experiences, with fused memory and processing. In contrast, AI models are static, unable to associate inputs with past experiences, and run on digital computers with physically separated memory and processing. We propose a hardware-software co-design, a semantic memory-based dynamic neural network (DNN) using memristor. The network associates incoming data with the past experience stored as semantic vectors. The network and the semantic memory are physically implemented on noise-robust ternary memristor-based Computing-In-Memory (CIM) and Content-Addressable Memory (CAM) circuits, respectively. We validate our co-designs, using a 40nm memristor macro, on ResNet and PointNet++ for classifying images and 3D points from the MNIST and ModelNet datasets, which not only achieves accuracy on par with software but also a 48.1% and 15.9% reduction in computational budget. Moreover, it delivers a 77.6% and 93.3% reduction in energy consumption.

cs.AR↗

arXiv2024

$R^3$: "This is My SQL, Are You With Me?" A Consensus-Based Multi-Agent System for Text-to-SQL Tasks

Large Language Models (LLMs) have demonstrated strong performance on various tasks. To unleash their power on the Text-to-SQL task, we propose $R^3$ (Review-Rebuttal-Revision), a consensus-based multi-agent system for Text-to-SQL tasks. $R^3$ outperforms the existing single LLM Text-to-SQL systems as well as the multi-agent Text-to-SQL systems by $1.3\%$ to $8.1\%$ on Spider and Bird. Surprisingly, we find that for Llama-3-8B, $R^3$ outperforms chain-of-thought prompting by over 20\%, even outperforming GPT-3.5 on the development set of Spider.

cs.CL↗

arXiv2024

Unified Multi-Modal Image Synthesis for Missing Modality Imputation

Multi-modal medical images provide complementary soft-tissue characteristics that aid in the screening and diagnosis of diseases. However, limited scanning time, image corruption and various imaging protocols often result in incomplete multi-modal images, thus limiting the usage of multi-modal data for clinical purposes. To address this issue, in this paper, we propose a novel unified multi-modal image synthesis method for missing modality imputation. Our method overall takes a generative adversarial architecture, which aims to synthesize missing modalities from any combination of available ones with a single model. To this end, we specifically design a Commonality- and Discrepancy-Sensitive Encoder for the generator to exploit both modality-invariant and specific information contained in input modalities. The incorporation of both types of information facilitates the generation of images with consistent anatomy and realistic details of the desired distribution. Besides, we propose a Dynamic Feature Unification Module to integrate information from a varying number of available modalities, which enables the network to be robust to random missing modalities. The module performs both hard integration and soft integration, ensuring the effectiveness of feature combination while avoiding information loss. Verified on two public multi-modal magnetic resonance datasets, the proposed method is effective in handling various synthesis tasks and shows superior performance compared to previous methods.

cs.CV↗

arXiv2024

GPT-4 vs. Human Translators: A Comprehensive Evaluation of Translation Quality Across Languages, Domains, and Expertise Levels

This study comprehensively evaluates the translation quality of Large Language Models (LLMs), specifically GPT-4, against human translators of varying expertise levels across multiple language pairs and domains. Through carefully designed annotation rounds, we find that GPT-4 performs comparably to junior translators in terms of total errors made but lags behind medium and senior translators. We also observe the imbalanced performance across different languages and domains, with GPT-4's translation capability gradually weakening from resource-rich to resource-poor directions. In addition, we qualitatively study the translation given by GPT-4 and human translators, and find that GPT-4 translator suffers from literal translations, but human translators sometimes overthink the background information. To our knowledge, this study is the first to evaluate LLMs against human translators and analyze the systematic differences between their outputs, providing valuable insights into the current state of LLM-based translation and its potential limitations.

cs.CL↗

arXiv2024

Text2TimeSeries: Enhancing Financial Forecasting through Time Series Prediction Updates with Event-Driven Insights from Large Language Models

Time series models, typically trained on numerical data, are designed to forecast future values. These models often rely on weighted averaging techniques over time intervals. However, real-world time series data is seldom isolated and is frequently influenced by non-numeric factors. For instance, stock price fluctuations are impacted by daily random events in the broader world, with each event exerting a unique influence on price signals. Previously, forecasts in financial markets have been approached in two main ways: either as time-series problems over price sequence or sentiment analysis tasks. The sentiment analysis tasks aim to determine whether news events will have a positive or negative impact on stock prices, often categorizing them into discrete labels. Recognizing the need for a more comprehensive approach to accurately model time series prediction, we propose a collaborative modeling framework that incorporates textual information about relevant events for predictions. Specifically, we leverage the intuition of large language models about future changes to update real number time series predictions. We evaluated the effectiveness of our approach on financial market data.

cs.CL↗

arXiv2024

LexMatcher: Dictionary-centric Data Collection for LLM-based Machine Translation

The fine-tuning of open-source large language models (LLMs) for machine translation has recently received considerable attention, marking a shift towards data-centric research from traditional neural machine translation. However, the area of data collection for instruction fine-tuning in machine translation remains relatively underexplored. In this paper, we present LexMatcher, a simple yet effective method for data curation, the design of which is driven by the coverage of senses found in bilingual dictionaries. The construction process comprises data retrieval from an existing corpus and data augmentation that supplements the infrequent senses of polysemous words. Utilizing LLaMA2 as our base model, our approach outperforms the established baselines on the WMT2022 test sets and also exhibits remarkable performance in tasks related to word sense disambiguation and specialized terminology translation. These results underscore the effectiveness of LexMatcher in enhancing LLM-based machine translation. The code, data, and models are available at https://github.com/ARIES-LM/Lexmatcher-MT.git.

cs.CL↗