arXiv subjects

Tao Yu

Publications and source records attributed to Tao Yu.

At least 217 records · Page 12Linked to original sources

arXiv2023

Fluctuation-based Adaptive Structured Pruning for Large Language Models

Network Pruning is a promising way to address the huge computing resource demands of the deployment and inference of Large Language Models (LLMs). Retraining-free is important for LLMs' pruning methods. However, almost all of the existing retraining-free pruning approaches for LLMs focus on unstructured pruning, which requires specific hardware support for acceleration. In this paper, we propose a novel retraining-free structured pruning framework for LLMs, named FLAP (FLuctuation-based Adaptive Structured Pruning). It is hardware-friendly by effectively reducing storage and enhancing inference speed. For effective structured pruning of LLMs, we highlight three critical elements that demand the utmost attention: formulating structured importance metrics, adaptively searching the global compressed model, and implementing compensation mechanisms to mitigate performance loss. First, FLAP determines whether the output feature map is easily recoverable when a column of weight is removed, based on the fluctuation pruning metric. Then it standardizes the importance scores to adaptively determine the global compressed model structure. At last, FLAP adds additional bias terms to recover the output feature maps using the baseline values. We thoroughly evaluate our approach on a variety of language benchmarks. Without any retraining, our method significantly outperforms the state-of-the-art methods, including LLM-Pruner and the extension of Wanda in structured pruning. The code is released at https://github.com/CASIA-IVA-Lab/FLAP.

cs.CL↗

arXiv2023

Coneheads: Hierarchy Aware Attention

Attention networks such as transformers have achieved state-of-the-art performance in many domains. These networks rely heavily on the dot product attention operator, which computes the similarity between two points by taking their inner product. However, the inner product does not explicitly model the complex structural properties of real world datasets, such as hierarchies between data points. To remedy this, we introduce cone attention, a drop-in replacement for dot product attention based on hyperbolic entailment cones. Cone attention associates two points by the depth of their lowest common ancestor in a hierarchy defined by hyperbolic cones, which intuitively measures the divergence of two points and gives a hierarchy aware similarity score. We test cone attention on a wide variety of models and tasks and show that it improves task-level performance over dot product attention and other baselines, and is able to match dot-product attention with significantly fewer parameters. Our results suggest that cone attention is an effective way to capture hierarchical relationships when calculating attention.

cs.LG↗

arXiv2023

Explicit representations and Azumaya loci of skein algebras of small surfaces

We construct finite dimensional representations of the Kauffman bracket skein algebra of the one-punctured torus and four-punctured sphere at all roots of unity. The representations are given by explicit formulas. They all have dimensions equal to the PI degrees of the skein algebras, and they realize all classical shadows. We then use the reducibility of these representations to determine the Azumaya loci. In particular, the Azumaya loci of these surfaces contain the smooth loci of the classical shadow varieties, with equality in the case of the one-punctured torus and proper containment in the case of the four-punctured sphere.

math.QA↗

arXiv2023

Chirality Enables Thermal Magnon Transistors

We report a theory of thermal spin pumping into proximity magnets under a transverse-bias-driven heat flow of magnons in magnetic films when the dipolar coupling to the magnetic gate is tuned to be "chiral". While there is no rectification of the magnon current in the film, we predict that chirality diverts a large percentage (50$\%$ for perfect chirality) of it into the gate. This transverse thermal spin pumping effect can be controlled by rotating the film magnetization and may help manage the heat flow in future magnonic circuits.

cond-mat.mes-hall↗

arXiv2023

OmniSeg3D: Omniversal 3D Segmentation via Hierarchical Contrastive Learning

Towards holistic understanding of 3D scenes, a general 3D segmentation method is needed that can segment diverse objects without restrictions on object quantity or categories, while also reflecting the inherent hierarchical structure. To achieve this, we propose OmniSeg3D, an omniversal segmentation method aims for segmenting anything in 3D all at once. The key insight is to lift multi-view inconsistent 2D segmentations into a consistent 3D feature field through a hierarchical contrastive learning framework, which is accomplished by two steps. Firstly, we design a novel hierarchical representation based on category-agnostic 2D segmentations to model the multi-level relationship among pixels. Secondly, image features rendered from the 3D feature field are clustered at different levels, which can be further drawn closer or pushed apart according to the hierarchical relationship between different levels. In tackling the challenges posed by inconsistent 2D segmentations, this framework yields a global consistent 3D feature field, which further enables hierarchical segmentation, multi-object selection, and global discretization. Extensive experiments demonstrate the effectiveness of our method on high-quality 3D segmentation and accurate hierarchical structure understanding. A graphical user interface further facilitates flexible interaction for omniversal 3D segmentation.

cs.CV↗

arXiv2023

Non-Hermitian Topological Magnonics

Dissipation in mechanics, optics, acoustics, and electronic circuits is nowadays recognized to be not always detrimental but can be exploited to achieve non-Hermitian topological phases or properties with functionalities for potential device applications. As elementary excitations of ordered magnetic moments that exist in various magnetic materials, magnons are the information carriers in magnonic devices with low-energy consumption for reprogrammable logic, non-reciprocal communication, and non-volatile memory functionalities. Non-Hermitian topological magnonics deals with the engineering of dissipation and/or gain for non-Hermitian topological phases or properties in magnets that are not achievable in the conventional Hermitian scenario, with associated functionalities cross-fertilized with their electronic, acoustic, optic, and mechanic counterparts, such as giant enhancement of magnonic frequency combs, magnon amplification, (quantum) sensing of the magnetic field with unprecedented sensitivity, magnon accumulation, and perfect absorption of microwaves. In this review article, we address the unified approach in constructing magnonic non-Hermitian Hamiltonian, introduce the basic non-Hermitian topological physics, and provide a comprehensive overview of the recent theoretical and experimental progress towards achieving distinct non-Hermitian topological phases or properties in magnonic devices, including exceptional points, exceptional nodal phases, non-Hermitian magnonic SSH model, and non-Hermitian skin effect. We emphasize the non-Hermitian Hamiltonian approach based on the Lindbladian or self-energy of the magnonic subsystem but address the physics beyond it as well, such as the crucial quantum jump effect in the quantum regime and non-Markovian dynamics. We provide a perspective for future opportunities and challenges before concluding this article.

cond-mat.mes-hall↗

arXiv2023

Batch Prompting: Efficient Inference with Large Language Model APIs

Performing inference on large volumes of samples with large language models (LLMs) can be computationally and financially costly in industry and real-world use. We propose batch prompting, a simple yet effective prompting approach that enables the LLM to run inference in batches, instead of one sample at a time. Our method reduces both token and time costs while retaining downstream performance. We theoretically demonstrate that under a few-shot in-context learning setting, the inference costs decrease almost inverse linearly with the number of samples in each batch. We extensively validate the effectiveness of batch prompting on ten datasets across commonsense QA, arithmetic reasoning, and NLI/NLU: batch prompting significantly~(up to 5x with six samples in batch) reduces the LLM (Codex) inference token and time costs while achieving better or comparable performance. For state-of-the-art Chat-based LLMs, e.g., GPT-3.5 and GPT-4, we show the benefits of batch prompting also hold. Further analysis shows that the number of samples in each batch and the complexity of tasks affect its performance. Moreover, batch prompting can be applied across different reasoning methods using LLMs. Our code can be found at the site https://github.com/xlang-ai/batch-prompting.

cs.CL↗

arXiv2023

HDhuman: High-quality Human Novel-view Rendering from Sparse Views

In this paper, we aim to address the challenge of novel view rendering of human performers who wear clothes with complex texture patterns using a sparse set of camera views. Although some recent works have achieved remarkable rendering quality on humans with relatively uniform textures using sparse views, the rendering quality remains limited when dealing with complex texture patterns as they are unable to recover the high-frequency geometry details that are observed in the input views. To this end, we propose HDhuman, which uses a human reconstruction network with a pixel-aligned spatial transformer and a rendering network with geometry-guided pixel-wise feature integration to achieve high-quality human reconstruction and rendering. The designed pixel-aligned spatial transformer calculates the correlations between the input views and generates human reconstruction results with high-frequency details. Based on the surface reconstruction results, the geometry-guided pixel-wise visibility reasoning provides guidance for multi-view feature integration, enabling the rendering network to render high-quality images at 2k resolution on novel views. Unlike previous neural rendering works that always need to train or fine-tune an independent network for a different scene, our method is a general framework that is able to generalize to novel subjects. Experiments show that our approach outperforms all the prior generic or specific methods on both synthetic data and real-world data.

cs.CV↗

arXiv2023

OpenAgents: An Open Platform for Language Agents in the Wild

Language agents show potential in being capable of utilizing natural language for varied and intricate tasks in diverse environments, particularly when built upon large language models (LLMs). Current language agent frameworks aim to facilitate the construction of proof-of-concept language agents while neglecting the non-expert user access to agents and paying little attention to application-level designs. We present OpenAgents, an open platform for using and hosting language agents in the wild of everyday life. OpenAgents includes three agents: (1) Data Agent for data analysis with Python/SQL and data tools; (2) Plugins Agent with 200+ daily API tools; (3) Web Agent for autonomous web browsing. OpenAgents enables general users to interact with agent functionalities through a web user interface optimized for swift responses and common failures while offering developers and researchers a seamless deployment experience on local setups, providing a foundation for crafting innovative language agents and facilitating real-world evaluations. We elucidate the challenges and opportunities, aspiring to set a foundation for future research and development of real-world language agents.

cs.CL↗

arXiv2023

QCFE: An efficient Feature engineering for query cost estimation

Query cost estimation is a classical task for database management. Recently, researchers apply the AI-driven model to implement query cost estimation for achieving high accuracy. However, two defects of feature design lead to poor cost estimation accuracy-time efficiency. On the one hand, existing works only encode the query plan and data statistics while ignoring some other important variables, like storage structure, hardware, database knobs, etc. These variables also have significant impact on the query cost. On the other hand, due to the straightforward encoding design, existing works suffer heavy representation learning burden on ineffective dimensions of input. To meet the above two problems, we first propose an efficient feature engineering for query cost estimation, called QCFE. Specifically, we design a novel feature called feature snapshot to efficiently integrate the influences of the ignored variables. Further, we propose a difference-propagation feature reduction method for query cost estimation to filter the useless features. The experimental results demonstrate our QCFE could largely improve the time-accuracy efficiency on extensive benchmarks.

cs.DB↗

arXiv2023

PARF: Primitive-Aware Radiance Fusion for Indoor Scene Novel View Synthesis

This paper proposes a method for fast scene radiance field reconstruction with strong novel view synthesis performance and convenient scene editing functionality. The key idea is to fully utilize semantic parsing and primitive extraction for constraining and accelerating the radiance field reconstruction process. To fulfill this goal, a primitive-aware hybrid rendering strategy was proposed to enjoy the best of both volumetric and primitive rendering. We further contribute a reconstruction pipeline conducts primitive parsing and radiance field learning iteratively for each input frame which successfully fuses semantic, primitive, and radiance information into a single framework. Extensive evaluations demonstrate the fast reconstruction ability, high rendering quality, and convenient editing functionality of our method.

cs.CV↗

arXiv2023

Center of the stated skein algebra

The stated skein algebra is a generalization of the Kauffman bracket skein algebra introduced in the study of quantum trace maps. When the quantum parameter is a root of unity, the stated skein algebra has a big center and is finitely generated as a module over the center. We give the center a simple description and calculate the dimension over center of the stated skein algebra.

math.QA↗

arXiv2023

Edge and corner skin effects of chirally coupled magnons characterized by a topological winding tuple

We investigate a long-ranged coupled and non-Hermitian two-dimensional array of nanomagnets, fabricated on a thin magnetic substrate and subjected to an in-plane magnetic field. We predict topology-driven edge and corner skin effects of magnetic eigenmodes with the localization position at boundaries precisely characterized by a topological winding tuple $({\cal W}_1,{\cal W}_2)$. By varying the direction of the in-plane field, all magnon states pile up either at different edges of the array with $({\cal W}_1=\pm 1,{\cal W}_2=0)$ or $({\cal W}_1=0,{\cal W}_2=\pm 1)$, or at different corners characterized by $({\cal W}_1=\pm 1,{\cal W}_2=\pm 1)$. Exploiting the non-Hermitian topology is potentially helpful for designing useful magnonic metasurface in the future.

cond-mat.mes-hall↗

arXiv2023

GraphIQA: Learning Distortion Graph Representations for Blind Image Quality Assessment

A good distortion representation is crucial for the success of deep blind image quality assessment (BIQA). However, most previous methods do not effectively model the relationship between distortions or the distribution of samples with the same distortion type but different distortion levels. In this work, we start from the analysis of the relationship between perceptual image quality and distortion-related factors, such as distortion types and levels. Then, we propose a Distortion Graph Representation (DGR) learning framework for IQA, named GraphIQA, in which each distortion is represented as a graph, i.e., DGR. One can distinguish distortion types by learning the contrast relationship between these different DGRs, and infer the ranking distribution of samples from different levels in a DGR. Specifically, we develop two sub-networks to learn the DGRs: a) Type Discrimination Network (TDN) that aims to embed DGR into a compact code for better discriminating distortion types and learning the relationship between types; b) Fuzzy Prediction Network (FPN) that aims to extract the distributional characteristics of the samples in a DGR and predicts fuzzy degrees based on a Gaussian prior. Experiments show that our GraphIQA achieves the state-of-the-art performance on many benchmark datasets of both synthetic and authentic distortions.

cs.MM↗

arXiv2023

Gating ferromagnetic resonance of magnetic insulators by superconductors via modulating electric-field radiation

We predict that ferromagnetic resonance in insulating magnetic film with inplane magnetization radiates electric fields polarized along the magnetization with opposite amplitudes at two sides of the magnetic insulator, which can be modulated strongly by adjacent superconductors. With a single superconductor adjacent to the magnetic insulator this radiated electric field is totally reflected with a $π$-phase shift, which thereby vanishes at the superconductor side and causes no influence on the ferromagnetic resonance. When the magnetic insulator is sandwiched by two superconductors, this reflection becomes back and forth, so the electric field exists at both superconductors that drives the Meissner supercurrent, which in turn shifts efficiently the ferromagnetic resonance. We predict an ultrastrong coupling between magnons in the yttrium iron garnet and Cooper pairs in NbN with the frequency shift achieving tens of percent of the bare ferromagnetic resonance.

cond-mat.supr-con↗

arXiv2023

Dynamic Spectrum Mixer for Visual Recognition

Recently, MLP-based vision backbones have achieved promising performance in several visual recognition tasks. However, the existing MLP-based methods directly aggregate tokens with static weights, leaving the adaptability to different images untouched. Moreover, Recent research demonstrates that MLP-Transformer is great at creating long-range dependencies but ineffective at catching high frequencies that primarily transmit local information, which prevents it from applying to the downstream dense prediction tasks, such as semantic segmentation. To address these challenges, we propose a content-adaptive yet computationally efficient structure, dubbed Dynamic Spectrum Mixer (DSM). The DSM represents token interactions in the frequency domain by employing the Discrete Cosine Transform, which can learn long-term spatial dependencies with log-linear complexity. Furthermore, a dynamic spectrum weight generation layer is proposed as the spectrum bands selector, which could emphasize the informative frequency bands while diminishing others. To this end, the technique can efficiently learn detailed features from visual input that contains both high- and low-frequency information. Extensive experiments show that DSM is a powerful and adaptable backbone for a range of visual recognition tasks. Particularly, DSM outperforms previous transformer-based and MLP-based models, on image classification, object detection, and semantic segmentation tasks, such as 83.8 \% top-1 accuracy on ImageNet, and 49.9 \% mIoU on ADE20K.

cs.CV↗

arXiv2023

ImmersiveNeRF: Hybrid Radiance Fields for Unbounded Immersive Light Field Reconstruction

This paper proposes a hybrid radiance field representation for unbounded immersive light field reconstruction which supports high-quality rendering and aggressive view extrapolation. The key idea is to first formally separate the foreground and the background and then adaptively balance learning of them during the training process. To fulfill this goal, we represent the foreground and background as two separate radiance fields with two different spatial mapping strategies. We further propose an adaptive sampling strategy and a segmentation regularizer for more clear segmentation and robust convergence. Finally, we contribute a novel immersive light field dataset, named THUImmersive, with the potential to achieve much larger space 6DoF immersive rendering effects compared with existing datasets, by capturing multiple neighboring viewpoints for the same scene, to stimulate the research and AR/VR applications in the immersive light field domain. Extensive experiments demonstrate the strong performance of our method for unbounded immersive light field reconstruction.

cs.CV↗

arXiv2023

Perspective on non-Hermitian physics in magnetic systems

A perspective on non-Hermitian physics in magnetic systems is addressed in this short article, including exceptional points, exceptional nodal phases, the non-Hermitian SSH model, and the non-Hermitian skin effect.

cond-mat.mes-hall↗