arXiv subjects

Shiming Zhang

Publications and source records attributed to Shiming Zhang.

14 recordsLinked to original sources

arXiv2026

SynLeaF: A Dual-Stage Multimodal Fusion Framework for Synthetic Lethality Prediction Across Pan- and Single-Cancer Contexts

Accurate prediction of synthetic lethality (SL) is important for guiding the development of cancer drugs and therapies. SL prediction faces significant challenges in the effective fusion of heterogeneous multi-source data. Existing multimodal methods often suffer from "modality laziness" due to disparate convergence speeds, which hinders the exploitation of complementary information. This is also one reason why most existing SL prediction models cannot perform well on both pan-cancer and single-cancer SL pair prediction. In this study, we propose SynLeaF, a dual-stage multimodal fusion framework for SL prediction across pan- and single-cancer contexts. The framework employs a VAE-based cross-encoder with a product of experts mechanism to fuse four omics data types (gene expression, mutation, methylation, and CNV), while simultaneously utilizing a relational graph convolutional network to capture structured gene representations from biomedical knowledge graphs. To mitigate modality laziness, SynLeaF introduces a dual-stage training mechanism employing featurelevel knowledge distillation with adaptive uni-modal teacher and ensemble strategies. In extensive experiments across eight specific cancer types and a pancancer dataset, SynLeaF achieves superior performance in 17 out of 19 scenarios. Ablation studies and gradient analyses further validate the critical contributions of the proposed fusion and distillation mechanisms to model robustness and generalization. To facilitate community use, a web server is available at https://synleaf.bioinformatics-lilab.cn.

q-bio.GN↗

arXiv2025

MC#: Mixture Compressor for Mixture-of-Experts Large Models

Mixture-of-Experts (MoE) effectively scales large language models (LLMs) and vision-language models (VLMs) by increasing capacity through sparse activation. However, preloading all experts into memory and activating multiple experts per input introduces significant computational and memory overhead, making the expert module a major contributor to model size and inference cost. To address this, we propose MC# (Mixture-Compressor-sharp), a framework that combines static quantization and dynamic expert pruning by leveraging the significance of experts and tokens for aggressive compression of MoE-LLMs/VLMs. To reduce storage and loading costs, we introduce Pre-Loading Mixed-Precision Quantization (PMQ), which optimizes bit allocation via linear programming, balancing expert importance and quantization error for a Pareto-optimal trade-off between size and performance. To reduce runtime computation, Online Top-any Pruning (OTP) uses Gumbel-Softmax sampling to dynamically select a subset of experts per token, enabling fine-grained control over activation. By combining PMQ's static bit-width optimization with OTP's dynamic routing, MC# achieves extreme compression with minimal accuracy loss. On DeepSeek-VL2, MC# achieves a 6.2 times weight reduction at 2.57 average bits with only a 1.7% accuracy drop across five multimodal benchmarks. Additionally, OTP reduces expert activation over 20% with less than 1% performance degradation, demonstrating strong potential for efficient MoE-based model deployment.

cs.LG↗

arXiv2024

Synergy of turbulence and thermo-diffusive effects on the intermittent boundary-layer flashback of swirling flames

We simulated the intermittent boundary-layer flashback (BLF) of hydrogen-enriched swirling flames using large-eddy simulation (LES) with the flame-surface-density (FSD) method. Three cases of intermittent BLF, characterized by periodic flame entry and exit of the mixing tube, are presented. The intermittent BLF characteristics varied with the hydrogen volume fraction. Small flame bulges entered and exited the mixing tube in low hydrogen-enrichment cases. The duration of intermittent BLF events and BLF depth increased as the hydrogen content increased. Meanwhile, a large flame tongue penetrating deeply upstream characterised the highest hydrogen-enrichment case.The mean BLF peak depths and standard deviations obtained through simulations aligned well with experimental data for low and moderate hydrogen-enrichment cases. However, LES-FSD underestimated the average BLF peak depth for the highest hydrogen-enrichment case.Analysis of the flow-flame interaction revealed two mechanisms underlying the intermittent BLF phenomena. The flame bulges' oscillation near the outlet is caused by the reverse flow induced by the recirculation zone. At the same time, the deep intermittent BLF occurrs due to the boundary layer separation induced by the large turbulent burning velocity, resulting from the synergy of turbulence and thermo-diffusive effects.

physics.flu-dyn↗

arXiv2024

Mixture Compressor for Mixture-of-Experts LLMs Gains More

Mixture-of-Experts large language models (MoE-LLMs) marks a significant step forward of language models, however, they encounter two critical challenges in practice: 1) expert parameters lead to considerable memory consumption and loading latency; and 2) the current activated experts are redundant, as many tokens may only require a single expert. Motivated by these issues, we investigate the MoE-LLMs and make two key observations: a) different experts exhibit varying behaviors on activation reconstruction error, routing scores, and activated frequencies, highlighting their differing importance, and b) not all tokens are equally important -- only a small subset is critical. Building on these insights, we propose MC, a training-free Mixture-Compressor for MoE-LLMs, which leverages the significance of both experts and tokens to achieve an extreme compression. First, to mitigate storage and loading overheads, we introduce Pre-Loading Mixed-Precision Quantization, which formulates the adaptive bit-width allocation as a Linear Programming problem, where the objective function balances multi-factors reflecting the importance of each expert. Additionally, we develop Online Dynamic Pruning, which identifies important tokens to retain and dynamically select activated experts for other tokens during inference to optimize efficiency while maintaining performance. Our MC integrates static quantization and dynamic pruning to collaboratively achieve extreme compression for MoE-LLMs with less accuracy loss, ensuring an optimal trade-off between performance and efficiency. Extensive experiments confirm the effectiveness of our approach. For instance, at 2.54 bits, MC compresses 76.6% of the model, with only a 3.8% average accuracy loss. During dynamic inference, we further reduce activated parameters by 15%, with a performance drop of less than 0.6%.

cs.LG↗

arXiv2024

SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models

Post-training quantization (PTQ) is an effective technique for compressing large language models (LLMs). However, while uniform-precision quantization is computationally efficient, it often compromises model performance. To address this, we propose SliM-LLM, a salience-driven mixed-precision quantization framework that allocates bit-widths at the group-wise. Our approach leverages the observation that important weights follow a structured distribution and introduces two key components: \textbf{1)} \textit{Salience-Determined Bit Allocation} adaptively assigns bit-widths to groups within each layer based on their salience; and \textbf{2)} \textit{Salience-Weighted Quantizer Calibration} optimizes quantizer parameters by incorporating element-level salience. With its structured partitioning, SliM-LLM provides a hardware-friendly solution that matches the efficiency of uniform quantization methods while improving accuracy. Experiments show that SliM-LLM achieves superior performance across various LLMs at low bit-widths. For example, a 2-bit quantized LLaMA-7B model reduces memory usage by nearly 6x compared to the floating-point baseline, decreases perplexity by 48\% compared to state-of-the-art gradient-free PTQ methods, and maintains GPU inference speed. Additionally, the extended version, SliM-LLM$^+$, which incorporates gradient-based quantization, further reduces perplexity by 35.1\%. Our code is available at https://github.com/Aaronhuang-778/SliM-LLM

cs.LG↗

arXiv2024

Continual Learning for Smart City: A Survey

With the digitization of modern cities, large data volumes and powerful computational resources facilitate the rapid update of intelligent models deployed in smart cities. Continual learning (CL) is a novel machine learning paradigm that constantly updates models to adapt to changing environments, where the learning tasks, data, and distributions can vary over time. Our survey provides a comprehensive review of continual learning methods that are widely used in smart city development. The content consists of three parts: 1) Methodology-wise. We categorize a large number of basic CL methods and advanced CL frameworks in combination with other learning paradigms including graph learning, spatial-temporal learning, multi-modal learning, and federated learning. 2) Application-wise. We present numerous CL applications covering transportation, environment, public health, safety, networks, and associated datasets related to urban computing. 3) Challenges. We discuss current problems and challenges and envision several promising research directions. We believe this survey can help relevant researchers quickly familiarize themselves with the current state of continual learning research used in smart city development and direct them to future research trends.

cs.LG↗

arXiv2024

A Survey of Route Recommendations: Methods, Applications, and Opportunities

Nowadays, with advanced information technologies deployed citywide, large data volumes and powerful computational resources are intelligentizing modern city development. As an important part of intelligent transportation, route recommendation and its applications are widely used, directly influencing citizens` travel habits. Developing smart and efficient travel routes based on big data (possibly multi-modal) has become a central challenge in route recommendation research. Our survey offers a comprehensive review of route recommendation work based on urban computing. It is organized by the following three parts: 1) Methodology-wise. We categorize a large volume of traditional machine learning and modern deep learning methods. Also, we discuss their historical relations and reveal the edge-cutting progress. 2) Application\-wise. We present numerous novel applications related to route commendation within urban computing scenarios. 3) We discuss current problems and challenges and envision several promising research directions. We believe that this survey can help relevant researchers quickly familiarize themselves with the current state of route recommendation research and then direct them to future research trends.

cs.AI↗

arXiv2024

BiLLM: Pushing the Limit of Post-Training Quantization for LLMs

Pretrained large language models (LLMs) exhibit exceptional general language processing capabilities but come with significant demands on memory and computational resources. As a powerful compression technology, binarization can extremely reduce model weights to a mere 1 bit, lowering the expensive computation and memory requirements. However, existing quantization techniques fall short of maintaining LLM performance under ultra-low bit-widths. In response to this challenge, we present BiLLM, a groundbreaking 1-bit post-training quantization scheme tailored for pretrained LLMs. Based on the weight distribution of LLMs, BiLLM first identifies and structurally selects salient weights, and minimizes the compression loss through an effective binary residual approximation strategy. Moreover, considering the bell-shaped distribution of the non-salient weights, we propose an optimal splitting search to group and binarize them accurately. BiLLM achieving for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families and evaluation metrics, outperforms SOTA quantization methods of LLM by significant margins. Moreover, BiLLM enables the binarization process of the LLM with 7 billion weights within 0.5 hours on a single GPU, demonstrating satisfactory time efficiency. Our code is available at https://github.com/Aaronhuang-778/BiLLM.

cs.LG↗

arXiv2023

Pruning random resistive memory for optimizing analogue AI

The rapid advancement of artificial intelligence (AI) has been marked by the large language models exhibiting human-like intelligence. However, these models also present unprecedented challenges to energy consumption and environmental sustainability. One promising solution is to revisit analogue computing, a technique that predates digital computing and exploits emerging analogue electronic devices, such as resistive memory, which features in-memory computing, high scalability, and nonvolatility. However, analogue computing still faces the same challenges as before: programming nonidealities and expensive programming due to the underlying devices physics. Here, we report a universal solution, software-hardware co-design using structural plasticity-inspired edge pruning to optimize the topology of a randomly weighted analogue resistive memory neural network. Software-wise, the topology of a randomly weighted neural network is optimized by pruning connections rather than precisely tuning resistive memory weights. Hardware-wise, we reveal the physical origin of the programming stochasticity using transmission electron microscopy, which is leveraged for large-scale and low-cost implementation of an overparameterized random neural network containing high-performance sub-networks. We implemented the co-design on a 40nm 256K resistive memory macro, observing 17.3% and 19.9% accuracy improvements in image and audio classification on FashionMNIST and Spoken digits datasets, as well as 9.8% (2%) improvement in PR (ROC) in image segmentation on DRIVE datasets, respectively. This is accompanied by 82.1%, 51.2%, and 99.8% improvement in energy efficiency thanks to analogue in-memory computing. By embracing the intrinsic stochasticity and in-memory computing, this work may solve the biggest obstacle of analogue computing systems and thus unleash their immense potential for next-generation AI hardware.

cs.ET↗

arXiv2023

Modeling the boundary-layer flashback of premixed hydrogen-enriched swirling flames at high pressures

We model the boundary-layer flashback (BLF) of CH$_4$/H$_2$/air swirling flames via large-eddy simulations with the flame-surface-density method (LES-FSD), in particular, at high pressures. A local displacement speed model tabulating the stretched flame speed is employed to account for the thermo-diffusive effects, flame surface curvature, and heat loss in LES-FSD. The LES-FSD well captures the propagation characteristics during the BLF of swirling flames. In the LES-FSD for lean CH$_4$/H$_2$/air flames at 2.5 bar, the critical equivalence ratio for flashback decreases with the increasing hydrogen volume fraction, consistent with the experiments. This is due to the improved modeling of effects of the flame stretch and heat loss on the local displacement speed. We also develop a simple model to predict the BLF limits of swirling flames. The model estimates the critical bulk velocity for given reactants and swirl number, via the balance between the flame-induced pressure rise and adverse pressure for boundary-layer separation. We validate the model against 14 datasets of CH$_4$/H$_2$/air swirling flame experiments, with the hydrogen volume fractions in fuel from 50% to 100%. The present model well estimates the flashback limits in various operating conditions.

physics.flu-dyn↗

arXiv2022

A new technology perspective of the Metaverse: its essence, framework and challenges

The Metaverse depicts a parallel digitalized world where virtuality and reality are fused. It has economic and social systems like those in the real world and provides intelligent services and applications. In this paper, we introduce the Metaverse from a new technology perspective, including its essence, corresponding technical framework, and potential technical challenges. Specifically, we analyze the essence of the Metaverse from its etymology and point out breakthroughs promising to be made in time, space, and contents of the Metaverse by citing Maslow's Hierarchy of Needs. Subsequently, we conclude four pillars of the Metaverse, named ubiquitous connections, space convergence, virtuality and reality interaction, and human-centered communication, and establish a corresponding technical framework. Additionally, we envision open issues and challenges of the Metaverse in the technical aspect. The work proposes a new technology perspective of the Metaverse and will provide further guidance for its technology development in the future.

cs.CY↗

arXiv2021

GearV: A Two-Gear Hypervisor for Mixed-Criticality IoT Systems

This paper presents GearV, a two-gear lightweight hypervisor architecture to address the some known challenges. By dividing hypervisor into some partitions, and dividing scheduling policies into Gear1 and Gear2 respectively, GearV creates a consolidated platform to run best-effort system and safety-critical system simultaneously with managed engineering effort. The two-gears architecture also simplifies retrofitting the virtualization systems. We believe that GearV can serves as a reasonable hypervisor architecture for the mix-critical IoT systems.

cs.DC↗

arXiv2020

Deep Learning in Ultrasound Elastography Imaging

It is known that changes in the mechanical properties of tissues are associated with the onset and progression of certain diseases. Ultrasound elastography is a technique to characterize tissue stiffness using ultrasound imaging either by measuring tissue strain using quasi-static elastography or natural organ pulsation elastography, or by tracing a propagated shear wave induced by a source or a natural vibration using dynamic elastography. In recent years, deep learning has begun to emerge in ultrasound elastography research. In this review, several common deep learning frameworks in the computer vision community, such as multilayer perceptron, convolutional neural network, and recurrent neural network are described. Then, recent advances in ultrasound elastography using such deep learning techniques are revisited in terms of algorithm development and clinical diagnosis. Finally, the current challenges and future developments of deep learning in ultrasound elastography are prospected.

eess.IV↗

arXiv2011

Inverse Queries For Multidimensional Spaces

Traditional spatial queries return, for a given query object $q$, all database objects that satisfy a given predicate, such as epsilon range and $k$-nearest neighbors. This paper defines and studies {\em inverse} spatial queries, which, given a subset of database objects $Q$ and a query predicate, return all objects which, if used as query objects with the predicate, contain $Q$ in their result. We first show a straightforward solution for answering inverse spatial queries for any query predicate. Then, we propose a filter-and-refinement framework that can be used to improve efficiency. We show how to apply this framework on a variety of inverse queries, using appropriate space pruning strategies. In particular, we propose solutions for inverse epsilon range queries, inverse $k$-nearest neighbor queries, and inverse skyline queries. Our experiments show that our framework is significantly more efficient than naive approaches.

cs.DB↗