arXiv subjects

Liang Zhao

Publications and source records attributed to Liang Zhao.

At least 361 records · Page 20Linked to original sources

arXiv2022

Interpretable Molecular Graph Generation via Monotonic Constraints

Designing molecules with specific properties is a long-lasting research problem and is central to advancing crucial domains such as drug discovery and material science. Recent advances in deep graph generative models treat molecule design as graph generation problems which provide new opportunities toward the breakthrough of this long-lasting problem. Existing models, however, have many shortcomings, including poor interpretability and controllability toward desired molecular properties. This paper focuses on new methodologies for molecule generation with interpretable and controllable deep generative models, by proposing new monotonically-regularized graph variational autoencoders. The proposed models learn to represent the molecules with latent variables and then learn the correspondence between them and molecule properties parameterized by polynomial functions. To further improve the intepretability and controllability of molecule generation towards desired properties, we derive new objectives which further enforce monotonicity of the relation between some latent variables and target molecule properties such as toxicity and clogP. Extensive experimental evaluation demonstrates the superiority of the proposed framework on accuracy, novelty, disentanglement, and control towards desired molecular properties. The code is open-source at https://anonymous.4open.science/r/MDVAE-FD2C.

cs.LG↗

arXiv2022

Black-box Node Injection Attack for Graph Neural Networks

Graph Neural Networks (GNNs) have drawn significant attentions over the years and been broadly applied to vital fields that require high security standard such as product recommendation and traffic forecasting. Under such scenarios, exploiting GNN's vulnerabilities and further downgrade its classification performance become highly incentive for adversaries. Previous attackers mainly focus on structural perturbations of existing graphs. Although they deliver promising results, the actual implementation needs capability of manipulating the graph connectivity, which is impractical in some circumstances. In this work, we study the possibility of injecting nodes to evade the victim GNN model, and unlike previous related works with white-box setting, we significantly restrict the amount of accessible knowledge and explore the black-box setting. Specifically, we model the node injection attack as a Markov decision process and propose GA2C, a graph reinforcement learning framework in the fashion of advantage actor critic, to generate realistic features for injected nodes and seamlessly merge them into the original graph following the same topology characteristics. Through our extensive experiments on multiple acknowledged benchmark datasets, we demonstrate the superior performance of our proposed GA2C over existing state-of-the-art methods. The data and source code are publicly accessible at: https://github.com/jumxglhf/GA2C.

cs.LG↗

arXiv2022

Accelerated Gradient-free Neural Network Training by Multi-convex Alternating Optimization

In recent years, even though Stochastic Gradient Descent (SGD) and its variants are well-known for training neural networks, it suffers from limitations such as the lack of theoretical guarantees, vanishing gradients, and excessive sensitivity to input. To overcome these drawbacks, alternating minimization methods have attracted fast-increasing attention recently. As an emerging and open domain, however, several new challenges need to be addressed, including 1) Convergence properties are sensitive to penalty parameters, and 2) Slow theoretical convergence rate. We, therefore, propose a novel monotonous Deep Learning Alternating Minimization (mDLAM) algorithm to deal with these two challenges. Our innovative inequality-constrained formulation infinitely approximates the original problem with non-convex equality constraints, enabling our convergence proof of the proposed mDLAM algorithm regardless of the choice of hyperparameters. Our mDLAM algorithm is shown to achieve a fast linear convergence by the Nesterov acceleration technique. Extensive experiments on multiple benchmark datasets demonstrate the convergence, effectiveness, and efficiency of the proposed mDLAM algorithm. Our code is available at \url{https://github.com/xianggebenben/mDLAM}.

math.OC↗

arXiv2022

Aligning Eyes between Humans and Deep Neural Network through Interactive Attention Alignment

While Deep Neural Networks (DNNs) are deriving the major innovations in nearly every field through their powerful automation, we are also witnessing the peril behind automation as a form of bias, such as automated racism, gender bias, and adversarial bias. As the societal impact of DNNs grows, finding an effective way to steer DNNs to align their behavior with the human mental model has become indispensable in realizing fair and accountable models. We propose a novel framework of Interactive Attention Alignment (IAA) that aims at realizing human-steerable Deep Neural Networks (DNNs). IAA leverages DNN model explanation method as an interactive medium that humans can use to unveil the cases of biased model attention and directly adjust the attention. In improving the DNN using human-generated adjusted attention, we introduce GRADIA, a novel computational pipeline that jointly maximizes attention quality and prediction accuracy. We evaluated IAA framework in Study 1 and GRADIA in Study 2 in a gender classification problem. Study 1 found applying IAA can significantly improve the perceived quality of model attention from human eyes. In Study 2, we found using GRADIA can (1) significantly improve the perceived quality of model attention and (2) significantly improve model performance in scenarios where the training samples are limited. We present implications for future interactive user interfaces design towards human-alignable AI.

cs.AI↗

arXiv2022

Convergence and Applications of ADMM on the Multi-convex Problems

In recent years, although the Alternating Direction Method of Multipliers (ADMM) has been empirically applied widely to many multi-convex applications, delivering an impressive performance in areas such as nonnegative matrix factorization and sparse dictionary learning, there remains a dearth of generic work on proposed ADMM with a convergence guarantee under mild conditions. In this paper, we propose a generic ADMM framework with multiple coupled variables in both objective and constraints. Convergence to a Nash point is proven with a sublinear convergence rate $o(1/k)$. Two important applications are discussed as special cases under our proposed ADMM framework. Extensive experiments on ten real-world datasets demonstrate the proposed framework's effectiveness, scalability, and convergence properties. We have released our code at \url{https://github.com/xianggebenben/miADMM}.

math.OC↗

arXiv2022

Remarkably Enhanced Dynamic Oxygen Migration on Graphene Oxide Supported by Copper Substrate

The dynamic covalent properties of graphene oxide (GO) are of fundamental interest to a broad range of scientific areas and technological applications. It remains a challenge to access the feasible dynamic reactions for reversibly breaking/reforming covalent bonds of oxygen functional groups on GO, although these reactions can be induced by photonic or mechanical routes, or mediated by adsorbed water. Here, using the density functional theory calculations, we demonstrate the remarkably enhanced dynamic oxygen migration along the basal plane of GO supported by copper substrate (GO@copper), with the C-O bond breaking reaction and proton transfer between the neighboring epoxy and hydroxyl groups. Compared to that on GO, the energy barrier of oxygen migration on GO@copper is sharply reduced to be less than or comparable to thermal fluctuations, and meanwhile the crystallographic match between GO and copper substrate induces new oxygen migration paths on GO@copper. This work sheds light on the understanding of metal substrate-enhanced dynamic properties of GO, and evidences the strategy to tune the activity of two-dimension-interfacial oxygen groups for various potential applications.

cond-mat.mtrl-sci↗

arXiv2021

Do Multi-Lingual Pre-trained Language Models Reveal Consistent Token Attributions in Different Languages?

During the past several years, a surge of multi-lingual Pre-trained Language Models (PLMs) has been proposed to achieve state-of-the-art performance in many cross-lingual downstream tasks. However, the understanding of why multi-lingual PLMs perform well is still an open domain. For example, it is unclear whether multi-Lingual PLMs reveal consistent token attributions in different languages. To address this, in this paper, we propose a Cross-lingual Consistency of Token Attributions (CCTA) evaluation framework. Extensive experiments in three downstream tasks demonstrate that multi-lingual PLMs assign significantly different attributions to multi-lingual synonyms. Moreover, we have the following observations: 1) the Spanish achieves the most consistent token attributions in different languages when it is used for training PLMs; 2) the consistency of token attributions strongly correlates with performance in downstream tasks.

cs.CL↗

arXiv2021

A Convergent ADMM Framework for Efficient Neural Network Training

As a well-known optimization framework, the Alternating Direction Method of Multipliers (ADMM) has achieved tremendous success in many classification and regression applications. Recently, it has attracted the attention of deep learning researchers and is considered to be a potential substitute to Gradient Descent (GD). However, as an emerging domain, several challenges remain unsolved, including 1) The lack of global convergence guarantees, 2) Slow convergence towards solutions, and 3) Cubic time complexity with regard to feature dimensions. In this paper, we propose a novel optimization framework to solve a general neural network training problem via ADMM (dlADMM) to address these challenges simultaneously. Specifically, the parameters in each layer are updated backward and then forward so that parameter information in each layer is exchanged efficiently. When the dlADMM is applied to specific architectures, the time complexity of subproblems is reduced from cubic to quadratic via a dedicated algorithm design utilizing quadratic approximations and backtracking techniques. Last but not least, we provide the first proof of convergence to a critical point sublinearly for an ADMM-type method (dlADMM) under mild conditions. Experiments on seven benchmark datasets demonstrate the convergence, efficiency, and effectiveness of our proposed dlADMM algorithm.

cs.LG↗

arXiv2021

Community-based Layerwise Distributed Training of Graph Convolutional Networks

The Graph Convolutional Network (GCN) has been successfully applied to many graph-based applications. Training a large-scale GCN model, however, is still challenging: Due to the node dependency and layer dependency of the GCN architecture, a huge amount of computational time and memory is required in the training process. In this paper, we propose a parallel and distributed GCN training algorithm based on the Alternating Direction Method of Multipliers (ADMM) to tackle the two challenges simultaneously. We first split GCN layers into independent blocks to achieve layer parallelism. Furthermore, we reduce node dependency by dividing the graph into several dense communities such that each of them can be trained with an agent in parallel. Finally, we provide solutions for all subproblems in the community-based ADMM algorithm. Preliminary results demonstrate that our proposed community-based ADMM training algorithm can lead to more than triple speedup while achieving the best performance compared with state-of-the-art methods.

cs.LG↗

arXiv2021

Adaptive Kernel Graph Neural Network

Graph neural networks (GNNs) have demonstrated great success in representation learning for graph-structured data. The layer-wise graph convolution in GNNs is shown to be powerful at capturing graph topology. During this process, GNNs are usually guided by pre-defined kernels such as Laplacian matrix, adjacency matrix, or their variants. However, the adoptions of pre-defined kernels may restrain the generalities to different graphs: mismatch between graph and kernel would entail sub-optimal performance. For example, GNNs that focus on low-frequency information may not achieve satisfactory performance when high-frequency information is significant for the graphs, and vice versa. To solve this problem, in this paper, we propose a novel framework - i.e., namely Adaptive Kernel Graph Neural Network (AKGNN) - which learns to adapt to the optimal graph kernel in a unified manner at the first attempt. In the proposed AKGNN, we first design a data-driven graph kernel learning mechanism, which adaptively modulates the balance between all-pass and low-pass filters by modifying the maximal eigenvalue of the graph Laplacian. Through this process, AKGNN learns the optimal threshold between high and low frequency signals to relieve the generality problem. Later, we further reduce the number of parameters by a parameterization trick and enhance the expressive power by a global readout function. Extensive experiments are conducted on acknowledged benchmark datasets and promising results demonstrate the outstanding performance of our proposed AKGNN by comparison with state-of-the-art GNNs. The source code is publicly available at: https://github.com/jumxglhf/AKGNN.

cs.LG↗

arXiv2021

KNAS: Green Neural Architecture Search

Many existing neural architecture search (NAS) solutions rely on downstream training for architecture evaluation, which takes enormous computations. Considering that these computations bring a large carbon footprint, this paper aims to explore a green (namely environmental-friendly) NAS solution that evaluates architectures without training. Intuitively, gradients, induced by the architecture itself, directly decide the convergence and generalization results. It motivates us to propose the gradient kernel hypothesis: Gradients can be used as a coarse-grained proxy of downstream training to evaluate random-initialized networks. To support the hypothesis, we conduct a theoretical analysis and find a practical gradient kernel that has good correlations with training loss and validation performance. According to this hypothesis, we propose a new kernel based architecture search approach KNAS. Experiments show that KNAS achieves competitive results with orders of magnitude faster than "train-then-test" paradigms on image classification tasks. Furthermore, the extremely low search cost enables its wide applications. The searched network also outperforms strong baseline RoBERTA-large on two text classification tasks. Codes are available at \url{https://github.com/Jingjing-NLP/KNAS} .

cs.LG↗

arXiv2021

Existence and convergence of solutions for nonlinear elliptic systems on graphs

We consider a kind of nonlinear systems on a locally finite graphs $G=(V,E)$. We prove via the mountain pass theorem that this kind of systems has a nontrivial ground state solution which depends on the parameter $λ$ with some suitable assumptions on the potentials. Moreover, we pay attention to the concentration behavior of these solutions and prove that, as $λ\to \infty$, these solutions converge to a ground state solution of a corresponding Dirichlet problem. Finally, we also provide some numerical experiments to illustrate our results.

math.AP↗

arXiv2021

Wafer-scale, full-coverage, acoustic self-limiting assembly of particles on flexible substrates

Self-limiting assembly of particles represents the state-of-the-art controllability in nanomanufacturing processes where the assembly stops at a designated stage1,2, providing a desirable platform for applications requiring delicate thickness control3-5. Most successes in self-limiting assembly are limited to self-assembled monolayers (SAM) of small molecules on inorganic, chemically homogeneous rigid substrates (e.g., Au and SiO2) through surface-interaction mechanisms6,7. Similar mechanisms, however, cannot achieve uniform assembly of particles on flexible polymer substrates8,9. The complex configurations and conformations of polymer chains create a surface with non-uniform distributions of chemical groups and phases. In addition, most assembly mechanisms require good solvent wettability, where many desirable but hard-to-wet particles and polymer substrates are excluded. Here, we demonstrate an acoustic self-limiting assembly of particles (ASAP) method to achieve wafer-scale, full-coverage, close-packed monolayers of hydrophobic particles on hydrophobic polymer substrates in aqueous solutions. We envision many applications in functional coatings and showcase its feasibility in structural coloration.

physics.atm-clus↗

arXiv2021

Heterogeneous Temporal Graph Neural Network

Graph neural networks (GNNs) have been broadly studied on dynamic graphs for their representation learning, majority of which focus on graphs with homogeneous structures in the spatial domain. However, many real-world graphs - i.e., heterogeneous temporal graphs (HTGs) - evolve dynamically in the context of heterogeneous graph structures. The dynamics associated with heterogeneity have posed new challenges for HTG representation learning. To solve this problem, in this paper, we propose heterogeneous temporal graph neural network (HTGNN) to integrate both spatial and temporal dependencies while preserving the heterogeneity to learn node representations over HTGs. Specifically, in each layer of HTGNN, we propose a hierarchical aggregation mechanism, including intra-relation, inter-relation, and across-time aggregations, to jointly model heterogeneous spatial dependencies and temporal dimensions. To retain the heterogeneity, intra-relation aggregation is first performed over each slice of HTG to attentively aggregate information of neighbors with the same type of relation, and then intra-relation aggregation is exploited to gather information over different types of relations; to handle temporal dependencies, across-time aggregation is conducted to exchange information across different graph slices over the HTG. The proposed HTGNN is a holistic framework tailored heterogeneity with evolution in time and space for HTG representation learning. Extensive experiments are conducted on the HTGs built from different real-world datasets and promising results demonstrate the outstanding performance of HTGNN by comparison with state-of-the-art baselines. Our built HTGs and code have been made publicly accessible at: https://github.com/YesLab-Code/HTGNN.

cs.LG↗

arXiv2021

Self-healing mechanism of lithium in lithium metal batteries

Li metal is an ideal anode material for use in state-of-the-art secondary batteries. However, Li-dendrite growth is a safety concern and results in low coulombic efficiency, which significantly restricts the commercial application of Li secondary batteries. Unfortunately, the Li deposition (growth) mechanism is poorly understood on the atomic scale. Here, we used machine learning to construct a Li potential model with quantum-mechanical computational accuracy. Molecular dynamics simulations in this study with this model revealed two self-healing mechanisms in a large Li-metal system, viz. surface self-healing and bulk self-healing, and identified three Li-dendrite morphologies under different conditions, viz. "needle", "mushroom", and "hemisphere". Finally, we introduce the concepts of local current density and variance in local current density to supplement the critical current density when evaluating the probability of self-healing.

cond-mat.mtrl-sci↗

arXiv2021

A Right Invariant Extended Kalman Filter for Object based SLAM

With the recent advance of deep learning based object recognition and estimation, it is possible to consider object level SLAM where the pose of each object is estimated in the SLAM process. In this paper, based on a novel Lie group structure, a right invariant extended Kalman filter (RI-EKF) for object based SLAM is proposed. The observability analysis shows that the proposed algorithm automatically maintains the correct unobservable subspace, while standard EKF (Std-EKF) based SLAM algorithm does not. This results in a better consistency for the proposed algorithm comparing to Std-EKF. Finally, simulations and real world experiments validate not only the consistency and accuracy of the proposed algorithm, but also the practicability of the proposed RI-EKF for object based SLAM problem. The MATLAB code of the algorithm is made publicly available.

cs.RO↗

arXiv2021

FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tiers

Federated learning (FL) involves training a model over massive distributed devices, while keeping the training data localized. This form of collaborative learning exposes new tradeoffs among model convergence speed, model accuracy, balance across clients, and communication cost, with new challenges including: (1) straggler problem, where the clients lag due to data or (computing and network) resource heterogeneity, and (2) communication bottleneck, where a large number of clients communicate their local updates to a central server and bottleneck the server. Many existing FL methods focus on optimizing along only one dimension of the tradeoff space. Existing solutions use asynchronous model updating or tiering-based synchronous mechanisms to tackle the straggler problem. However, the asynchronous methods can easily create a network communication bottleneck, while tiering may introduce biases as tiering favors faster tiers with shorter response latencies. To address these issues, we present FedAT, a novel Federated learning method with Asynchronous Tiers under Non-i.i.d. data. FedAT synergistically combines synchronous intra-tier training and asynchronous cross-tier training. By bridging the synchronous and asynchronous training through tiering, FedAT minimizes the straggler effect with improved convergence speed and test accuracy. FedAT uses a straggler-aware, weighted aggregation heuristic to steer and balance the training for further accuracy improvement. FedAT compresses the uplink and downlink communications using an efficient, polyline-encoding-based compression algorithm, therefore minimizing the communication cost. Results show that FedAT improves the prediction performance by up to 21.09%, and reduces the communication cost by up to 8.5x, compared to state-of-the-art FL methods.

cs.DC↗

arXiv2021

A Method for Inferring Polymers Based on Linear Regression and Integer Programming

A novel framework has recently been proposed for designing the molecular structure of chemical compounds with a desired chemical property using both artificial neural networks and mixed integer linear programming. In this paper, we design a new method for inferring a polymer based on the framework. For this, we introduce a new way of representing a polymer as a form of monomer and define new descriptors that feature the structure of polymers. We also use linear regression as a building block of constructing a prediction function in the framework. The results of our computational experiments reveal a set of chemical properties on polymers to which a prediction function constructed with linear regression performs well. We also observe that the proposed method can infer polymers with up to 50 non-hydrogen atoms in a monomer form.

cs.LG↗