arXiv subjects

Feng Ji

Publications and source records attributed to Feng Ji.

At least 73 records · Page 4Linked to original sources

arXiv2023

AMTSS: An Adaptive Multi-Teacher Single-Student Knowledge Distillation Framework For Multilingual Language Inference

Knowledge distillation is of key importance to launching multilingual pre-trained language models for real applications. To support cost-effective language inference in multilingual settings, we propose AMTSS, an adaptive multi-teacher single-student distillation framework, which allows distilling knowledge from multiple teachers to a single student. We first introduce an adaptive learning strategy and teacher importance weight, which enables a student to effectively learn from max-margin teachers and easily adapt to new languages. Moreover, we present a shared student encoder with different projection layers in support of multiple languages, which contributes to largely reducing development and machine cost. Experimental results show that AMTSS gains competitive results on the public XNLI dataset and the realistic industrial dataset AliExpress (AE) in the E-commerce scenario.

cs.CL↗

arXiv2023

Leveraging Label Non-Uniformity for Node Classification in Graph Neural Networks

In node classification using graph neural networks (GNNs), a typical model generates logits for different class labels at each node. A softmax layer often outputs a label prediction based on the largest logit. We demonstrate that it is possible to infer hidden graph structural information from the dataset using these logits. We introduce the key notion of label non-uniformity, which is derived from the Wasserstein distance between the softmax distribution of the logits and the uniform distribution. We demonstrate that nodes with small label non-uniformity are harder to classify correctly. We theoretically analyze how the label non-uniformity varies across the graph, which provides insights into boosting the model performance: increasing training samples with high non-uniformity or dropping edges to reduce the maximal cut size of the node set of small non-uniformity. These mechanisms can be easily added to a base GNN model. Experimental results demonstrate that our approach improves the performance of many benchmark base models.

cs.LG↗

arXiv2023

Effective Numerical Simulations of Synchronous Generator System

Synchronous generator system is a complicated dynamical system for energy transmission, which plays an important role in modern industrial production. In this article, we propose some predictor-corrector methods and structure-preserving methods for a generator system based on the first benchmark model of subsynchronous resonance, among which the structure-preserving methods preserve a Dirac structure associated with the so-called port-Hamiltonian descriptor systems. To illustrate this, the simplified generator system in the form of index-1 differential-algebraic equations has been derived. Our analyses provide the global error estimates for a special class of structure-preserving methods called Gauss methods, which guarantee their superior performance over the PSCAD/EMTDC and the predictor-corrector methods in terms of computational stability. Numerical simulations are implemented to verify the effectiveness and advantages of our methods.

math.NA↗

arXiv2023

Distributional Signals for Node Classification in Graph Neural Networks

In graph neural networks (GNNs), both node features and labels are examples of graph signals, a key notion in graph signal processing (GSP). While it is common in GSP to impose signal smoothness constraints in learning and estimation tasks, it is unclear how this can be done for discrete node labels. We bridge this gap by introducing the concept of distributional graph signals. In our framework, we work with the distributions of node labels instead of their values and propose notions of smoothness and non-uniformity of such distributional graph signals. We then propose a general regularization method for GNNs that allows us to encode distributional smoothness and non-uniformity of the model output in semi-supervised node classification tasks. Numerical experiments demonstrate that our method can significantly improve the performance of most base GNN models in different problem settings.

eess.SP↗

arXiv2023

Graph Signal Processing over a Probability Space of Shift Operators

Graph signal processing (GSP) uses a shift operator to define a Fourier basis for the set of graph signals. The shift operator is often chosen to capture the graph topology. However, in many applications, the graph topology may be unknown a priori, its structure uncertain, or generated randomly from a predefined set for each observation. Each graph topology gives rise to a different shift operator. In this paper, we develop a GSP framework over a probability space of shift operators. We develop the corresponding notions of Fourier transform, MFC filters, and band-pass filters, which subsumes classical GSP theory as the special case where the probability space consists of a single shift operator. We show that an MFC filter under this framework is the expectation of random convolution filters in classical GSP, while the notion of bandlimitedness requires additional wiggle room from being simply a fixed point of a band-pass filter. We develop a mechanism that facilitates mapping from one space of shift operators to another, which allows our framework to be applied to a rich set of scenarios. We demonstrate how the theory can be applied by using both synthetic and real datasets.

eess.SP↗

arXiv2023

To further understand graph signals

Graph signal processing (GSP) is a framework to analyze and process graph-structured data. Many research works focus on developing tools such as Graph Fourier transforms (GFT), filters, and neural network models to handle graph signals. Such approaches have successfully taken care of ``signal processing'' in many circumstances. In this paper, we want to put emphasis on ``graph signals'' themselves. Although there are characterizations of graph signals using the notion of bandwidth derived from GFT, we want to argue here that graph signals may contain hidden geometric information of the network, independent of (graph) Fourier theories. We shall provide a framework to understand such information, and demonstrate how new knowledge on ``graph signals'' can help with ``signal processing''.

eess.SP↗

arXiv2023

Node-Specific Space Selection via Localized Geometric Hyperbolicity in Graph Neural Networks

Many graph neural networks have been developed to learn graph representations in either Euclidean or hyperbolic space, with all nodes' representations embedded in a single space. However, a graph can have hyperbolic and Euclidean geometries at different regions of the graph. Thus, it is sub-optimal to indifferently embed an entire graph into a single space. In this paper, we explore and analyze two notions of local hyperbolicity, describing the underlying local geometry: geometric (Gromov) and model-based, to determine the preferred space of embedding for each node. The two hyperbolicities' distributions are aligned using the Wasserstein metric such that the calculated geometric hyperbolicity guides the choice of the learned model hyperbolicity. As such our model Joint Space Graph Neural Network (JSGNN) can leverage both Euclidean and hyperbolic spaces during learning by allowing node-specific geometry space selection. We evaluate our model on both node classification and link prediction tasks and observe promising performance compared to baseline models.

cs.LG↗

arXiv2023

Learning on heterogeneous graphs using high-order relations

A heterogeneous graph consists of different vertices and edges types. Learning on heterogeneous graphs typically employs meta-paths to deal with the heterogeneity by reducing the graph to a homogeneous network, guide random walks or capture semantics. These methods are however sensitive to the choice of meta-paths, with suboptimal paths leading to poor performance. In this paper, we propose an approach for learning on heterogeneous graphs without using meta-paths. Specifically, we decompose a heterogeneous graph into different homogeneous relation-type graphs, which are then combined to create higher-order relation-type representations. These representations preserve the heterogeneity of edges and retain their edge directions while capturing the interaction of different vertex types multiple hops apart. This is then complemented with attention mechanisms to distinguish the importance of the relation-type based neighbors and the relation-types themselves. Experiments demonstrate that our model generally outperforms other state-of-the-art baselines in the vertex classification task on three commonly studied heterogeneous graph datasets.

stat.ML↗

arXiv2023

Graph signal processing with categorical perspective

In this paper, we propose a framework for graph signal processing using category theory. The aim is to generalize a few recent works on probabilistic approaches to graph signal processing, which handle signal and graph uncertainties.

eess.SP↗

arXiv2023

On distributional graph signals

Graph signal processing (GSP) studies graph-structured data, where the central concept is the vector space of graph signals. To study a vector space, we have many useful tools up our sleeves. However, uncertainty is omnipresent in practice, and using a vector to model a real signal can be erroneous in some situations. In this paper, we want to use the Wasserstein space as a replacement for the vector space of graph signals, to account for signal stochasticity. The Wasserstein is strictly more general in which the classical graph signal space embeds isometrically. An element in the Wasserstein space is called a distributional graph signal. On the other hand, signal processing for a probability space of graphs has been proposed in the literature. In this work, we propose a unified framework that also encompasses existing theories regarding graph uncertainty. We develop signal processing tools to study the new notion of distributional graph signals. We also demonstrate how the theory can be applied by using real datasets.

eess.SP↗

arXiv2022

Learning to Expand: Reinforced Pseudo-relevance Feedback Selection for Information-seeking Conversations

Information-seeking conversation systems are increasingly popular in real-world applications, especially for e-commerce companies. To retrieve appropriate responses for users, it is necessary to compute the matching degrees between candidate responses and users' queries with historical dialogue utterances. As the contexts are usually much longer than responses, it is thus necessary to expand the responses (usually short) with richer information. Recent studies on pseudo-relevance feedback (PRF) have demonstrated its effectiveness in query expansion for search engines, hence we consider expanding response using PRF information. However, existing PRF approaches are either based on heuristic rules or require heavy manual labeling, which are not suitable for solving our task. To alleviate this problem, we treat the PRF selection for response expansion as a learning task and propose a reinforced learning method that can be trained in an end-to-end manner without any human annotations. More specifically, we propose a reinforced selector to extract useful PRF terms to enhance response candidates and a BERT-based response ranker to rank the PRF-enhanced responses. The performance of the ranker serves as a reward to guide the selector to extract useful PRF terms, which boosts the overall task performance. Extensive experiments on both standard benchmarks and commercial datasets prove the superiority of our reinforced PRF term selector compared with other potential soft or hard selection methods. Both case studies and quantitative analysis show that our model is capable of selecting meaningful PRF terms to expand response candidates and also achieving the best results compared with all baselines on a variety of evaluation metrics. We have also deployed our method on online production in an e-commerce company, which shows a significant improvement over the existing online ranking system.

cs.CL↗

arXiv2022

Sampling of Correlated Bandlimited Continuous Signals by Joint Time-vertex Graph Fourier Transform

When sampling multiple signals, the correlation between the signals can be exploited to reduce the overall number of samples. In this paper, we study the sampling theory of multiple correlated signals, using correlation to sample them at the lowest sampling rate. Based on the correlation between signal sources, we model multiple continuous-time signals as continuous time-vertex graph signals. The graph signals are projected onto orthogonal bases to remove spatial correlation and reduce dimensions by graph Fourier transform. When the bandwidths of the original signals and the reduced dimension signals are given, we prove the minimum sampling rate required for recovery of the original signals, and propose a feasible sampling scheme.

eess.SP↗

arXiv2022

On semi shift invariant graph filters

In graph signal processing, one of the most important subjects is the study of filters, i.e., linear transformations that capture relations between graph signals. One of the most important families of filters is the space of shift invariant filters, defined as transformations commute with a preferred graph shift operator. Shift invariant filters have a wide range of applications in graph signal processing and graph neural networks. A shift invariant filter can be interpreted geometrically as an information aggregation procedure (from local neighborhood), and can be computed easily using matrix multiplication. However, there are still drawbacks to using solely shift invariant filters in applications, such as being restrictively homogeneous. In this paper, we generalize shift invariant filters by introducing and studying semi shift invariant filters. We give an application of semi shift invariant filters with a new signal processing framework, the subgraph signal processing. Moreover, we also demonstrate how semi shift invariant filters can be used in graph neural networks.

eess.SP↗

arXiv2022

SGAT: Simplicial Graph Attention Network

Heterogeneous graphs have multiple node and edge types and are semantically richer than homogeneous graphs. To learn such complex semantics, many graph neural network approaches for heterogeneous graphs use metapaths to capture multi-hop interactions between nodes. Typically, features from non-target nodes are not incorporated into the learning procedure. However, there can be nonlinear, high-order interactions involving multiple nodes or edges. In this paper, we present Simplicial Graph Attention Network (SGAT), a simplicial complex approach to represent such high-order interactions by placing features from non-target nodes on the simplices. We then use attention mechanisms and upper adjacencies to generate representations. We empirically demonstrate the efficacy of our approach with node classification tasks on heterogeneous graph datasets and further show SGAT's ability in extracting structural information by employing random node features. Numerical experiments indicate that SGAT performs better than other current state-of-the-art heterogeneous graph learning methods.

cs.LG↗

arXiv2022

Abstract message passing and distributed graph signal processing

Graph signal processing is a framework to handle graph structured data. The fundamental concept is graph shift operator, giving rise to the graph Fourier transform. While the graph Fourier transform is a centralized procedure, distributed graph signal processing algorithms are needed to address challenges such as scalability and privacy. In this paper, we develop a theory of distributed graph signal processing based on the classical notion of message passing. However, we generalize the definition of a message to permit more abstract mathematical objects. The framework provides an alternative point of view that avoids the iterative nature of existing approaches to distributed graph signal processing. Moreover, our framework facilitates investigating theoretical questions such as solubility of distributed problems.

eess.SP↗

arXiv2021

Sampling Theory of Bandlimited Continuous-Time Graph Signals

A continuous-time graph signal can be viewed as a time series of graph signals. It generalizes both the classical continuous-time signal and ordinary graph signal. Therefore, such a signal can be considered as a function on two domains: the graph domain and the time domain. In this paper, we consider the sampling theory of bandlimited continuous-time graph signals. To formulate the sampling problem, we need to consider the interaction between the graph and time domains. We describe an explicit procedure to determine a discrete sampling set for perfect signal recovery. Moreover, in analogous to the Nyquist-Shannon sampling theorem, we give an explicit formula for the minimal sample rate.

eess.SP↗

arXiv2021

Recovery of Graph Signals from Sign Measurements

Sampling and interpolation have been extensively studied, in order to reconstruct or estimate the entire graph signal from the signal values on a subset of vertexes, of which most achievements are about continuous signals. While in a lot of signal processing tasks, signals are not fully observed, and only the signs of signals are available, for example a rating system may only provide several simple options. In this paper, the reconstruction of band-limited graph signals based on sign sampling is discussed and a greedy sampling strategy is proposed. The simulation experiments are presented, and the greedy sampling algorithm is compared with random sampling algorithm, which verify the validity of the proposed approach.

eess.SP↗

arXiv2021

"Wait, I'm Still Talking!" Predicting the Dialogue Interaction Behavior Using Imagine-Then-Arbitrate Model

Producing natural and accurate responses like human beings is the ultimate goal of intelligent dialogue agents. So far, most of the past works concentrate on selecting or generating one pertinent and fluent response according to current query and its context. These models work on a one-to-one environment, making one response to one utterance each round. However, in real human-human conversations, human often sequentially sends several short messages for readability instead of a long message in one turn. Thus messages will not end with an explicit ending signal, which is crucial for agents to decide when to reply. So the first step for an intelligent dialogue agent is not replying but deciding if it should reply at the moment. To address this issue, in this paper, we propose a novel Imagine-then-Arbitrate (ITA) neural dialogue model to help the agent decide whether to wait or to make a response directly. Our method has two imaginator modules and an arbitrator module. The two imaginators will learn the agent's and user's speaking style respectively, generate possible utterances as the input of the arbitrator, combining with dialogue history. And the arbitrator decides whether to wait or to make a response to the user directly. To verify the performance and effectiveness of our method, we prepared two dialogue datasets and compared our approach with several popular models. Experimental results show that our model performs well on addressing ending prediction issue and outperforms baseline models.

cs.CL↗