arXiv subjects

Haifeng Zhang

Publications and source records attributed to Haifeng Zhang.

At least 73 records · Page 4Linked to original sources

arXiv2022

A Game-Theoretic Approach for Improving Generalization Ability of TSP Solvers

In this paper, we introduce a two-player zero-sum framework between a trainable \emph{Solver} and a \emph{Data Generator} to improve the generalization ability of deep learning-based solvers for Traveling Salesman Problem (TSP). Grounded in \textsl{Policy Space Response Oracle} (PSRO) methods, our two-player framework outputs a population of best-responding Solvers, over which we can mix and output a combined model that achieves the least exploitability against the Generator, and thereby the most generalizable performance on different TSP tasks. We conduct experiments on a variety of TSP instances with different types and sizes. Results suggest that our Solvers achieve the state-of-the-art performance even on tasks the Solver never meets, whilst the performance of other deep learning-based Solvers drops sharply due to over-fitting. To demonstrate the principle of our framework, we study the learning outcome of the proposed two-player game and demonstrate that the exploitability of the Solver population decreases during training, and it eventually approximates the Nash equilibrium along with the Generator.

cs.LG↗

arXiv2022

Settling the Variance of Multi-Agent Policy Gradients

Policy gradient (PG) methods are popular reinforcement learning (RL) methods where a baseline is often applied to reduce the variance of gradient estimates. In multi-agent RL (MARL), although the PG theorem can be naturally extended, the effectiveness of multi-agent PG (MAPG) methods degrades as the variance of gradient estimates increases rapidly with the number of agents. In this paper, we offer a rigorous analysis of MAPG methods by, firstly, quantifying the contributions of the number of agents and agents' explorations to the variance of MAPG estimators. Based on this analysis, we derive the optimal baseline (OB) that achieves the minimal variance. In comparison to the OB, we measure the excess variance of existing MARL algorithms such as vanilla MAPG and COMA. Considering using deep neural networks, we also propose a surrogate version of OB, which can be seamlessly plugged into any existing PG methods in MARL. On benchmarks of Multi-Agent MuJoCo and StarCraft challenges, our OB technique effectively stabilises training and improves the performance of multi-agent PPO and COMA algorithms by a significant margin.

cs.LG↗

arXiv2022

Learning to Identify Top Elo Ratings: A Dueling Bandits Approach

The Elo rating system is widely adopted to evaluate the skills of (chess) game and sports players. Recently it has been also integrated into machine learning algorithms in evaluating the performance of computerised AI agents. However, an accurate estimation of the Elo rating (for the top players) often requires many rounds of competitions, which can be expensive to carry out. In this paper, to improve the sample efficiency of the Elo evaluation (for top players), we propose an efficient online match scheduling algorithm. Specifically, we identify and match the top players through a dueling bandits framework and tailor the bandit algorithm to the gradient-based update of Elo. We show that it reduces the per-step memory and time complexity to constant, compared to the traditional likelihood maximization approaches requiring $O(t)$ time. Our algorithm has a regret guarantee of $\tilde{O}(\sqrt{T})$, sublinear in the number of competition rounds and has been extended to the multidimensional Elo ratings for handling intransitive games. We empirically demonstrate that our method achieves superior convergence speed and time efficiency on a variety of gaming tasks.

cs.LG↗

arXiv2022

GCS: Graph-based Coordination Strategy for Multi-Agent Reinforcement Learning

Many real-world scenarios involve a team of agents that have to coordinate their policies to achieve a shared goal. Previous studies mainly focus on decentralized control to maximize a common reward and barely consider the coordination among control policies, which is critical in dynamic and complicated environments. In this work, we propose factorizing the joint team policy into a graph generator and graph-based coordinated policy to enable coordinated behaviours among agents. The graph generator adopts an encoder-decoder framework that outputs directed acyclic graphs (DAGs) to capture the underlying dynamic decision structure. We also apply the DAGness-constrained and DAG depth-constrained optimization in the graph generator to balance efficiency and performance. The graph-based coordinated policy exploits the generated decision structure. The graph generator and coordinated policy are trained simultaneously to maximize the discounted return. Empirical evaluations on Collaborative Gaussian Squeeze, Cooperative Navigation, and Google Research Football demonstrate the superiority of the proposed method.

cs.MA↗

arXiv2021

Large deviation and anomalous fluctuations scaling in degree assortativity on configuration networks

By constructing a multicanonical Monte Carlo simulation, we obtain the full probability distribution $ρ_N(r)$ of the degree assortativity coefficient $r$ on configuration networks of size $N$ by using the multiple histogram reweighting method. We suggest that $ρ_N(r)$ obeys a large deviation principle, $ρ_N \left(r-r_N^* \right) \asymp {e^{ - {N^ξ}I\left( {r- r_N^* } \right)}}$, where the rate function $I$ is convex and possesses its unique minimum at $r=r_N^*$, and $ξ$ is an exponent that scales $ρ_N$'s with $N$. We show that $ξ=1$ for Poisson random graphs, and $ξ\geq1$ for scale-free networks in which $ξ$ is a decreasing function of the degree distribution exponent $γ$. Our results reveal that the fluctuations of $r$ exhibits an anomalous scaling with $N$ in highly heterogeneous networks.

cond-mat.stat-mech↗

arXiv2020

Non-Markovian Majority-Vote model

Non-Markovian dynamics pervades human activity and social networks and it induces memory effects and burstiness in a wide range of processes including inter-event time distributions, duration of interactions in temporal networks and human mobility. Here we propose a non-Markovian Majority-Vote model (NMMV) that introduces non-Markovian effects in the standard (Markovian) Majority-Vote model (SMV). The SMV model is one of the simplest two-state stochastic models for studying opinion dynamics, and displays a continuous order-disorder phase transition at a critical noise. In the NMMV model we assume that the probability that an agent changes state is not only dependent on the majority state of his neighbors but it also depends on his {\em age}, i.e. how long the agent has been in his current state. The NMMV model has two regimes: the aging regime implies that the probability that an agent changes state is decreasing with his age, while in the anti-aging regime the probability that an agent changes state is increasing with his age. Interestingly, we find that the critical noise at which we observe the order-disorder phase transition is a non-monotonic function of the rate $β$ of the aging (anti-aging) process. In particular the critical noise in the aging regime displays a maximum as a function of $β$ while in the anti-aging regime displays a minimum. This implies that the aging/anti-aging dynamics can retard/anticipate the transition and that there is an optimal rate $β$ for maximally perturbing the value of the critical noise. The analytical results obtained in the framework of the heterogeneous mean-field approach are validated by extensive numerical simulations on a large variety of network topologies.

cond-mat.stat-mech↗

arXiv2020

Improving Knowledge Tracing via Pre-training Question Embeddings

Knowledge tracing (KT) defines the task of predicting whether students can correctly answer questions based on their historical response. Although much research has been devoted to exploiting the question information, plentiful advanced information among questions and skills hasn't been well extracted, making it challenging for previous work to perform adequately. In this paper, we demonstrate that large gains on KT can be realized by pre-training embeddings for each question on abundant side information, followed by training deep KT models on the obtained embeddings. To be specific, the side information includes question difficulty and three kinds of relations contained in a bipartite graph between questions and skills. To pre-train the question embeddings, we propose to use product-based neural networks to recover the side information. As a result, adopting the pre-trained embeddings in existing deep KT models significantly outperforms state-of-the-art baselines on three common KT datasets.

cs.IR↗

arXiv2020

Bi-level Actor-Critic for Multi-agent Coordination

Coordination is one of the essential problems in multi-agent systems. Typically multi-agent reinforcement learning (MARL) methods treat agents equally and the goal is to solve the Markov game to an arbitrary Nash equilibrium (NE) when multiple equilibra exist, thus lacking a solution for NE selection. In this paper, we treat agents \emph{unequally} and consider Stackelberg equilibrium as a potentially better convergence point than Nash equilibrium in terms of Pareto superiority, especially in cooperative environments. Under Markov games, we formally define the bi-level reinforcement learning problem in finding Stackelberg equilibrium. We propose a novel bi-level actor-critic learning method that allows agents to have different knowledge base (thus intelligent), while their actions still can be executed simultaneously and distributedly. The convergence proof is given, while the resulting learning algorithm is tested against the state of the arts. We found that the proposed bi-level actor-critic algorithm successfully converged to the Stackelberg equilibria in matrix games and find an asymmetric solution in a highway merge environment.

cs.MA↗

arXiv2020

Effect of overlap on spreading dynamics on multiplex networks

In spite of the study of epidemic dynamics on single-layer networks has received considerable attention, the epidemic dynamics on multiplex networks is still limited and is facing many challenges. In this work, we consider the susceptible-infected-susceptible-type (SIS) epidemic model on multiplex networks and investigate the effect of overlap among layers on the spreading dynamics. To do so, we assume that the prerequisite of one $S$-node to be infected is that there is at least one infectious neighbor in each layer. A remarkable result is that the overlap can alter the nature of the phase transition for the onset of epidemic outbreak. Specifically speaking, the system undergoes a usual continuous phase transition when two layers are completely overlapped. Otherwise, a discontinuous phase transition is observed, accompanied by the occurrence of a bistable region in which a disease-free phase and an endemic phase are coexisting. As the degree of the overlap decreases, the bistable region is enlarged. The results are validated by both simulation and mean-field theory.

cond-mat.stat-mech↗

arXiv2019

A Multifuntional Frequency-selective Polarization Converter for broadband backward-scattering reduction

In this paper, we combine the design of band-pass frequency selective surfaces (FSSs) with polarization converters to realize a broadband frequency-selective polarization converter (FSPC) with lowbackward scattering, which consists of the top polarization conversion layer backed by a multi-layer bandpass FSS. It is numerically demonstrated that the 1 dB transmission window can be obtained from 8.5 GHz to 11 GHz with a 25.6% fractional bandwidth (FBW), and the bandwidth of reflection below -10 dB is up to 92% from 5.6 GHz to 15.13 GHz. Moreover, the proposed device can achieve two polarization conversion bands (5.66-6.9 GHz and 12.8-15.2GHz) with the polarization conversion ratio over 90%. Besides, by arranging the proposed structure in a checkerboard-like distribution, the backward scattering energy can be reduced in a wide frequency band ranging from 4 to 16 GHz. Both simulation and experimental results are in good agreements, which demonstrates our design strategy. Compared with the conventional polarization conversion designs, the proposed design presents an extra frequency-selective performance and hence can be applied to various practical situations, for instance, working as radomes to transmit the in-band signals with high-efficiency while keeping low-backward scattering for the out-of-band waves.

physics.app-ph↗

arXiv2019

Signal Instructed Coordination in Cooperative Multi-agent Reinforcement Learning

In many real-world problems, a team of agents need to collaborate to maximize the common reward. Although existing works formulate this problem into a centralized learning with decentralized execution framework, which avoids the non-stationary problem in training, their decentralized execution paradigm limits the agents' capability to coordinate. Inspired by the concept of correlated equilibrium, we propose to introduce a coordination signal to address this limitation, and theoretically show that following mild conditions, decentralized agents with the coordination signal can coordinate their individual policies as manipulated by a centralized controller. The idea of introducing coordination signal is to encapsulate coordinated strategies into the signals, and use the signals to instruct the collaboration in decentralized execution. To encourage agents to learn to exploit the coordination signal, we propose Signal Instructed Coordination (SIC), a novel coordination module that can be integrated with most existing MARL frameworks. SIC casts a common signal sampled from a pre-defined distribution to all agents, and introduces an information-theoretic regularization to facilitate the consistency between the observed signal and agents' policies. Our experiments show that SIC consistently improves performance over well-recognized MARL models in both matrix games and a predator-prey game with high-dimensional strategy space.

cs.MA↗

arXiv2019

Learning to Design Games: Strategic Environments in Reinforcement Learning

In typical reinforcement learning (RL), the environment is assumed given and the goal of the learning is to identify an optimal policy for the agent taking actions through its interactions with the environment. In this paper, we extend this setting by considering the environment is not given, but controllable and learnable through its interaction with the agent at the same time. This extension is motivated by environment design scenarios in the real-world, including game design, shopping space design and traffic signal design. Theoretically, we find a dual Markov decision process (MDP) w.r.t. the environment to that w.r.t. the agent, and derive a policy gradient solution to optimizing the parametrized environment. Furthermore, discontinuous environments are addressed by a proposed general generative framework. Our experiments on a Maze game design task show the effectiveness of the proposed algorithms in generating diverse and challenging Mazes against various agent settings.

cs.AI↗

arXiv2019

Measuring outcome correlation for spin-s Bell cat-state and geometric phase induced spin parity effect

In terms of quantum probability statistics the Bell inequality (BI) and its violation are extended to spin-$s$ entangled Schrödinger cat-state (called the Bell cat-state) with both parallel and antiparallel spin-polarizations. The BI is never ever violated for the measuring outcome probabilities evaluated over entire two-spin Hilbert space except the spin-$1/2$ entangled states. A universal Bell-type inequality (UBI) denoted by $p_{s}^{lc}\leq0$ is formulated with the local realistic model under the condition that the measuring outcomes are restricted in the subspace of spin coherent states. A spin parity effect is observed that the UBI can be violated only by the Bell cat-states of half-integer but not the integer spins. The violation of UBI is seen to be a direct result of non-trivial Berry phase between the spin coherent states of south- and north-pole gauges for half-integer spin, while the geometric phase is trivial for the integer spins. A maximum violation bound of UBI is found as $p_{s}^{\max}$=1, which is valid for arbitrary half-integer spin-$s$ states.

quant-ph↗

arXiv2019

Spontaneous symmetry breaking and discontinuous phase transition for spreading dynamics in multiplex networks

We propose a spreading model in multilayer networks and study the nature of nonequilibrium phase transition in the model. The model integrates the susceptible-infected-susceptible (or susceptible-infected-recovered) spreading dynamics with a biased diffusion process among different layers. A parameter $α$ is introduced to control the bias of the diffusion process, such that each individual prefers to move to one layer with more infected (or recovered) neighbors for larger values of $α$. Using stochastic simulations and mean-field theory, we show that the type of phase transition from a disease-free phase to an endemic phase depends on the value of $α$. When $α$ is small enough, the system undergoes a usual continuous phase transition as an effective spreading rate $β$ increases, as in single-layer networks. Interestingly, when $α$ exceeds a critical value the system shows either a hybrid two-step phase transition or a one-step discontinuous phase transition as $β$ increases. The former contains a continuous transition between the disease-free phase and a low-prevalence endemic phase, and a discontinuous transition between the low-prevalence endemic phase and a high-prevalence endemic phase. For the latter, only a discontinuous transition occurs from the disease-free phase directly to the high-prevalence endemic phase. Moreover, we show that the discontinuous transition is always accompanied by a spontaneous symmetry breaking in occupation probabilities of individuals in each layer.

physics.soc-ph↗

arXiv2018

Layout Design for Intelligent Warehouse by Evolution with Fitness Approximation

With the rapid growth of the express industry, intelligent warehouses that employ autonomous robots for carrying parcels have been widely used to handle the vast express volume. For such warehouses, the warehouse layout design plays a key role in improving the transportation efficiency. However, this work is still done by human experts, which is expensive and leads to suboptimal results. In this paper, we aim to automate the warehouse layout designing process. We propose a two-layer evolutionary algorithm to efficiently explore the warehouse layout space, where an auxiliary objective fitness approximation model is introduced to predict the outcome of the designed warehouse layout and a two-layer population structure is proposed to incorporate the approximation model into the ordinary evolution framework. Empirical experiments show that our method can efficiently design effective warehouse layouts that outperform both heuristic-designed and vanilla evolution-designed warehouse layouts.

cs.AI↗

arXiv2018

Extended Bell inequality and maximum violation

The original formula of Bell inequality (BI) in terms of two-spin singlet has to be modified for the entangled-state with parallel spin polarization. Based on classical statistics of the particle-number correlation, we prove in this paper an extended BI, which is valid for two-spin entangled states with both parallel and antiparallel polarizations. The BI and its violation can be formulated in a unified formalism based on the spin coherent-state quantum probability statistics with the state-density operator, which is separated to the local and non-local parts. The local part gives rise to the BI, while the violation is a direct result of the non-local quantum interference between two components of entangled state. The Bell measuring outcome correlation denoted by $P_{B}$ is always less than or at most equal to one for the local realistic model ($P_{B}^{lc}\leq1$) regardless of the specific superposition coefficients of entangled state. Including the non-local quantum interference the maximum violation of BI is found as $P_{B}^{\max}$ $=2$, which, however depends on state parameters and three measuring directions as well. Our result is suitable for entangled photon pairs.

quant-ph↗

arXiv2018

Maximum violation of Wigner inequality for two-spin entangled states with parallel and antiparallel polarizations

The experimental test of Bell's inequality is mainly focused on Clauser-Horne-Shimony-Holt (CHSH) form, which provides a quantitative bound, while little attention has been pained on the violation of Wigner inequality (WI). Based on the spin coherent state quantum probability statistics we in the present paper extend the WI and its violation to arbitrary two-spin entangled states with antiparallel and parallel spin-polarizations. The local part of density operator gives rise to the WI while the violation is a direct result of non-local interference between two components of the entangled states. The Wigner measuring outcome correlation denoted by $W$ is always less than or at most equal to zero for the local realist model ($% W_{lc_{}}\leq 0$) regardless of the specific initial state. On the other hand the violation of\ WI is characterized by any positive value of $W$, which possesses a maximum violation bound $W_{\max }$ $=1/2$. We conclude that the WI is equally convenient for the experimental test of violation by the quantum entanglement.

quant-ph↗

arXiv2018

Double phase transition of the Ising model in core-periphery networks

We study the phase transition of the Ising model in networks with core-periphery structures. By Monte Carlo simulations, we show that prior to the order-disorder phase transition the system organizes into an inhomogeneous intermediate phase in which core nodes are much more ordered than peripheral nodes. Interestingly, the susceptibility shows double peaks at two distinct temperatures. We find that, if the connections between core and periphery increase linearly with network size, the first peak does not exhibit any size-dependent effect, and the second one diverges in the limit of infinite network size. Otherwise, if the connections between core and periphery scale sub-linearly with the network size, both peaks of the susceptibility diverge as power laws in the thermodynamic limit. This suggests the appearance of a double transition phenomenon in the Ising model for the latter case. Moreover, we develop a mean-field theory that agrees well with the simulations.

cond-mat.stat-mech↗