arXiv subjects

Yang Yu

Publications and source records attributed to Yang Yu.

At least 361 records · Page 20Linked to original sources

arXiv2021

Calculus of Consent via MARL: Legitimating the Collaborative Governance Supplying Public Goods

Public policies that supply public goods, especially those involve collaboration by limiting individual liberty, always give rise to controversies over governance legitimacy. Multi-Agent Reinforcement Learning (MARL) methods are appropriate for supporting the legitimacy of the public policies that supply public goods at the cost of individual interests. Among these policies, the inter-regional collaborative pandemic control is a prominent example, which has become much more important for an increasingly inter-connected world facing a global pandemic like COVID-19. Different patterns of collaborative strategies have been observed among different systems of regions, yet it lacks an analytical process to reason for the legitimacy of those strategies. In this paper, we use the inter-regional collaboration for pandemic control as an example to demonstrate the necessity of MARL in reasoning, and thereby legitimizing policies enforcing such inter-regional collaboration. Experimental results in an exemplary environment show that our MARL approach is able to demonstrate the effectiveness and necessity of restrictions on individual liberty for collaborative supply of public goods. Different optimal policies are learned by our MARL agents under different collaboration levels, which change in an interpretable pattern of collaboration that helps to balance the losses suffered by regions of different types, and consequently promotes the overall welfare. Meanwhile, policies learned with higher collaboration levels yield higher global rewards, which illustrates the benefit of, and thus provides a novel justification for the legitimacy of, promoting inter-regional collaboration. Therefore, our method shows the capability of MARL in computationally modeling and supporting the theory of calculus of consent, developed by Nobel Prize winner J. M. Buchanan.

cs.AI↗

arXiv2021

An Introduction of mini-AlphaStar

StarCraft II (SC2) is a real-time strategy game in which players produce and control multiple units to fight against opponent's units. Due to its difficulties, such as huge state space, various action space, a long time horizon, and imperfect information, SC2 has been a research hotspot in reinforcement learning. Recently, an agent called AlphaStar (AS) has been proposed, which shows good performance, obtaining a high win rate of 99.8% against human players. We implemented a mini-scaled version of it called mini-AlphaStar (mAS) based on AS's paper and pseudocode. The difference between AS and mAS is that we substituted the hyper-parameters of AS with smaller ones for mini-scale training. Codes of mAS are all open-sourced (https://github.com/liuruoze/mini-AlphaStar) for future research.

cs.AI↗

arXiv2021

Regret Minimization Experience Replay in Off-Policy Reinforcement Learning

In reinforcement learning, experience replay stores past samples for further reuse. Prioritized sampling is a promising technique to better utilize these samples. Previous criteria of prioritization include TD error, recentness and corrective feedback, which are mostly heuristically designed. In this work, we start from the regret minimization objective, and obtain an optimal prioritization strategy for Bellman update that can directly maximize the return of the policy. The theory suggests that data with higher hindsight TD error, better on-policiness and more accurate Q value should be assigned with higher weights during sampling. Thus most previous criteria only consider this strategy partially. We not only provide theoretical justifications for previous criteria, but also propose two new methods to compute the prioritization weight, namely ReMERN and ReMERT. ReMERN learns an error network, while ReMERT exploits the temporal ordering of states. Both methods outperform previous prioritized sampling algorithms in challenging RL benchmarks, including MuJoCo, Atari and Meta-World.

cs.LG↗

arXiv2021

Efficient Reinforcement Learning for StarCraft by Abstract Forward Models and Transfer Learning

Injecting human knowledge is an effective way to accelerate reinforcement learning (RL). However, these methods are underexplored. This paper presents our discovery that an abstract forward model (thought-game (TG)) combined with transfer learning (TL) is an effective way. We take StarCraft II as our study environment. With the help of a designed TG, the agent can learn a 99% win-rate on a 64x64 map against the Level-7 built-in AI, using only 1.08 hours in a single commercial machine. We also show that the TG method is not as restrictive as it was thought to be. It can work with roughly designed TGs, and can also be useful when the environment changes. Comparing with previous model-based RL, we show TG is more effective. We also present a TG hypothesis that gives the influence of different fidelity levels of TG. For real games that have unequal state and action spaces, we proposed a novel XfrNet of which usefulness is validated while achieving a 90% win-rate against the cheating Level-10 AI. We argue that the TG method might shed light on further studies of efficient RL with human knowledge.

cs.LG↗

arXiv2021

A general framework for scintillation in nanophotonics

Bombardment of materials by high-energy particles (e.g., electrons, nuclei, X- and $γ$-ray photons) often leads to light emission, known generally as scintillation. Scintillation is ubiquitous and enjoys widespread applications in many areas such as medical imaging, X-ray non-destructive inspection, night vision, electron microscopy, and high-energy particle detectors. A large body of research focuses on finding new materials optimized for brighter, faster, and more controlled scintillation. Here, we develop a fundamentally different approach based on integrating nanophotonic structures into scintillators to enhance their emission. To start, we develop a unified and ab initio theory of nanophotonic scintillators that accounts for the key aspects of scintillation: the energy loss by high-energy particles, as well as the light emission by non-equilibrium electrons in arbitrary nanostructured optical systems. This theoretical framework allows us, for the first time, to experimentally demonstrate nearly an order-of-magnitude enhancement of scintillation, in both electron-induced, and X-ray-induced scintillation. Our theory also allows the discovery of structures that could eventually achieve several orders-of-magnitude scintillation enhancement. The framework and results shown here should enable the development of a new class of brighter, faster, and higher-resolution scintillators with tailored and optimized performances - with many potential applications where scintillators are used.

physics.optics↗

arXiv2021

QPLEX: Duplex Dueling Multi-Agent Q-Learning

We explore value-based multi-agent reinforcement learning (MARL) in the popular paradigm of centralized training with decentralized execution (CTDE). CTDE has an important concept, Individual-Global-Max (IGM) principle, which requires the consistency between joint and local action selections to support efficient local decision-making. However, in order to achieve scalability, existing MARL methods either limit representation expressiveness of their value function classes or relax the IGM consistency, which may suffer from instability risk or may not perform well in complex domains. This paper presents a novel MARL approach, called duPLEX dueling multi-agent Q-learning (QPLEX), which takes a duplex dueling network architecture to factorize the joint value function. This duplex dueling structure encodes the IGM principle into the neural network architecture and thus enables efficient value function learning. Theoretical analysis shows that QPLEX achieves a complete IGM function class. Empirical experiments on StarCraft II micromanagement tasks demonstrate that QPLEX significantly outperforms state-of-the-art baselines in both online and offline data collection settings, and also reveal that QPLEX achieves high sample efficiency and can benefit from offline datasets without additional online exploration.

cs.LG↗

arXiv2021

Labor-right Protecting Dispatch of Meal Delivery Platforms

The boom in the meal delivery industry brings growing concern about the labor rights of riders. Current dispatch policies of meal-delivery platforms focus mainly on satisfying consumers or minimizing the number of riders for cost savings. There are few discussions on improving the working conditions of riders by algorithm design. The lack of concerns on labor rights in mechanism and dispatch design has resulted in a very large time waste for riders and their risky driving. In this research, we propose a queuing-model-based framework to discuss optimal dispatch policy with the goal of labor rights protection. We apply our framework to develop an algorithm minimizing the waiting time of food delivery riders with guaranteed user experience. Our framework also allows us to manifest the value of restaurants' data about their offline-order numbers on improving the benefits of riders.

cs.PF↗

arXiv2021

UserBERT: Contrastive User Model Pre-training

User modeling is critical for personalized web applications. Existing user modeling methods usually train user models from user behaviors with task-specific labeled data. However, labeled data in a target task may be insufficient for training accurate user models. Fortunately, there are usually rich unlabeled user behavior data which encode rich information of user characteristics and interests. Thus, pre-training user models on unlabeled user behavior data has the potential to improve user modeling for many downstream tasks. In this paper, we propose a contrastive user model pre-training method named UserBERT. Two self-supervision tasks are incorporated in UserBERT for user model pre-training on unlabeled user behavior data to empower user modeling. The first one is masked behavior prediction, which aims to model the relatedness between user behaviors. The second one is behavior sequence matching, which aims to capture the inherent user interests that are consistent in different periods. In addition, we propose a medium-hard negative sampling framework to select informative negative samples for better contrastive pre-training. We maintain a synchronously updated candidate behavior pool and an asynchronously updated candidate behavior sequence pool to select the locally hardest negative behaviors and behavior sequences in an efficient way. Extensive experiments on two real-world datasets in different tasks show that UserBERT can effectively improve various user models.

cs.IR↗

arXiv2021

NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application

Pre-trained language models (PLMs) like BERT have made great progress in NLP. News articles usually contain rich textual information, and PLMs have the potentials to enhance news text modeling for various intelligent news applications like news recommendation and retrieval. However, most existing PLMs are in huge size with hundreds of millions of parameters. Many online news applications need to serve millions of users with low latency tolerance, which poses huge challenges to incorporating PLMs in these scenarios. Knowledge distillation techniques can compress a large PLM into a much smaller one and meanwhile keeps good performance. However, existing language models are pre-trained and distilled on general corpus like Wikipedia, which has some gaps with the news domain and may be suboptimal for news intelligence. In this paper, we propose NewsBERT, which can distill PLMs for efficient and effective news intelligence. In our approach, we design a teacher-student joint learning and distillation framework to collaboratively learn both teacher and student models, where the student model can learn from the learning experience of the teacher model. In addition, we propose a momentum distillation method by incorporating the gradients of teacher model into the update of student model to better transfer useful knowledge learned by the teacher model. Extensive experiments on two real-world datasets with three tasks show that NewsBERT can effectively improve the model performance in various intelligent news applications with much smaller models.

cs.CL↗

arXiv2021

The Flare and Warp of the Young Stellar Disk traced with LAMOST DR5 OB-type stars

We present analysis of the spatial density structure for the outer disk from 8$-$14 \,kpc with the LAMOST DR5 13534 OB-type stars and observe similar flaring on north and south sides of the disk implying that the flaring structure is symmetrical about the Galactic plane, for which the scale height at different Galactocentric distance is from 0.14 to 0.5 \,kpc. By using the average slope to characterize the flaring strength we find that the thickness of the OB stellar disk is similar but flaring is slightly stronger compared to the thin disk as traced by red giant branch stars, possibly implying that secular evolution is not the main contributor to the flaring but perturbation scenarios such as interactions with passing dwarf galaxies should be more possible. When comparing the scale height of OB stellar disk of the north and south sides with the gas disk, the former one is slightly thicker than the later one by $\approx$ 33 and 9 \,pc, meaning that one could tentatively use young OB-type stars to trace the gas properties. Meanwhile, we unravel that the radial scale length of the young OB stellar disk is 1.17 $\pm$ 0.05 \,kpc, which is shorter than that of the gas disk, confirming that the gas disk is more extended than stellar disk. What is more, by considering the mid-plane displacements ($Z_{0}$) in our density model we find that almost all of $Z_{0}$ are within 100 \,pc with the increasing trend as Galactocentric distance increases.

astro-ph.GA↗

arXiv2021

Neural-to-Tree Policy Distillation with Policy Improvement Criterion

While deep reinforcement learning has achieved promising results in challenging decision-making tasks, the main bones of its success --- deep neural networks are mostly black-boxes. A feasible way to gain insight into a black-box model is to distill it into an interpretable model such as a decision tree, which consists of if-then rules and is easy to grasp and be verified. However, the traditional model distillation is usually a supervised learning task under a stationary data distribution assumption, which is violated in reinforcement learning. Therefore, a typical policy distillation that clones model behaviors with even a small error could bring a data distribution shift, resulting in an unsatisfied distilled policy model with low fidelity or low performance. In this paper, we propose to address this issue by changing the distillation objective from behavior cloning to maximizing an advantage evaluation. The novel distillation objective maximizes an approximated cumulative reward and focuses more on disastrous behaviors in critical states, which controls the data shift effect. We evaluate our method on several Gym tasks, a commercial fight game, and a self-driving car simulator. The empirical results show that the proposed method can preserve a higher cumulative reward than behavior cloning and learn a more consistent policy to the original one. Moreover, by examining the extracted rules from the distilled decision trees, we demonstrate that the proposed method delivers reasonable and robust decisions.

cs.LG↗

arXiv2021

Imitate TheWorld: A Search Engine Simulation Platform

Recent E-commerce applications benefit from the growth of deep learning techniques. However, we notice that many works attempt to maximize business objectives by closely matching offline labels which follow the supervised learning paradigm. This results in models obtain high offline performance in terms of Area Under Curve (AUC) and Normalized Discounted Cumulative Gain (NDCG), but cannot consistently increase the revenue metrics such as purchases amount of users. Towards the issues, we build a simulated search engine AESim that can properly give feedback by a well-trained discriminator for generated pages, as a dynamic dataset. Different from previous simulation platforms which lose connection with the real world, ours depends on the real data in AliExpress Search: we use adversarial learning to generate virtual users and use Generative Adversarial Imitation Learning (GAIL) to capture behavior patterns of users. Our experiments also show AESim can better reflect the online performance of ranking models than classic ranking metrics, implying AESim can play a surrogate of AliExpress Search and evaluate models without going online.

cs.AI↗

arXiv2021

HieRec: Hierarchical User Interest Modeling for Personalized News Recommendation

User interest modeling is critical for personalized news recommendation. Existing news recommendation methods usually learn a single user embedding for each user from their previous behaviors to represent their overall interest. However, user interest is usually diverse and multi-grained, which is difficult to be accurately modeled by a single user embedding. In this paper, we propose a news recommendation method with hierarchical user interest modeling, named HieRec. Instead of a single user embedding, in our method each user is represented in a hierarchical interest tree to better capture their diverse and multi-grained interest in news. We use a three-level hierarchy to represent 1) overall user interest; 2) user interest in coarse-grained topics like sports; and 3) user interest in fine-grained topics like football. Moreover, we propose a hierarchical user interest matching framework to match candidate news with different levels of user interest for more accurate user interest targeting. Extensive experiments on two real-world datasets validate our method can effectively improve the performance of user modeling for personalized news recommendation.

cs.IR↗

arXiv2021

Sand Creep Motion in Slow Spin-up Experiment: An Analogue of Regolith Migration on Asteroids

We studied the creep motion of granular materials in a gradient potential field that is created using a slow spin-up experiment device. Natural sand confined in the acrylic box is spun up by a controlled turntable and the surface flows are captured using video-based measurements. Various spin-up accelerations were considered to understand the responses of creep motion on different accelerating paths. Convergent behaviors in the morphological change of sand surface were observed in the final steady state. To quantify the quasi-static spin-up process, we examined the net flux and the surface slope as a function of the spin rate and offset from the rotation axis. Evolution of sand creep motion demonstrated behaviors similar to regolith migration in numeric simulations, showing intermittency like general sheared granular systems. We noticed the sand surface approaches criticality as the spin-up proceeding, consistent with the observation that top-shaped asteroids near limiting spin rate take on critical shape. Comparisons to large-scale numeric simulations and analytical solutions reveal underlying similarities between our experiments and the million-year evolution of asteroid regolith under YORP acceleration, which raises the possibility of studying asteroid surface processes in laboratory analogue experiments.

astro-ph.EP↗

arXiv2021

Learning Action-Transferable Policy with Action Embedding

Transfer learning (TL) is a promising way to improve the sample efficiency of reinforcement learning. However, how to efficiently transfer knowledge across tasks with different state-action spaces is investigated at an early stage. Most previous studies only addressed the inconsistency across different state spaces by learning a common feature space, without considering that similar actions in different action spaces of related tasks share similar semantics. In this paper, we propose a method to learning action embeddings by leveraging this idea, and a framework that learns both state embeddings and action embeddings to transfer policy across tasks with different state and action spaces. Our experimental results on various tasks show that the proposed method can not only learn informative action embeddings but accelerate policy learning.

cs.LG↗

arXiv2021

Exact Higher-order Bulk-boundary Correspondence of Corner-localized States

We demonstrate that the presence of a localized state at the corner of an insulating domain is not always a predictor of a certain non-trivial higher-order topological invariant, even though they appear to co-exist in the same Hamiltonian parameter space. Our analysis of $C^n$-symmetric crystalline insulators and their multi-layer stacks reveals that topological corner states are not necessarily correlated with other well-established higher-order boundary observables, such as fractional corner charge or filling anomaly. In a $C^3$-symmetric breathing Kagome lattice, for example, we show that the bulk polarization, which successfully predicts the fractional corner anomaly, fails to be the relevant topological invariant for zero-energy corner states; instead, these corner states can be exactly explained by the decoration of topological edges. Also, while the zero-energy corner states in $C^4$-symmetric topological crystalline insulators have long been conjectured to be the result of the bulk polarization at quarter-filling, we correct this misconception by introducing a proper bulk invariant at half-filling and establishing a precise bulk-corner correspondence. By refining several bulk-corner correspondences in two-dimensional topological crystalline insulators, our work motivates further development of rigorous theoretical grounds for associating the existence of corner states with higher-order topology of host materials.

cond-mat.mes-hall↗

arXiv2021

ASBSO: An Improved Brain Storm Optimization With Flexible Search Length and Memory-Based Selection

Brain storm optimization (BSO) is a newly proposed population-based optimization algorithm, which uses a logarithmic sigmoid transfer function to adjust its search range during the convergent process. However, this adjustment only varies with the current iteration number and lacks of flexibility and variety which makes a poor search effciency and robustness of BSO. To alleviate this problem, an adaptive step length structure together with a success memory selection strategy is proposed to be incorporated into BSO. This proposed method, adaptive step length based on memory selection BSO, namely ASBSO, applies multiple step lengths to modify the generation process of new solutions, thus supplying a flexible search according to corresponding problems and convergent periods. The novel memory mechanism, which is capable of evaluating and storing the degree of improvements of solutions, is used to determine the selection possibility of step lengths. A set of 57 benchmark functions are used to test ASBSO's search ability, and four real-world problems are adopted to show its application value. All these test results indicate the remarkable improvement in solution quality, scalability, and robustness of ASBSO.

cs.NE↗

arXiv2021

TurboTransformers: An Efficient GPU Serving System For Transformer Models

The transformer is the most critical algorithm innovation of the Nature Language Processing (NLP) field in recent years. Unlike the Recurrent Neural Network (RNN) models, Transformers can process on dimensions of sequence lengths in parallel, therefore leading to better accuracy on long sequences. However, efficient deployments of them for online services in data centers equipped with GPUs are not easy. First, more computation introduced by transformer structures makes it more challenging to meet the latency and throughput constraints of serving. Second, NLP tasks take in sentences of variable length. The variability of input dimensions brings a severe problem to efficient memory management and serving optimization. This paper designed a transformer serving system called TurboTransformers, which consists of a computing runtime and a serving framework to solve the above challenges. Three innovative features make it stand out from other similar works. An efficient parallel algorithm is proposed for GPU-based batch reduction operations, like Softmax and LayerNorm, major hot spots besides BLAS routines. A memory allocation algorithm, which better balances the memory footprint and allocation/free efficiency, is designed for variable-length input situations. A serving framework equipped with a new batch scheduler using dynamic programming achieves the optimal throughput on variable-length requests. The system can achieve the state-of-the-art transformer model serving performance on GPU platforms and can be seamlessly integrated into your PyTorch code with a few lines of code.

cs.DC↗