arXiv subjects

Zihan Zhang

Publications and source records attributed to Zihan Zhang.

At least 199 records · Page 11Linked to original sources

arXiv2022

PromptBERT: Improving BERT Sentence Embeddings with Prompts

We propose PromptBERT, a novel contrastive learning method for learning better sentence representation. We firstly analyze the drawback of current sentence embedding from original BERT and find that it is mainly due to the static token embedding bias and ineffective BERT layers. Then we propose the first prompt-based sentence embeddings method and discuss two prompt representing methods and three prompt searching methods to make BERT achieve better sentence embeddings. Moreover, we propose a novel unsupervised training objective by the technology of template denoising, which substantially shortens the performance gap between the supervised and unsupervised settings. Extensive experiments show the effectiveness of our method. Compared to SimCSE, PromptBert achieves 2.29 and 2.58 points of improvement based on BERT and RoBERTa in the unsupervised setting.

cs.CL↗

arXiv2022

Predicting Blossom Date of Cherry Tree With Support Vector Machine and Recurrent Neural Network

Our project probes the relationship between temperatures and the blossom date of cherry trees. Through modeling, future flowering will become predictive, helping the public plan travels and avoid pollen season. To predict the date when the cherry trees will blossom exactly could be viewed as a multiclass classification problem, so we applied the multi-class Support Vector Classifier (SVC) and Recurrent Neural Network (RNN), particularly Long Short-term Memory (LSTM), to formulate the problem. In the end, we evaluate and compare the performance of these approaches to find out which one might be more applicable in reality.

cs.LG↗

arXiv2022

Hybrid Supervised and Reinforcement Learning for the Design and Optimization of Nanophotonic Structures

From higher computational efficiency to enabling the discovery of novel and complex structures, deep learning has emerged as a powerful framework for the design and optimization of nanophotonic circuits and components. However, both data-driven and exploration-based machine learning strategies have limitations in their effectiveness for nanophotonic inverse design. Supervised machine learning approaches require large quantities of training data to produce high-performance models and have difficulty generalizing beyond training data given the complexity of the design space. Unsupervised and reinforcement learning-based approaches on the other hand can have very lengthy training or optimization times associated with them. Here we demonstrate a hybrid supervised learning and reinforcement learning approach to the inverse design of nanophotonic structures and show this approach can reduce training data dependence, improve the generalizability of model predictions, and shorten exploratory training times by orders of magnitude. The presented strategy thus addresses a number of contemporary deep learning-based challenges, while opening the door for new design methodologies that leverage multiple classes of machine learning algorithms to produce more effective and practical solutions for photonic design.

cs.LG↗

arXiv2022

The HCCL System for the NIST SRE21

This paper describes the systems developed by the HCCL team for the NIST 2021 speaker recognition evaluation (NIST SRE21).We first explore various state-of-the-art speaker embedding extractors combined with a novel circle loss to obtain discriminative deep speaker embeddings. Considering that cross-channel and cross-linguistic speaker recognition are the key challenges of SRE21, we introduce several techniques to reduce the cross-domain mismatch. Specifically, Codec and speech enhancement are directly applied to the raw speech to eliminate the codecs and the environment noise mismatch. We denote the methods that work directly on speech to eliminate the relatively explicit mismatches collectively as data adaptation methods. Experiments show that data adaption methods achieve 15\% improvements over our baseline. Furthermore, some popular back-ends domain adaptation algorithms are deployed on speaker embeddings to alleviate speaker performance degradation caused by the implicit mismatch. Score calibration is a major failure for us in SRE21. The reason is that score calibration with too many parameters easily lead to overfitting problems.

cs.SD↗

arXiv2022

Horizon-Free Reinforcement Learning in Polynomial Time: the Power of Stationary Policies

This paper gives the first polynomial-time algorithm for tabular Markov Decision Processes (MDP) that enjoys a regret bound \emph{independent on the planning horizon}. Specifically, we consider tabular MDP with $S$ states, $A$ actions, a planning horizon $H$, total reward bounded by $1$, and the agent plays for $K$ episodes. We design an algorithm that achieves an $O\left(\mathrm{poly}(S,A,\log K)\sqrt{K}\right)$ regret in contrast to existing bounds which either has an additional $\mathrm{polylog}(H)$ dependency~\citep{zhang2020reinforcement} or has an exponential dependency on $S$~\citep{li2021settling}. Our result relies on a sequence of new structural lemmas establishing the approximation power, stability, and concentration property of stationary policies, which can have applications in other problems related to Markov chains.

cs.LG↗

arXiv2022

A Pilot Study of Relating MYCN-Gene Amplification with Neuroblastoma-Patient CT Scans

Neuroblastoma is one of the most common cancers in infants, and the initial diagnosis of this disease is difficult. At present, the MYCN gene amplification (MNA) status is detected by invasive pathological examination of tumor samples. This is time-consuming and may have a hidden impact on children. To handle this problem, we adopt multiple machine learning (ML) algorithms to predict the presence or absence of MYCN gene amplification. The dataset is composed of retrospective CT images of 23 neuroblastoma patients. Different from previous work, we develop the algorithm without manually-segmented primary tumors which is time-consuming and not practical. Instead, we only need the coordinate of the center point and the number of tumor slices given by a subspecialty-trained pediatric radiologist. Specifically, CNN-based method uses pre-trained convolutional neural network, and radiomics-based method extracts radiomics features. Our results show that CNN-based method outperforms the radiomics-based method.

eess.IV↗

arXiv2022

A New Metric on Symmetric Group and Applications to Block Permutation Codes

Permutation codes have received a great attention due to various applications. For different applications, one needs permutation codes under different metrics. The generalized Cayley metric was introduced by Chee and Vu [4] and this metric includes several other metrics as special cases. However, the generalized Cayley metric is not easily computable in general. Therefore the block permutation metric was introduced by Yang et al. [22] as the generalized Cayley metric and the block permutation metric have the same magnitude. However, the block permutation metric lacks the symmetry property which restricts more advanced algebraic tools to be involved. In this paper, by introducing a novel metric closely related to the block permutation metric, we build a bridge between some advanced algebraic methods and codes in the block permutation metric. More specifically, based on some techniques from algebraic function fields originated in [19], we give an algebraic-geometric construction of codes in the novel metric with reasonably good parameters. By observing a trivial relation between the novel metric and block permutation metric, we then produce non-systematic codes in block permutation metric that improve all known results given in [21, 22]. More importantly, based on our non-systematic codes, we provide an explicit and systematic construction of codes in block permutation metric which improves the systematic result shown in [22]. In the end, we demonstrate that our codes in the novel metric itself have reasonably good parameters by showing that our construction beats the corresponding Gilbert-Varshamov bound.

cs.IT↗

arXiv2022

GANimator: Neural Motion Synthesis from a Single Sequence

We present GANimator, a generative model that learns to synthesize novel motions from a single, short motion sequence. GANimator generates motions that resemble the core elements of the original motion, while simultaneously synthesizing novel and diverse movements. Existing data-driven techniques for motion synthesis require a large motion dataset which contains the desired and specific skeletal structure. By contrast, GANimator only requires training on a single motion sequence, enabling novel motion synthesis for a variety of skeletal structures e.g., bipeds, quadropeds, hexapeds, and more. Our framework contains a series of generative and adversarial neural networks, each responsible for generating motions in a specific frame rate. The framework progressively learns to synthesize motion from random noise, enabling hierarchical control over the generated motion content across varying levels of detail. We show a number of applications, including crowd simulation, key-frame editing, style transfer, and interactive control, which all learn from a single input sequence. Code and data for this paper are at https://peizhuoli.github.io/ganimator.

cs.GR↗

arXiv2022

Back-ends Selection for Deep Speaker Embeddings

Probabilistic Linear Discriminant Analysis (PLDA) was the dominant and necessary back-end for early speaker recognition approaches, like i-vector and x-vector. However, with the development of neural networks and margin-based loss functions, we can obtain deep speaker embeddings (DSEs), which have advantages of increased inter-class separation and smaller intra-class distances. In this case, PLDA seems unnecessary or even counterproductive for the discriminative embeddings, and cosine similarity scoring (Cos) achieves better performance than PLDA in some situations. Motivated by this, in this paper, we systematically explore how to select back-ends (Cos or PLDA) for deep speaker embeddings to achieve better performance in different situations. By analyzing PLDA and the properties of DSEs extracted from models with different numbers of segment-level layers, we make the conjecture that Cos is better in same-domain situations and PLDA is better in cross-domain situations. We conduct experiments on VoxCeleb and NIST SRE datasets in four application situations, single-/multi-domain training and same-/cross-domain test, to validate our conjecture and briefly explain why back-ends adaption algorithms work.

cs.SD↗

arXiv2022

Room-temperature superconductivity in heavy rare earth metal substituted sodalite-like clathrate hexahydrides under moderate pressure

Room temperature superconductivity is a dream that mankind has been chasing for a century. In recent years, the synthesis of H3S, LaH10 and C-S-H system has gradually made this dream a reality. But the extreme pressures required for the metallization of hydrogen-based superconductors limit their applications. In this work, we design a series of high temperature superconductors that can be stable at moderate pressures by incorporating heavy rare earth elements Yb/Lu into sodalite-like clathrate hydrides. In particular, the critical temperatures of Y3LuH24, YLuH12 and YLu3H24 are 283 K, 275 K and 288 K, respectively, which are close to or have reached room temperature, and the required pressure for stabilization of these hydrides is about 120 GPa which is significantly lower than that of reported room temperature superconductors. Our work provides an effective method for the rational design of low-pressure stabilized hydrogen-based superconductors with high-Tc and will stimulate further experimental exploration.

cond-mat.supr-con↗

arXiv2022

Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topics

Recent work incorporates pre-trained word embeddings such as BERT embeddings into Neural Topic Models (NTMs), generating highly coherent topics. However, with high-quality contextualized document representations, do we really need sophisticated neural models to obtain coherent and interpretable topics? In this paper, we conduct thorough experiments showing that directly clustering high-quality sentence embeddings with an appropriate word selecting method can generate more coherent and diverse topics than NTMs, achieving also higher efficiency and simplicity.

cs.CL↗

arXiv2022

Long-Tailed Classification with Gradual Balanced Loss and Adaptive Feature Generation

The real-world data distribution is essentially long-tailed, which poses great challenge to the deep model. In this work, we propose a new method, Gradual Balanced Loss and Adaptive Feature Generator (GLAG) to alleviate imbalance. GLAG first learns a balanced and robust feature model with Gradual Balanced Loss, then fixes the feature model and augments the under-represented tail classes on the feature level with the knowledge from well-represented head classes. And the generated samples are mixed up with real training samples during training epochs. Gradual Balanced Loss is a general loss and it can combine with different decoupled training methods to improve the original performance. State-of-the-art results have been achieved on long-tail datasets such as CIFAR100-LT, ImageNetLT, and iNaturalist, which demonstrates the effectiveness of GLAG for long-tailed visual recognition.

cs.CV↗

arXiv2021

Improved Variance-Aware Confidence Sets for Linear Bandits and Linear Mixture MDP

This paper presents new \emph{variance-aware} confidence sets for linear bandits and linear mixture Markov Decision Processes (MDPs). With the new confidence sets, we obtain the follow regret bounds: For linear bandits, we obtain an $\tilde{O}(poly(d)\sqrt{1 + \sum_{k=1}^{K}σ_k^2})$ data-dependent regret bound, where $d$ is the feature dimension, $K$ is the number of rounds, and $σ_k^2$ is the \emph{unknown} variance of the reward at the $k$-th round. This is the first regret bound that only scales with the variance and the dimension but \emph{no explicit polynomial dependency on $K$}. When variances are small, this bound can be significantly smaller than the $\tildeΘ\left(d\sqrt{K}\right)$ worst-case regret bound. For linear mixture MDPs, we obtain an $\tilde{O}(poly(d, \log H)\sqrt{K})$ regret bound, where $d$ is the number of base models, $K$ is the number of episodes, and $H$ is the planning horizon. This is the first regret bound that only scales \emph{logarithmically} with $H$ in the reinforcement learning with linear function approximation setting, thus \emph{exponentially improving} existing results, and resolving an open problem in \citep{zhou2020nearly}. We develop three technical ideas that may be of independent interest: 1) applications of the peeling technique to both the input norm and the variance magnitude, 2) a recursion-based estimator for the variance, and 3) a new convex potential lemma that generalizes the seminal elliptical potential lemma.

cs.LG↗

arXiv2021

Improving Non-autoregressive Generation with Mixup Training

While pre-trained language models have achieved great success on various natural language understanding tasks, how to effectively leverage them into non-autoregressive generation tasks remains a challenge. To solve this problem, we present a non-autoregressive generation model based on pre-trained transformer models. To bridge the gap between autoregressive and non-autoregressive models, we propose a simple and effective iterative training method called MIx Source and pseudo Target (MIST). Unlike other iterative decoding methods, which sacrifice the inference speed to achieve better performance based on multiple decoding iterations, MIST works in the training stage and has no effect on inference time. Our experiments on three generation benchmarks including question generation, summarization and paraphrase generation, show that the proposed framework achieves the new state-of-the-art results for fully non-autoregressive models. We also demonstrate that our method can be used to a variety of pre-trained models. For instance, MIST based on the small pre-trained model also obtains comparable performance with seq2seq models.

cs.CL↗

arXiv2021

Superconductivity of CH4-intercalated H3S under high pressure

The discovery of the high temperature superconducting state in compounds of hydrogen, carbon and sulfur with the critical temperature (Tc) of 288 K at high pressure is an important milestone towards room-temperature superconductors. Here, we have extensively investigated the high-pressure phases of CS2H10, and found four phases Cmc21, P3m1, P-3m1 and Pm. Among them, P3m1 can be dynamically stable at pressure as low as 50 GPa, and Cmc21 has high Tc of 155 K at 150 GPa. Both of Cmc21 and P3m1 are host-guest hydrides, in which CH4 molecules are inserted into Im-3m-H3S and R3m-H3S sublattices, respectively. Their Tc is dominated by the H3S lattice inside. The insertion of CH4 greatly reduces the pressure required for the stability of the original H3S lattice, but it has a negative impact on superconductivity that cannot be ignored. By studying the effect of CH4 insertion in the H3S lattice, we can design hydrides with Tc close to that of H3S and a greatly reduced pressure required for stability.

cond-mat.supr-con↗

arXiv2021

Is Reinforcement Learning More Difficult Than Bandits? A Near-optimal Algorithm Escaping the Curse of Horizon

Episodic reinforcement learning and contextual bandits are two widely studied sequential decision-making problems. Episodic reinforcement learning generalizes contextual bandits and is often perceived to be more difficult due to long planning horizon and unknown state-dependent transitions. The current paper shows that the long planning horizon and the unknown state-dependent transitions (at most) pose little additional difficulty on sample complexity. We consider the episodic reinforcement learning with $S$ states, $A$ actions, planning horizon $H$, total reward bounded by $1$, and the agent plays for $K$ episodes. We propose a new algorithm, \textbf{M}onotonic \textbf{V}alue \textbf{P}ropagation (MVP), which relies on a new Bernstein-type bonus. Compared to existing bonus constructions, the new bonus is tighter since it is based on a well-designed monotonic value function. In particular, the \emph{constants} in the bonus should be subtly setting to ensure optimism and monotonicity. We show MVP enjoys an $O\left(\left(\sqrt{SAK} + S^2A\right) \poly\log \left(SAHK\right)\right)$ regret, approaching the $Ω\left(\sqrt{SAK}\right)$ lower bound of \emph{contextual bandits} up to logarithmic terms. Notably, this result 1) \emph{exponentially} improves the state-of-the-art polynomial-time algorithms by Dann et al. [2019] and Zanette et al. [2019] in terms of the dependency on $H$, and 2) \emph{exponentially} improves the running time in [Wang et al. 2020] and significantly improves the dependency on $S$, $A$ and $K$ in sample complexity.

cs.LG↗

arXiv2021

Design Principles for High Temperature Superconductors with Hydrogen-based Alloy Backbone at Moderate Pressure

Hydrogen-based superconductors provide a route to the long-sought goal of room-temperature superconductivity, but the high pressures required to metallize these materials limit their immediate application. For example, carbonaceous sulfur hydride, the first room-temperature superconductor, can reach a critical temperature (Tc) of 288 K only at the extreme pressure of 267 GPa. The next recognized challenge is the realization of room-temperature superconductivity at significantly lower pressures. Here, we propose a strategy for the rational design of high-temperature superconductors at low pressures by alloying small-radius elements and hydrogen to form ternary hydride superconductors with alloy backbones. We identify a hitherto unknown fluorite-type backbone in compositions of the form AXH8, which exhibit high temperature superconductivity at moderate pressures. The Fm-3m phase of LaBeH8, with a fluorite-type H-Be alloy backbone, is predicted to be metastable and superconducting with a Tc ~ 191 K at 50 GPa; a substantially lower pressure than that required by the geometrically similar clathrate hydride LaH10 (170 GPa). Our approach paves the way for finding high-Tc ternary hydride superconductors at conditions close to ambient pressures.

cond-mat.supr-con↗

arXiv2021

Learning from My Friends: Few-Shot Personalized Conversation Systems via Social Networks

Personalized conversation models (PCMs) generate responses according to speaker preferences. Existing personalized conversation tasks typically require models to extract speaker preferences from user descriptions or their conversation histories, which are scarce for newcomers and inactive users. In this paper, we propose a few-shot personalized conversation task with an auxiliary social network. The task requires models to generate personalized responses for a speaker given a few conversations from the speaker and a social network. Existing methods are mainly designed to incorporate descriptions or conversation histories. Those methods can hardly model speakers with so few conversations or connections between speakers. To better cater for newcomers with few resources, we propose a personalized conversation model (PCM) that learns to adapt to new speakers as well as enabling new speakers to learn from resource-rich speakers. Particularly, based on a meta-learning based PCM, we propose a task aggregator (TA) to collect other speakers' information from the social network. The TA provides prior knowledge of the new speaker in its meta-learning. Experimental results show our methods outperform all baselines in appropriateness, diversity, and consistency with speakers.

cs.CL↗