arXiv subjects

Cheng Li

Publications and source records attributed to Cheng Li.

At least 379 records · Page 21Linked to original sources

arXiv2020

Sparse Spectrum Gaussian Process for Bayesian Optimization

We propose a novel sparse spectrum approximation of Gaussian process (GP) tailored for Bayesian optimization. Whilst the current sparse spectrum methods provide desired approximations for regression problems, it is observed that this particular form of sparse approximations generates an overconfident GP, i.e. it produces less epistemic uncertainty than the original GP. Since the balance between predictive mean and the predictive variance is the key determinant to the success of Bayesian optimization, the current sparse spectrum methods are less suitable for it. We derive a new regularized marginal likelihood for finding the optimal frequencies to fix this over-confidence issue, particularly for Bayesian optimization. The regularizer trades off the accuracy in the model fitting with a targeted increase in the predictive variance of the resultant GP. Specifically, we use the entropy of the global maximum distribution from the posterior GP as the regularizer that needs to be maximized. Since this distribution cannot be calculated analytically, we first propose a Thompson sampling based approach and then a more efficient sequential Monte Carlo based approach to estimate it. Later, we also show that the Expected Improvement acquisition function can be used as a proxy for the maximum distribution, thus making the whole process further efficient. Experiments show considerable improvement to Bayesian optimization convergence rate over the vanilla sparse spectrum method and over a full GP when its covariance matrix is ill-conditioned due to the presence of a large number of observations.

cs.LG↗

arXiv2020

XSP: Across-Stack Profiling and Analysis of Machine Learning Models on GPUs

There has been a rapid proliferation of machine learning/deep learning (ML) models and wide adoption of them in many application domains. This has made profiling and characterization of ML model performance an increasingly pressing task for both hardware designers and system providers, as they would like to offer the best possible system to serve ML models with the target latency, throughput, cost, and energy requirements while maximizing resource utilization. Such an endeavor is challenging as the characteristics of an ML model depend on the interplay between the model, framework, system libraries, and the hardware (or the HW/SW stack). Existing profiling tools are disjoint, however, and only focus on profiling within a particular level of the stack, which limits the thoroughness and usefulness of the profiling results. This paper proposes XSP - an across-stack profiling design that gives a holistic and hierarchical view of ML model execution. XSP leverages distributed tracing to aggregate and correlates profile data from different sources. XSP introduces a leveled and iterative measurement approach that accurately captures the latencies at all levels of the HW/SW stack in spite of the profiling overhead. We couple the profiling design with an automated analysis pipeline to systematically analyze 65 state-of-the-art ML models. We demonstrate that XSP provides insights which would be difficult to discern otherwise.

cs.LG↗

arXiv2020

A Complex KBQA System using Multiple Reasoning Paths

Multi-hop knowledge based question answering (KBQA) is a complex task for natural language understanding. Many KBQA approaches have been proposed in recent years, and most of them are trained based on labeled reasoning path. This hinders the system's performance as many correct reasoning paths are not labeled as ground truth, and thus they cannot be learned. In this paper, we introduce an end-to-end KBQA system which can leverage multiple reasoning paths' information and only requires labeled answer as supervision. We conduct experiments on several benchmark datasets containing both single-hop simple questions as well as muti-hop complex questions, including WebQuestionSP (WQSP), ComplexWebQuestion-1.1 (CWQ), and PathQuestion-Large (PQL), and demonstrate strong performance.

cs.LG↗

arXiv2020

Wolf-Rayet galaxies in SDSS-IV MaNGA. I. Catalog construction and sample properties

Wolf-Rayet (WR) galaxies are a rare population of galaxies that host living high-mass stars during their WR phase (i.e. WR stars) and are thus expected to provide interesting constraints on the stellar Initial Mass Function, massive star formation, stellar evolution models, etc. Spatially resolved spectroscopy should in principle provide a more efficient way of identifying WR galaxies than single-fiber surveys of galactic centers such as SDSS-I & II, as WR stars should be more preferentially found in discs. Using IFU data from the ongoing SDSS-IV MaNGA survey, we have performed a thorough search for WR galaxies. We first identify H II regions in each datacube and carry out full spectral fitting to the stacked spectra. We then visually inspect the residual spectrum of each H II region and identify WR regions that present a significant "blue bump" at 4600-4750 A. The resulting WR catalog includes 267 WR regions of ~500pc (radius) sizes, distributed in 90 galaxies from the current sample of MaNGA (MaNGA Product Launch 7). We find WR regions are exclusively found in galaxies that show bluest colors and highest star formation rates for their mass. Most WR galaxies have late-type morphologies and show relatively large asymmetry in their images, implying that WR regions are more preferentially found in interacting/merging galaxies. We estimate the stellar mass function of WR galaxies and the mass-dependent detection rate. The detection rate of WR galaxies is typically ~2%, with weak dependence on stellar mass. This detection rate is about 40 times higher than previous studies with SDSS single fiber data, and by a factor of 2 lower than the CALIFA-based WR catalog. We make comparisons with SDSS and CALIFA studies, and conclude that different detection rates can be explained mainly by three factors: spatial coverage, spectral signal-to-noise ratio, and redshift ranges of the parent sample.

astro-ph.GA↗

arXiv2020

Structure/property relationship of semi-crystalline polymer during tensile deformation: A molecular dynamics approach

A coarse-grained molecular dynamics model of linear polyethylene-like polymer chain system was built to investigate the responds of structure and mechanical properties during uniaxial deformation. The influence of chain length, temperature, and strain rate were studied. The molecular dynamic tests showed that yielding may governed by different mechanisms at temperatures above and below Tg. Melt-recrystallization was observed at higher temperature, and destruction of crystal structures was observed at lower temperatures beyond yield point. While the higher temperature and lower strain rate have similar effects on mechanical properties. The correlated influences of time and temperature in the microscopic structures are more complicated. The evolution of microscopic characteristics such as the orientation parameter, the bond length, and the content of trans-trans conformation were calculated from the simulation. The results showed that the temperature have double effects on polymer chains. Higher temperature on one hand makes the chains more flexible, while on the other hand shortens the relaxation time of polymers. It is the interaction of these two aspects that determine the orientation parameter. During deformation, the trans conformation has experienced a rising process after the first drop process. And these microscopic structure parameters exhibit critical transaction, which are closely related to the yield point. A hypothetical model was thus proposed to describe the micro-structure and property relations based the investigations of this study.

cond-mat.soft↗

arXiv2020

Estimating dust attenuation from galactic spectra. I. methodology and tests

We develop a method to estimate the dust attenuation curve of galaxies from full spectral fitting of their optical spectra. Motivated from previous studies, we separate the small-scale features from the large-scale spectral shape, by performing a moving average method to both the observed spectrum and the simple stellar population model spectra. The intrinsic dust-free model spectrum is then derived by fitting the observed ratio of the small-scale to large-scale (S/L) components with the S/L ratios of the SSP models. The selective dust attenuation curve is then determined by comparing the observed spectrum with the dust-free model spectrum. One important advantage of this method is that the estimated dust attenuation curve is independent of the shape of theoretical dust attenuation curves. We have done a series of tests on a set of mock spectra covering wide ranges of stellar age and metallicity. We show that our method is able to recover the input dust attenuation curve accurately, although the accuracy depends slightly on signal-to-noise ratio of the spectra. We have applied our method to a number of edge-on galaxies with obvious dust lanes from the ongoing MaNGA survey, deriving their dust attenuation curves and $E(B-V)$ maps, as well as dust-free images in $g$, $r$, and $i$ bands. These galaxies show obvious dust lane features in their original images, which largely disappear after we have corrected the effect of dust attenuation. The vertical brightness profiles of these galaxies become axis-symmetric and can well be fitted by a simple model proposed for the disk vertical structure. Comparing the estimated dust attenuation curve with the three commonly-adopted model curves, we find that the Calzetti curve provides the best description of the estimated curves for the inner region of galaxies, while the Milky Way and SMC curves work better for the outer region.

astro-ph.GA↗

arXiv2020

Hierarchically Fair Federated Learning

When the federated learning is adopted among competitive agents with siloed datasets, agents are self-interested and participate only if they are fairly rewarded. To encourage the application of federated learning, this paper employs a management strategy, i.e., more contributions should lead to more rewards. We propose a novel hierarchically fair federated learning (HFFL) framework. Under this framework, agents are rewarded in proportion to their pre-negotiated contribution levels. HFFL+ extends this to incorporate heterogeneous models. Theoretical analysis and empirical evaluation on several datasets confirm the efficacy of our frameworks in upholding fairness and thus facilitating federated learning in the competitive settings.

cs.LG↗

arXiv2020

Downstream Model Design of Pre-trained Language Model for Relation Extraction Task

Supervised relation extraction methods based on deep neural network play an important role in the recent information extraction field. However, at present, their performance still fails to reach a good level due to the existence of complicated relations. On the other hand, recently proposed pre-trained language models (PLMs) have achieved great success in multiple tasks of natural language processing through fine-tuning when combined with the model of downstream tasks. However, original standard tasks of PLM do not include the relation extraction task yet. We believe that PLMs can also be used to solve the relation extraction problem, but it is necessary to establish a specially designed downstream task model or even loss function for dealing with complicated relations. In this paper, a new network architecture with a special loss function is designed to serve as a downstream model of PLMs for supervised relation extraction. Experiments have shown that our method significantly exceeded the current optimal baseline models across multiple public datasets of relation extraction.

cs.CL↗

arXiv2020

Optimal Iterative Threshold-Kernel Estimation of Jump Diffusion Processes

In this paper, we propose a new threshold-kernel jump-detection method for jump-diffusion processes, which iteratively applies thresholding and kernel methods in an approximately optimal way to achieve improved finite-sample performance. We use the expected number of jump misclassifications as the objective function to optimally select the threshold parameter of the jump detection scheme. We prove that the objective function is quasi-convex and obtain a new second-order infill approximation of the optimal threshold in closed form. The approximate optimal threshold depends not only on the spot volatility, but also the jump intensity and the value of the jump density at the origin. Estimation methods for these quantities are then developed, where the spot volatility is estimated by a kernel estimator with thresholding and the value of the jump density at the origin is estimated by a density kernel estimator applied to those increments deemed to contain jumps by the chosen thresholding criterion. Due to the interdependency between the model parameters and the approximate optimal estimators built to estimate them, a type of iterative fixed-point algorithm is developed to implement them. Simulation studies for a prototypical stochastic volatility model show that it is not only feasible to implement the higher-order local optimal threshold scheme but also that this is superior to those based only on the first order approximation and/or on average values of the parameters over the estimation time period.

math.ST↗

arXiv2020

Incorporating Expert Prior in Bayesian Optimisation via Space Warping

Bayesian optimisation is a well-known sample-efficient method for the optimisation of expensive black-box functions. However when dealing with big search spaces the algorithm goes through several low function value regions before reaching the optimum of the function. Since the function evaluations are expensive in terms of both money and time, it may be desirable to alleviate this problem. One approach to subside this cold start phase is to use prior knowledge that can accelerate the optimisation. In its standard form, Bayesian optimisation assumes the likelihood of any point in the search space being the optimum is equal. Therefore any prior knowledge that can provide information about the optimum of the function would elevate the optimisation performance. In this paper, we represent the prior knowledge about the function optimum through a prior distribution. The prior distribution is then used to warp the search space in such a way that space gets expanded around the high probability region of function optimum and shrinks around low probability region of optimum. We incorporate this prior directly in function model (Gaussian process), by redefining the kernel matrix, which allows this method to work with any acquisition function, i.e. acquisition agnostic approach. We show the superiority of our method over standard Bayesian optimisation method through optimisation of several benchmark functions and hyperparameter tuning of two algorithms: Support Vector Machine (SVM) and Random forest.

cs.LG↗

arXiv2020

DLBricks: Composable Benchmark Generation to Reduce Deep Learning Benchmarking Effort on CPUs (Extended)

The past few years have seen a surge of applying Deep Learning (DL) models for a wide array of tasks such as image classification, object detection, machine translation, etc. While DL models provide an opportunity to solve otherwise intractable tasks, their adoption relies on them being optimized to meet latency and resource requirements. Benchmarking is a key step in this process but has been hampered in part due to the lack of representative and up-to-date benchmarking suites. This is exacerbated by the fast-evolving pace of DL models. This paper proposes DLBricks, a composable benchmark generation design that reduces the effort of developing, maintaining, and running DL benchmarks on CPUs. DLBricks decomposes DL models into a set of unique runnable networks and constructs the original model's performance using the performance of the generated benchmarks. DLBricks leverages two key observations: DL layers are the performance building blocks of DL models and layers are extensively repeated within and across DL models. Since benchmarks are generated automatically and the benchmarking time is minimized, DLBricks can keep up-to-date with the latest proposed models, relieving the pressure of selecting representative DL models. Moreover, DLBricks allows users to represent proprietary models within benchmark suites. We evaluate DLBricks using $50$ MXNet models spanning $5$ DL tasks on $4$ representative CPU systems. We show that DLBricks provides an accurate performance estimate for the DL models and reduces the benchmarking time across systems (e.g. within $95\%$ accuracy and up to $4.4\times$ benchmarking time speedup on Amazon EC2 c5.xlarge).

cs.LG↗

arXiv2020

Incorporating Expert Prior Knowledge into Experimental Design via Posterior Sampling

Scientific experiments are usually expensive due to complex experimental preparation and processing. Experimental design is therefore involved with the task of finding the optimal experimental input that results in the desirable output by using as few experiments as possible. Experimenters can often acquire the knowledge about the location of the global optimum. However, they do not know how to exploit this knowledge to accelerate experimental design. In this paper, we adopt the technique of Bayesian optimization for experimental design since Bayesian optimization has established itself as an efficient tool for optimizing expensive black-box functions. Again, it is unknown how to incorporate the expert prior knowledge about the global optimum into Bayesian optimization process. To address it, we represent the expert knowledge about the global optimum via placing a prior distribution on it and we then derive its posterior distribution. An efficient Bayesian optimization approach has been proposed via posterior sampling on the posterior distribution of the global optimum. We theoretically analyze the convergence of the proposed algorithm and discuss the robustness of incorporating expert prior. We evaluate the efficiency of our algorithm by optimizing synthetic functions and tuning hyperparameters of classifiers along with a real-world experiment on the synthesis of short polymer fiber. The results clearly demonstrate the advantages of our proposed method.

stat.ML↗

arXiv2020

DLSpec: A Deep Learning Task Exchange Specification

Deep Learning (DL) innovations are being introduced at a rapid pace. However, the current lack of standard specification of DL tasks makes sharing, running, reproducing, and comparing these innovations difficult. To address this problem, we propose DLSpec, a model-, dataset-, software-, and hardware-agnostic DL specification that captures the different aspects of DL tasks. DLSpec has been tested by specifying and running hundreds of DL tasks.

cs.LG↗

arXiv2020

MLModelScope: A Distributed Platform for Model Evaluation and Benchmarking at Scale

Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that researchers are hard-pressed to analyze and study them. The complicated procedures for evaluating innovations, along with the lack of standard and efficient ways of specifying and provisioning ML/DL evaluation, is a major "pain point" for the community. This paper proposes MLModelScope, an open-source, framework/hardware agnostic, extensible and customizable design that enables repeatable, fair, and scalable model evaluation and benchmarking. We implement the distributed design with support for all major frameworks and hardware, and equip it with web, command-line, and library interfaces. To demonstrate MLModelScope's capabilities we perform parallel evaluation and show how subtle changes to model evaluation pipeline affects the accuracy and HW/SW stack choices affect performance.

cs.DC↗

arXiv2020

Benanza: Automatic $μ$Benchmark Generation to Compute "Lower-bound" Latency and Inform Optimizations of Deep Learning Models on GPUs

As Deep Learning (DL) models have been increasingly used in latency-sensitive applications, there has been a growing interest in improving their response time. An important venue for such improvement is to profile the execution of these models and characterize their performance to identify possible optimization opportunities. However, the current profiling tools lack the highly desired abilities to characterize ideal performance, identify sources of inefficiency, and quantify the benefits of potential optimizations. Such deficiencies have led to slow characterization/optimization cycles that cannot keep up with the fast pace at which new DL models are introduced. We propose Benanza, a sustainable and extensible benchmarking and analysis design that speeds up the characterization/optimization cycle of DL models on GPUs. Benanza consists of four major components: a model processor that parses models into an internal representation, a configurable benchmark generator that automatically generates micro-benchmarks given a set of models, a database of benchmark results, and an analyzer that computes the "lower-bound" latency of DL models using the benchmark data and informs optimizations of model execution. The "lower-bound" latency metric estimates the ideal model execution on a GPU system and serves as the basis for identifying optimization opportunities in frameworks or system libraries. We used Benanza to evaluate 30 ONNX models in MXNet, ONNX Runtime, and PyTorch on 7 GPUs ranging from Kepler to the latest Turing, and identified optimizations in parallel layer execution, cuDNN convolution algorithm selection, framework inefficiency, layer fusion, and using Tensor Cores.

cs.LG↗

arXiv2020

Accelerated Bayesian Optimization throughWeight-Prior Tuning

Bayesian optimization (BO) is a widely-used method for optimizing expensive (to evaluate) problems. At the core of most BO methods is the modeling of the objective function using a Gaussian Process (GP) whose covariance is selected from a set of standard covariance functions. From a weight-space view, this models the objective as a linear function in a feature space implied by the given covariance K, with an arbitrary Gaussian weight prior ${\bf w} \sim \mathcal{N} ({\bf 0}, {\bf I})$. In many practical applications there is data available that has a similar (covariance) structure to the objective, but which, having different form, cannot be used directly in standard transfer learning. In this paper we show how such auxiliary data may be used to construct a GP covariance corresponding to a more appropriate weight prior for the objective function. Building on this, we show that we may accelerate BO by modeling the objective function using this (learned) weight prior, which we demonstrate on both test functions and a practical application to short-polymer fibre manufacture.

stat.ML↗

arXiv2019

Non-rigid Registration Method between 3D CT Liver Data and 2D Ultrasonic Images based on Demons Model

The non-rigid registration between CT data and ultrasonic images of liver can facilitate the diagnosis and treatment, which has been widely studied in recent years. To improve the registration accuracy of the Demons model on the non-rigid registration between 3D CT liver data and 2D ultrasonic images, a novel boundary extraction and enhancement method based on radial directional local intuitionistic fuzzy entropy in the polar coordinates has been put forward, and a new registration workflow has been provided. Experiments show that our method can acquire high-accuracy registration results. Experiments also show that the accuracy of the results of our method is higher than that of the original Demons method and the Demons method using simulated ultrasonic image by Field II. The operation time of our registration workflow is about 30 seconds, and it can be used in the surgery.

eess.IV↗

arXiv2019

X-Net: Brain Stroke Lesion Segmentation Based on Depthwise Separable Convolution and Long-range Dependencies

The morbidity of brain stroke increased rapidly in the past few years. To help specialists in lesion measurements and treatment planning, automatic segmentation methods are critically required for clinical practices. Recently, approaches based on deep learning and methods for contextual information extraction have served in many image segmentation tasks. However, their performances are limited due to the insufficient training of a large number of parameters, which sometimes fail in capturing long-range dependencies. To address these issues, we propose a depthwise separable convolution based X-Net that designs a nonlocal operation namely Feature Similarity Module (FSM) to capture long-range dependencies. The adopted depthwise convolution allows to reduce the network size, while the developed FSM provides a more effective, dense contextual information extraction and thus facilitates better segmentation. The effectiveness of X-Net was evaluated on an open dataset Anatomical Tracings of Lesions After Stroke (ATLAS) with superior performance achieved compared to other six state-of-the-art approaches. We make our code and models available at https://github.com/Andrewsher/X-Net.

eess.IV↗