arXiv subjects

Yang You

Publications and source records attributed to Yang You.

242 records · Page 14Linked to original sources

arXiv2018

ImageNet Training in Minutes

Finishing 90-epoch ImageNet-1k training with ResNet-50 on a NVIDIA M40 GPU takes 14 days. This training requires 10^18 single precision operations in total. On the other hand, the world's current fastest supercomputer can finish 2 * 10^17 single precision operations per second (Dongarra et al 2017, https://www.top500.org/lists/2017/06/). If we can make full use of the supercomputer for DNN training, we should be able to finish the 90-epoch ResNet-50 training in one minute. However, the current bottleneck for fast DNN training is in the algorithm level. Specifically, the current batch size (e.g. 512) is too small to make efficient use of many processors. For large-scale DNN training, we focus on using large-batch data-parallelism synchronous SGD without losing accuracy in the fixed epochs. The LARS algorithm (You, Gitman, Ginsburg, 2017, arXiv:1708.03888) enables us to scale the batch size to extremely large case (e.g. 32K). We finish the 100-epoch ImageNet training with AlexNet in 11 minutes on 1024 CPUs. About three times faster than Facebook's result (Goyal et al 2017, arXiv:1706.02677), we finish the 90-epoch ImageNet training with ResNet-50 in 20 minutes on 2048 KNLs without losing accuracy. State-of-the-art ImageNet training speed with ResNet-50 is 74.9% top-1 test accuracy in 15 minutes. We got 74.9% top-1 test accuracy in 64 epochs, which only needs 14 minutes. Furthermore, when we increase the batch size to above 16K, our accuracy is much higher than Facebook's on corresponding batch sizes. Our source code is available upon request.

cs.CV↗

arXiv2017

Large Batch Training of Convolutional Networks

A common way to speed up training of large convolutional networks is to add computational units. Training is then performed using data-parallel synchronous Stochastic Gradient Descent (SGD) with mini-batch divided between computational units. With an increase in the number of nodes, the batch size grows. But training with large batch size often results in the lower model accuracy. We argue that the current recipe for large batch training (linear learning rate scaling with warm-up) is not general enough and training may diverge. To overcome this optimization difficulties we propose a new training algorithm based on Layer-wise Adaptive Rate Scaling (LARS). Using LARS, we scaled Alexnet up to a batch size of 8K, and Resnet-50 to a batch size of 32K without loss in accuracy.

cs.CV↗

arXiv2017

Scaling Deep Learning on GPU and Knights Landing clusters

The speed of deep neural networks training has become a big bottleneck of deep learning research and development. For example, training GoogleNet by ImageNet dataset on one Nvidia K20 GPU needs 21 days. To speed up the training process, the current deep learning systems heavily rely on the hardware accelerators. However, these accelerators have limited on-chip memory compared with CPUs. To handle large datasets, they need to fetch data from either CPU memory or remote processors. We use both self-hosted Intel Knights Landing (KNL) clusters and multi-GPU clusters as our target platforms. From an algorithm aspect, current distributed machine learning systems are mainly designed for cloud systems. These methods are asynchronous because of the slow network and high fault-tolerance requirement on cloud systems. We focus on Elastic Averaging SGD (EASGD) to design algorithms for HPC clusters. Original EASGD used round-robin method for communication and updating. The communication is ordered by the machine rank ID, which is inefficient on HPC clusters. First, we redesign four efficient algorithms for HPC systems to improve EASGD's poor scaling on clusters. Async EASGD, Async MEASGD, and Hogwild EASGD are faster \textcolor{black}{than} their existing counterparts (Async SGD, Async MSGD, and Hogwild SGD, resp.) in all the comparisons. Finally, we design Sync EASGD, which ties for the best performance among all the methods while being deterministic. In addition to the algorithmic improvements, we use some system-algorithm codesign techniques to scale up the algorithms. By reducing the percentage of communication from 87% to 14%, our Sync EASGD achieves 5.3x speedup over original EASGD on the same platform. We get 91.5% weak scaling efficiency on 4253 KNL cores, which is higher than the state-of-the-art implementation.

cs.DC↗

arXiv2017

Resource Allocation for a Full-Duplex Base Station Aided OFDMA System

Exploiting full-duplex (FD) technology on base stations (BSs) is a promising solution to enhancing the system performance. Motivated by this, we revisit a full-duplex base station (FD-BS) aided OFDMA system, which consists of one BS, several uplink/downlink users and multiple subcarriers. A joint 3-dimensional (3D) mapping scheme among subcarriers, down-link users (DUEs), uplink users (UUEs) is considered as well as an associated power allocation optimization. In detail, we first decompose the complex 3D mapping problem into three 2-dimensional sub ones and solve them by using the iterative Hungarian method, respectively. Then based on the Lagrange dual method, we sequentially solve the power allocation and 3- dimensional mapping problem by fixing a dual point. Finally, the optimal solution can be obtained by utilizing the sub-gradient method. Unlike existing work that only solves either 3D mapping or power allocation problem but with a high computation complexity, we tackle both of them and have successfully reduced computation complexity from exponential to polynomial order. Numerical simulations are conducted to verify the proposed scheme.

cs.IT↗

arXiv2017

A Joint Resource Allocation Scheme for Multi-User Full-Duplex OFDMA Systems

Full-duplex is a promising technology and great system performance enhancement can be achieved especially when we apply it to base stations. In this paper, we consider a Multi-User OFDMA network consisting of one Full-duplex base station, several uplink users, and a number of downlink users. A joint subcarrier, downlink user and uplink user mapping and power allocation optimization problem is investigated. In detail, we first decompose the 3-dimensional mapping problem into three 2-dimensional sub-problems and solve them by iteratively using classical Hungarian Method. Then, based on the dual method, we sequentially solve the power allocation and 3-dimensional mapping problem at each dual point. And the optimal solution to the dual problem is derived by using sub-gradient method. Unlike existing methods that only solve one sub-problem but with a high computation complexity, we tackle both of them and successfully reducing computation complexity from exponential to polynomial order. Numerical simulations are conducted to verify the proposed system.

cs.IT↗

arXiv2016

A Vertical Channel Model of Molecular Communication based on Alcohol Molecules

The study of Molecular Communication(MC) is more and more prevalence, and channel model of MC plays an important role in the MC System. Since different propagation environment and modulation techniques produce different channel model, most of the research about MC are in horizontal direction,but in nature the communications between nano machines are in short range and some of the information transportation are in the vertical direction, such as transpiration of plants, biological pump in ocean, and blood transportation from heart to brain. Therefore, this paper we propose a vertical channel model which nano-machines communicate with each other in the vertical direction based on pure diffusion. We first propose a vertical molecular communication model, we mainly considered the gravity as the factor, though the channel model is also affected by other main factors, such as the flow of the medium, the distance between the transmitter and the receiver, the delay or sensitivity of the transmitter and the receiver. Secondly, we set up a test-bed for this vertical channel model, in order to verify the difference between the theory result and the experiment data. At last, we use the data we get from the experiment and the non-linear least squares method to get the parameters to make our channel model more accurate.

cs.ET↗

arXiv2013

Retrieval of Sparse Solutions of Multiple-Measurement Vectors via Zero-point Attracting Projection

A new sparse signal recovery algorithm for multiple-measurement vectors (MMV) problem is proposed in this paper. The sparse representation is iteratively drawn based on the idea of zero-point attracting projection (ZAP). In each iteration, the solution is first updated along the negative gradient direction of an approximate $\ell_{2,0}$ norm to encourage sparsity, and then projected to the solution space to satisfy the under-determined equation. A variable step size scheme is adopted further to accelerate the convergence as well as to improve the recovery accuracy. Numerical simulations demonstrate that the performance of the proposed algorithm exceeds the references in various aspects, as well as when applied to the Modulated Wideband Converter, where recovering MMV problem is crucial to its performance.

cs.IT↗

arXiv2013

Zero-point attracting projection algorithm for sequential compressive sensing

Sequential Compressive Sensing, which may be widely used in sensing devices, is a popular topic of recent research. This paper proposes an online recovery algorithm for sparse approximation of sequential compressive sensing. Several techniques including warm start, fast iteration, and variable step size are adopted in the proposed algorithm to improve its online performance. Finally, numerical simulations demonstrate its better performance than the relative art.

cs.IT↗