arXiv subjects

Lu Yu

Publications and source records attributed to Lu Yu.

At least 91 records · Page 5Linked to original sources

arXiv2022

Mirror Descent Strikes Again: Optimal Stochastic Convex Optimization under Infinite Noise Variance

We study stochastic convex optimization under infinite noise variance. Specifically, when the stochastic gradient is unbiased and has uniformly bounded $(1+κ)$-th moment, for some $κ\in (0,1]$, we quantify the convergence rate of the Stochastic Mirror Descent algorithm with a particular class of uniformly convex mirror maps, in terms of the number of iterations, dimensionality and related geometric parameters of the optimization problem. Interestingly this algorithm does not require any explicit gradient clipping or normalization, which have been extensively used in several recent empirical and theoretical works. We complement our convergence results with information-theoretic lower bounds showing that no other algorithm using only stochastic first-order oracles can achieve improved rates. Our results have several interesting consequences for devising online/streaming stochastic approximation algorithms for problems arising in robust statistics and machine learning.

stat.ML↗

arXiv2021

SAIL: Self-Augmented Graph Contrastive Learning

This paper studies learning node representations with graph neural networks (GNNs) for unsupervised scenario. Specifically, we derive a theoretical analysis and provide an empirical demonstration about the non-steady performance of GNNs over different graph datasets, when the supervision signals are not appropriately defined. The performance of GNNs depends on both the node feature smoothness and the locality of graph structure. To smooth the discrepancy of node proximity measured by graph topology and node feature, we proposed SAIL - a novel \underline{S}elf-\underline{A}ugmented graph contrast\underline{i}ve \underline{L}earning framework, with two complementary self-distilling regularization modules, \emph{i.e.}, intra- and inter-graph knowledge distillation. We demonstrate the competitive performance of SAIL on a variety of graph applications. Even with a single GNN layer, SAIL has consistently competitive or even better performance on various benchmark datasets, comparing with state-of-the-art baselines.

cs.LG↗

arXiv2021

End-to-End Learned Image Compression with Quantized Weights and Activations

End-to-end Learned image compression (LIC) has reached the traditional hand-crafted methods such as BPG (HEVC intra) in terms of the coding gain. However, the large network size prohibits the usage of LIC on resource-limited embedded systems. This paper reduces the network complexity by quantizing both weights and activations. 1) For the weight quantization, we study different kinds of grouping and quantization scheme at first. A channel-wise non-linear quantization scheme is determined based on the coding gain analysis. After that, we propose a fine tuning scheme to clip the weights within a certain range so that the quantization error can be reduced. 2) For the activation quantization, we first propose multiple non-linear quantization codebooks with different maximum dynamic ranges. By selecting an optimal one through a multiplexer, the quantization range can be saturated to the greatest extent. In addition, we also exploit the mean-removed quantization for the analysis transform outputs in order to reduce the bit-width cost for the specific channel with the large non-zero mean. By quantizing each weight and activation element from 32-bit floating point to 8-bit fixed point, the memory cost for both weight and activation can be reduced by 75% with negligible coding performance loss. As a result, our quantized LIC can still outperform BPG in terms of MS-SSIM. To our best knowledge, this is the first work to give a complete analysis on the coding gain and the memory cost for a quantized LIC network, which validates the feasibility of the hardware implementation.

eess.IV↗

arXiv2021

Scrybe: A Secure Audit Trail for Clinical Trial Data Fusion

Clinical trials are a multi-billion dollar industry. One of the biggest challenges facing the clinical trial research community is satisfying Part 11 of Title 21 of the Code of Federal Regulations and ISO 27789. These controls provide audit requirements that guarantee the reliability of the data contained in the electronic records. Context-aware smart devices and wearable IoT devices have become increasingly common in clinical trials. Electronic Data Capture (EDC) and Clinical Data Management Systems (CDMS) do not currently address the new challenges introduced using these devices. The healthcare digital threat landscape is continually evolving, and the prevalence of sensor fusion and wearable devices compounds the growing attack surface. We propose Scrybe, a permissioned blockchain, to store proof of clinical trial data provenance. We illustrate how Scrybe addresses each control and the limitations of the Ethereum-based blockchains. Finally, we provide a proof-of-concept integration with REDCap to show tamper resistance.

cs.CR↗

arXiv2021

Fully Neural Network Mode Based Intra Prediction of Variable Block Size

Intra prediction is an essential component in the image coding. This paper gives an intra prediction framework completely based on neural network modes (NM). Each NM can be regarded as a regression from the neighboring reference blocks to the current coding block. (1) For variable block size, we utilize different network structures. For small blocks 4x4 and 8x8, fully connected networks are used, while for large blocks 16x16 and 32x32, convolutional neural networks are exploited. (2) For each prediction mode, we develop a specific pre-trained network to boost the regression accuracy. When integrating into HEVC test model, we can save 3.55%, 3.03% and 3.27% BD-rate for Y, U, V components compared with the anchor. As far as we know, this is the first work to explore a fully NM based framework for intra prediction, and we reach a better coding gain with a lower complexity compared with the previous work.

eess.IV↗

arXiv2021

Subjective evaluation of traditional and learning-based image coding methods

We conduct a subjective experiment to compare the performance of traditional image coding methods and learning-based image coding methods. HEVC and VVC, the state-of-the-art traditional coding methods, are used as the representative traditional methods. The learning-based methods used contain not only CNN-based methods, but also a GAN-based method, all of which are advanced or typical. Single Stimuli (SS), which is also called Absolute Category Rating (ACR), is adopted as the methodology of the experiment to obtain perceptual quality of images. Additionally, we utilize some typical and frequently used objective quality metrics to evaluate the coding methods in the experiment as comparison. The experiment shows that CNN-based and GAN-based methods can perform better than traditional methods in low bit-rates. In high bit-rates, however, it is hard to verify whether CNN-based methods are superior to traditional methods. Because the GAN method does not provide models with high target bit-rates, we cannot exactly tell the performance of the GAN method in high bit-rates. Furthermore, some popular objective quality metrics have not shown the ability well to measure quality of images generated by learning-based coding methods, especially the GAN-based one.

cs.CV↗

arXiv2021

False Discovery Rates in Biological Networks

The increasing availability of data has generated unprecedented prospects for network analyses in many biological fields, such as neuroscience (e.g., brain networks), genomics (e.g., gene-gene interaction networks), and ecology (e.g., species interaction networks). A powerful statistical framework for estimating such networks is Gaussian graphical models, but standard estimators for the corresponding graphs are prone to large numbers of false discoveries. In this paper, we introduce a novel graph estimator based on knockoffs that imitate the partial correlation structures of unconnected nodes. We show that this new estimator guarantees accurate control of the false discovery rate in theory, simulations, and biological applications, and we provide easy-to-use R code.

stat.ME↗

arXiv2020

DeepI2I: Enabling Deep Hierarchical Image-to-Image Translation by Transferring from GANs

Image-to-image translation has recently achieved remarkable results. But despite current success, it suffers from inferior performance when translations between classes require large shape changes. We attribute this to the high-resolution bottlenecks which are used by current state-of-the-art image-to-image methods. Therefore, in this work, we propose a novel deep hierarchical Image-to-Image Translation method, called DeepI2I. We learn a model by leveraging hierarchical features: (a) structural information contained in the shallow layers and (b) semantic information extracted from the deep layers. To enable the training of deep I2I models on small datasets, we propose a novel transfer learning method, that transfers knowledge from pre-trained GANs. Specifically, we leverage the discriminator of a pre-trained GANs (i.e. BigGAN or StyleGAN) to initialize both the encoder and the discriminator and the pre-trained generator to initialize the generator of our model. Applying knowledge transfer leads to an alignment problem between the encoder and generator. We introduce an adaptor network to address this. On many-class image-to-image translation on three datasets (Animal faces, Birds, and Foods) we decrease mFID by at least 35% when compared to the state-of-the-art. Furthermore, we qualitatively and quantitatively demonstrate that transfer learning significantly improves the performance of I2I systems, especially for small datasets. Finally, we are the first to perform I2I translations for domains with over 100 classes.

cs.CV↗

arXiv2020

Compressing Facial Makeup Transfer Networks by Collaborative Distillation and Kernel Decomposition

Although the facial makeup transfer network has achieved high-quality performance in generating perceptually pleasing makeup images, its capability is still restricted by the massive computation and storage of the network architecture. We address this issue by compressing facial makeup transfer networks with collaborative distillation and kernel decomposition. The main idea of collaborative distillation is underpinned by a finding that the encoder-decoder pairs construct an exclusive collaborative relationship, which is regarded as a new kind of knowledge for low-level vision tasks. For kernel decomposition, we apply the depth-wise separation of convolutional kernels to build a light-weighted Convolutional Neural Network (CNN) from the original network. Extensive experiments show the effectiveness of the compression method when applied to the state-of-the-art facial makeup transfer network -- BeautyGAN.

cs.CV↗

arXiv2020

Addressing Class-Imbalance Problem in Personalized Ranking

Pairwise ranking models have been widely used to address recommendation problems. The basic idea is to learn the rank of users' preferred items through separating items into \emph{positive} samples if user-item interactions exist, and \emph{negative} samples otherwise. Due to the limited number of observable interactions, pairwise ranking models face serious \emph{class-imbalance} issues. Our theoretical analysis shows that current sampling-based methods cause the vertex-level imbalance problem, which makes the norm of learned item embeddings towards infinite after a certain training iterations, and consequently results in vanishing gradient and affects the model inference results. We thus propose an efficient \emph{\underline{Vi}tal \underline{N}egative \underline{S}ampler} (VINS) to alleviate the class-imbalance issue for pairwise ranking model, in particular for deep learning models optimized by gradient methods. The core of VINS is a bias sampler with reject probability that will tend to accept a negative candidate with a larger degree weight than the given positive item. Evaluation results on several real datasets demonstrate that the proposed sampling method speeds up the training procedure 30\% to 50\% for ranking models ranging from shallow to deep, while maintaining and even improving the quality of ranking results in top-N item recommendation.

cs.IR↗

arXiv2020

Leidenfrost drop impact on inclined superheated substrates

In real applications, drops always impact on solid walls with various inclinations. For the oblique impact of a Leidenfrost drop, which has a vapor layer under its bottom surface to prevent its direct contact with the superheated substrate, the drop can nearly frictionlessly slide along the substrate accompanied by the spreading and the retracting. To individually study these processes, we experimentally observe ethanol drops impact on superheated inclined substrates using high-speed imaging from two different views synchronously. We first study the dynamic Leidenfrost temperature, which mainly depends on the normal Weber number ${We}_\perp $. Then the substrate temperature is set to be high enough to study the Leidenfrost drop behaviors. During the spreading process, drops always keep uniform. And the maximum spreading factor $D_m/D_0$ follows a power-law dependence on the large normal Weber number ${We}_\perp $ as $D_m/D_0 = \sqrt{We_\perp /12+2}$ for $We_\perp \geq 30$. During the retracting process, drops with low impact velocities become non-uniform due to the gravity effect. For the sliding process, the residence time of all studied drops is nearly a constant, which is not affected by the inclination and $We$ number. The frictionless vapor layer results in the dimensionless sliding distance $L/D_0$ follows a power-law dependence on the parallel Weber number $We_\parallel$ as $L/D_0 \propto We_\parallel^{1/2}$. Without direct contact with the substrate, the behaviors of drops can be separately determined by ${We}_\perp $ and $We_\parallel$. When the impact velocity is too high, the drop fragments into many tiny droplets, which is called the splashing phenomenon. The critical splashing criterion is found to be $We_\perp ^*\simeq$ 120 or $K_\perp = We_\perp Re_\perp^{1/2} \simeq$ 5300 in the current parameter regime.

physics.flu-dyn↗

arXiv2020

An Analysis of Constant Step Size SGD in the Non-convex Regime: Asymptotic Normality and Bias

Structured non-convex learning problems, for which critical points have favorable statistical properties, arise frequently in statistical machine learning. Algorithmic convergence and statistical estimation rates are well-understood for such problems. However, quantifying the uncertainty associated with the underlying training algorithm is not well-studied in the non-convex setting. In order to address this shortcoming, in this work, we establish an asymptotic normality result for the constant step size stochastic gradient descent (SGD) algorithm--a widely used algorithm in practice. Specifically, based on the relationship between SGD and Markov Chains [DDB19], we show that the average of SGD iterates is asymptotically normally distributed around the expected value of their unique invariant distribution, as long as the non-convex and non-smooth objective function satisfies a dissipativity property. We also characterize the bias between this expected value and the critical points of the objective function under various local regularity conditions. Together, the above two results could be leveraged to construct confidence intervals for non-convex problems that are trained using the SGD algorithm.

stat.ML↗

arXiv2020

Semantic Drift Compensation for Class-Incremental Learning

Class-incremental learning of deep networks sequentially increases the number of classes to be classified. During training, the network has only access to data of one task at a time, where each task contains several classes. In this setting, networks suffer from catastrophic forgetting which refers to the drastic drop in performance on previous tasks. The vast majority of methods have studied this scenario for classification networks, where for each new task the classification layer of the network must be augmented with additional weights to make room for the newly added classes. Embedding networks have the advantage that new classes can be naturally included into the network without adding new weights. Therefore, we study incremental learning for embedding networks. In addition, we propose a new method to estimate the drift, called semantic drift, of features and compensate for it without the need of any exemplars. We approximate the drift of previous tasks based on the drift that is experienced by current task data. We perform experiments on fine-grained datasets, CIFAR100 and ImageNet-Subset. We demonstrate that embedding networks suffer significantly less from catastrophic forgetting. We outperform existing methods which do not require exemplars and obtain competitive results compared to methods which store exemplars. Furthermore, we show that our proposed SDC when combined with existing methods to prevent forgetting consistently improves results.

cs.CV↗

arXiv2020

Protocol Proxy: An FTE-based Covert Channel

In a hostile network environment, users must communicate without being detected. This involves blending in with the existing traffic. In some cases, a higher degree of secrecy is required. We present a proof-of-concept format transforming encryption (FTE)-based covert channel for tunneling TCP traffic through protected static protocols. Protected static protocols are UDP-based protocols with variable fields that cannot be blocked without collateral damage, such as power grid failures. We (1) convert TCP traffic to UDP traffic, (2) introduce observation-based FTE, and (3) model interpacket timing with a deterministic Hidden Markov Model (HMM). The resulting Protocol Proxy has a very low probability of detection and is an alternative to current covert channels. We tunnel a TCP session through a UDP protocol and guarantee delivery. Observation-based FTE ensures traffic cannot be detected by traditional rule-based analysis or DPI. A deterministic HMM ensures the Protocol Proxy accurately models interpacket timing to avoid detection by side-channel analysis. Finally, the choice of a protected static protocol foils stateful protocol analysis and causes collateral damage with false positives.

cs.CR↗

arXiv2019

Privacy Preserving Count Statistics

The ability to preserve user privacy and anonymity is important. One of the safest ways to maintain privacy is to avoid storing personally identifiable information (PII), which poses a challenge for maintaining useful user statistics. Probabilistic counting has been used to find the cardinality of a multiset when precise counting is too resource intensive. In this paper, probabilistic counting is used as an anonymization technique that provides a reliable estimate of the number of unique users. We extend previous work in probabilistic counting by considering its use for preserving user anonymity, developing application guidelines and including hash collisions in the estimate. Our work complements previous method by attempting to explore the causes of the deviation of uncorrected estimate from the real value. The experimental results show that if the proper register size is used, collision compensation provides estimates are as good as, if not better than, the original probabilistic counting. We develop a new anonymity metric to precisely quantify the degree of anonymity the algorithm provides.

cs.CR↗

arXiv2019

Three Dimensional Convolutional Neural Network Pruning with Regularization-Based Method

Despite enjoying extensive applications in video analysis, three-dimensional convolutional neural networks (3D CNNs)are restricted by their massive computation and storage consumption. To solve this problem, we propose a threedimensional regularization-based neural network pruning method to assign different regularization parameters to different weight groups based on their importance to the network. Further we analyze the redundancy and computation cost for each layer to determine the different pruning ratios. Experiments show that pruning based on our method can lead to 2x theoretical speedup with only 0.41% accuracy loss for 3DResNet18 and 3.28% accuracy loss for C3D. The proposed method performs favorably against other popular methods for model compression and acceleration.

cs.LG↗

arXiv2019

Learning Metrics from Teachers: Compact Networks for Image Embedding

Metric learning networks are used to compute image embeddings, which are widely used in many applications such as image retrieval and face recognition. In this paper, we propose to use network distillation to efficiently compute image embeddings with small networks. Network distillation has been successfully applied to improve image classification, but has hardly been explored for metric learning. To do so, we propose two new loss functions that model the communication of a deep teacher network to a small student network. We evaluate our system in several datasets, including CUB-200-2011, Cars-196, Stanford Online Products and show that embeddings computed using small student networks perform significantly better than those computed using standard networks of similar size. Results on a very compact network (MobileNet-0.25), which can be used on mobile devices, show that the proposed method can greatly improve Recall@1 results from 27.5\% to 44.6\%. Furthermore, we investigate various aspects of distillation for embeddings, including hint and attention layers, semi-supervised learning and cross quality distillation. (Code is available at https://github.com/yulu0724/EmbeddingDistillation.)

cs.CV↗

arXiv2018

Weakly Supervised Domain-Specific Color Naming Based on Attention

The majority of existing color naming methods focuses on the eleven basic color terms of the English language. However, in many applications, different sets of color names are used for the accurate description of objects. Labeling data to learn these domain-specific color names is an expensive and laborious task. Therefore, in this article we aim to learn color names from weakly labeled data. For this purpose, we add an attention branch to the color naming network. The attention branch is used to modulate the pixel-wise color naming predictions of the network. In experiments, we illustrate that the attention branch correctly identifies the relevant regions. Furthermore, we show that our method obtains state-of-the-art results for pixel-wise and image-wise classification on the EBAY dataset and is able to learn color names for various domains.

cs.CV↗