arXiv subjects

Oren Yuval

Publications and source records attributed to Oren Yuval.

3 recordsLinked to original sources

arXiv2025

Cross Validation for Correlated Data in Regression and Classification Models, with Applications to Deep Learning

We present a methodology for model evaluation and selection where the sampling mechanism violates the i.i.d. assumption. Our methodology involves a formulation of the bias between the standard Cross-Validation (CV) estimator and the mean generalization error, denoted by $w_{cv}$, and practical data-based procedures to estimate this term. This concept was introduced in the literature only in the context of a linear model with squared error loss as the criterion for prediction performance. Our proposed bias-corrected CV estimator, $\text{CV}_c=\text{CV}+w_{cv}$, can be applied to any learning model, including deep neural networks, and to a wide class of criteria for prediction performance in regression and classification tasks. We demonstrate the applicability of the proposed methodology in various scenarios where the data contains complex correlation structures (such as clustered and spatial relationships) with synthetic data and real-world datasets, providing evidence that the estimator $\text{CV}_c$ is better than the standard CV estimator. This paper is an expanded version of our published conference paper.

stat.ME↗

arXiv2023

Mixed Semi-Supervised Generalized-Linear-Regression with Applications to Deep-Learning and Interpolators

We present a methodology for using unlabeled data to design semi-supervised learning (SSL) methods that improve the predictive performance of supervised learning for regression tasks. The main idea is to design different mechanisms for integrating the unlabeled data, and include in each of them a mixing parameter $\alpha$, controlling the weight given to the unlabeled data. Focusing on Generalized Linear Models (GLM) and linear interpolators classes of models, we analyze the characteristics of different mixing mechanisms, and prove that it is consistently beneficial to integrate the unlabeled data with some nonzero mixing ratio $\alpha>0$, in terms of predictive performance. Moreover, we provide a rigorous framework to estimate the best mixing ratio where mixed-SSL delivers the best predictive performance, while using the labeled and unlabeled data on hand. The effectiveness of our methodology in delivering substantial improvement compared to the standard supervised models, in a variety of settings, is demonstrated empirically through extensive simulation, providing empirical support for our theoretical analysis. We also demonstrate the applicability of our methodology (with some heuristic modifications) to improve more complex models, such as deep neural networks, in real-world regression tasks

stat.ME↗

arXiv2020

Semi-Supervised Empirical Risk Minimization: Using unlabeled data to improve prediction

We present a general methodology for using unlabeled data to design semi supervised learning (SSL) variants of the Empirical Risk Minimization (ERM) learning process. Focusing on generalized linear regression, we analyze of the effectiveness of our SSL approach in improving prediction performance. The key ideas are carefully considering the null model as a competitor, and utilizing the unlabeled data to determine signal-noise combinations where SSL outperforms both supervised learning and the null model. We then use SSL in an adaptive manner based on estimation of the signal and noise. In the special case of linear regression with Gaussian covariates, we prove that the non-adaptive SSL version is in fact not capable of improving on both the supervised estimator and the null model simultaneously, beyond a negligible O(1/n) term. On the other hand, the adaptive model presented in this work, can achieve a substantial improvement over both competitors simultaneously, under a variety of settings. This is shown empirically through extensive simulations, and extended to other scenarios, such as non-Gaussian covariates, misspecified linear regression, or generalized linear regression with non-linear link functions.

stat.ML↗