Missing Values and the Dimensionality of Expected Returns - Missing Value Imputation in Machine Learning Asset Pricing

Aug 2, 2026

Upload by Logic

Combining many cross-sectional return predictors in machine learning often forces missing-value imputation because dropping incomplete stocks can remove most of the sample. The study compares ad hoc mean imputation with maximum likelihood methods, including EM. Despite expectations of divergence, the imputations yield nearly identical inferences for mean returns and Sharpe ratios. The paper attributes invariance to weak cross-sectional correlations and limited variance explained by principal components, then confirms similar behavior in neural network portfolios.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages53

Reading Time2 h 14 min

AccessFree to download

Abstract

Introduction

Motivation: Why missing values matter

Predictors, expected returns, and model comparisons

Dimensionality and out-of-sample tests

Invariance across imputation and forecasting methods

arXiv :2207 . 13071v3 [ stat .ME] 4 May 2023 Missing Values and the Dimensionality of Expected Returns Andrew Y. Chen Federal Reserve Board Jack McCoy Columbia University May 2023* Abstract Combining many cross-sectional return predictors (for example, in machine learning) often requires imputing missing values. We compare adhoc mean imputation with several methods including maximum likelihood. Surprisingly, maximum likelihood and ad-hoc methods lead to similar results. This is because predictors are largely independent: Correlations cluster near zero and 10 principal components (PCs) span less than 50% of total variance. Independence implies observed predictors are uninformative about missing predictors, making ad-hoc methods valid. In PC regression tests, 50 PCs are required to capture equal-weighted expected returns (30 PCs value-weighted), regardless of the imputation. We ﬁnd similar invariance in neural network portfolios. JEL Classiﬁcation: G0, G1 Keywords: stock market predictability, stock market anomalies, missing values * First submitted to [arXiv.org:](arXiv.org:) July 20, 2022. Emails: [andrew.y.chen@frb.gov](andrew.y.chen@frb.gov) (Chen, corresponding author) [and jmccoy26@gsb.columbia.edu](and jmccoy26@gsb.columbia.edu) (McCoy). This project originated from many conversations with Fabian Winkler. We thank Heiner Beckmeyer, Charlie Clarke, Harry Mamaysky, Markus Pelger, Yinan Su, and an anonymous referee for helpful comments. The views expressed herein are those of the authors and do not necessarily reﬂect the position of the Board of Governors ofthe Federal Reserve or the Federal Reserve System. 1 Introduction A growing literature applies methods from machine learning to asset pricing. These studies combine the information in dozens, or even hundreds, of crosssectional stock return predictors. Freyberger et al. (2020) combine 62; Kelly et al.(2023) combine 138; and Han et al. (2022) combine up to 193 predictors. Each of these papers ﬁnds that there are economic gains to expanding the set of predictors beyond the ﬁve used in Fama and French (2015) .1 Buried in this literature is the problem of missing values. When learning from many predictors, the standard practice of dropping stocks with missing values is often untenable. For example, applying the standard practice to the 125 mostobserved predictors inthe Chen and Zimmermann (2022) dataset drops 99.8% of stocks. So even though imputing missing values may seem dangerous, machine learning researchers often have no choice but to impute. We ﬁnd that simply imputing with cross-sectional averages (as in Kozak, Nagel and Santosh (2020); Gu, Kelly and Xiu (2020)) does a surprisingly good job of capturing expected returns. We compare this simple mean imputation with several alternatives in tests that sort stocks on ﬁtted expected returns (a la Lewellen (2015)) . The expected return models include principal components regression, gradient-boosted regression trees, and neural networks. For almost all expected return models, all of the imputations lead to very similar inferences about mean returns and Sharpe ratios. This invariance holds even for imputations that employ maximum likelihood via the expectation-maximization (EM) algorithm. This invariance comes from the fact that cross-sectional predictors are largely uncorrelated (Green, Hand and Zhang (2013); McLean and Pontiff (2016); Chen and Zimmermann (2022)) . Almost all cross-sectional correlations lie between ¡0.25 and Å0.25, and the ﬁrst 10 principal components span only 50% of total variance. As a result, observed predictors provide little information about the missing predictors, and one might as well simply impute missing values with cross-sectional means. More precisely, our EM imputation boils down to a set 1 Lewellen (2015) ﬁnds that linear regressions on 15 predictors, including size, B/M, momentum, proﬁtability, and investment lead to an annualized Sharpe Ratio of around 0.8 (e

FAQs

Why is imputing missing values necessary when using many stock return predictors?

With dozens or hundreds of predictors, dropping stocks with missing values can become impractical and remove nearly the entire sample. Imputation is therefore often required to continue building predictive models and portfolios.

How do ad hoc mean imputation and maximum likelihood (EM) compare in this study?

Both approaches produce very similar results in tests of mean returns and Sharpe ratios. The invariance persists even when maximum likelihood is implemented via the EM algorithm.

What explains the invariance in results across different imputations?

The paper argues that cross-sectional predictors are largely uncorrelated and that the first principal components span limited total variance. As a result, observed predictors contain little information about missing predictors, making simple ad hoc imputation effectively valid.

Aug 2, 2026

Upload by Logic