A simulation study examining the effects of 'regularization' on estimates of genetic covariance matrices for small samples is presented. This is achieved by penalizing the likelihood, and three types of penalties are examined. It is shown that regularized estimation can substantially enhance the accuracy of estimates of genetic parameters. Penalties shrinking estimates of genetic covariances or correlations towards their phenotypic counterparts acted somewhat differently to those aimed reducing the spread of sample eigenvalues. While improvements of estimates were found to be comparable overall, shrinkage of genetic towards phenotypic correlations resulted in least bias.

Eigenvalues and eigenvectors of covariance matrices are important statistics for multivariate problems in many applications, including quantitative genetics. Estimates of these quantities are subject to different types of bias. This article reviews and extends the existing theory on these biases, considering a balanced one-way classification and restricted maximum-likelihood estimation. Biases are due to the spread of sample roots and arise from ignoring selected principal components when imposing constraints on the parameter space, to ensure positive semidefinite estimates or to estimate covariance matrices of chosen, reduced rank. In addition, it is shown that reduced-rank estimators that consider only the leading eigenvalues and -vectors of the 'between-group' covariance matrix may be biased due to selecting the wrong subset of principal components. In a genetic context, with groups representing families, this bias is inverse proportional to the degree of genetic relationship among family members, but is independent of sample size. Theoretical results are supplemented by a simulation study, demonstrating close agreement between predicted and observed bias for large samples. It is emphasized that the rank of the genetic covariance matrix should be chosen sufficiently large to accommodate all important genetic principal components, even though, paradoxically, this may require including a number of components with negligible eigenvalues. A strategy for rank selection in practical analyses is outlined.

Multivariate estimation fitting a common structure to estimates of genetic and environmental covariance matrices is examined in a simple simulation study. It is shown that such parsimonious estimation can considerably reduce sampling variation. However, if the assumption of similarity in structure does not hold at least approximately, bias in estimates of the genetic covariance matrix can be substantial. For small samples and more than a few traits, structured estimation is likely to reduce mean square error even if bias is quite large. Hence such models should be used cautiously.

Obtaining accurate estimates of the genetic covariance matrix ∑G for multivariate data is a fundamental task in quantitative genetics and important for both evolutionary biologists and plant or animal breeders. Classical methods for estimating ∑G are well known to suffer from substantial sampling errors; importantly, its leading eigenvalues are systematically overestimated. This article proposes a framework that exploits information in the phenotypic covariance matrix ∑P in a new way to obtain more accurate estimates of ∑G. The approach focuses on the "canonical heritabilities" (the eigenvalues of ∑P⁻¹∑G), which may be estimated with more precision than those of ∑G because ∑P is estimated more accurately. Our method uses penalized maximum likelihood and shrinkage to reduce bias in estimates of the canonical heritabilities. This in turn can be exploited to get substantial reductions in bias for estimates of the eigenvalues of ∑G and a reduction in sampling errors for estimates of ∑G. Simulations show that improvements are greatest when sample sizes are small and the canonical heritabilities are closely spaced. An application to data from beef cattle demonstrates the efficacy this approach and the effect on estimates of heritabilities and correlations. Penalized estimation is recommended for multivariate analyses involving more than a few traits or problems with limited data.

Fitting only the leading principal components allows genetic covariance matrices to be modelled parsimoniously, yielding reduced rank estimates. If principal components with non-zero variances are omitted from the model, genetic variation is moved into the covariance matrices for residuals or other random effects. The resulting bias in estimates of genetic eigen-values and -vectors is examined.