The North American Breeding Bird Survey (BBS) provides data that can be used in complex, multiscale analyses of population change, while controlling for scale‐specific nuisance factors. Many alternative models can be fit to the data, but most model selection procedures are not appropriate for hierarchical models. Leave‐one‐out cross‐validation (LOOCV), in which relative model fit is assessed by omitting an observation and assessing the prediction of a model fit using the remainder of the data, provides a reasonable approach for assessing models, but is time consuming and not feasible to apply for all observations in large data sets. We report the first large‐scale formal model selection for BBS data, applying LOOCV to stratified random samples of observations from BBS data. Our results are for 548 species of North American birds, comparing the fit of four alternative models that differ in year effect structures and in descriptions of extra‐Poisson overdispersion. We use a hierarchical model among species to evaluate posterior probabilities that models are best for individual species. Models in which differences in year effects are conditionally independent (D models) were generally favored over models in which year effects are modeled by a slope parameter and a random year effect (S models), and models in which extra‐Poisson overdispersion effects are independent and t‐distributed (H models) tended to be favored over models where overdispersion was independent and normally distributed. Our conclusions lead us to recommend a change from the conventional S model to D and H models for the vast majority of species (544/548). Comparison of estimated population trends based on the favored model relative to the S model currently used for BBS summaries indicates no consistent differences in estimated trends. Of the 18 species that showed large differences in estimated trends between models, estimated trends from the default S model were more extreme, reflecting the influence of the slope parameter in that model for species that are undergoing large population changes. WAIC, a computationally simpler alternative to LOOCV, does not appear to be a reliable alternative to LOOCV.