Physical activity has shown to be inversely associated with cognitive decline in older people. Whether this association is already present in early life has not been investigated previously. The association between early life physical activity and cognition was studied in 1,241 subjects aged 62-85 years, in a prospective population-based study. Physical activity between ages 15 and 25 years was asked retrospectively. The findings suggest a positive association between regular physical activity early in life and level of information processing speed at older age in men, not in women. The association could not be explained by current physical activity or other lifestyle factors. This finding supports the cognitive reserve hypothesis, and might suggest that early life physical activity may delay late-life cognitive deficits.