The presence of an ApoE ε4 allele (ε4+) increases the risk of developing Alzheimer's disease (AD). Previous studies support an adverse relationship between ε4+ status and brain structure and function in mild cognitive impairment and AD; in contrast, the presence of an ε2 allele may be protective. Whether these findings reflect disease-related effects or pre-existing endophenotypes, however, remains unclear. The present study examined the influence of ApoE allele status on brain structure solely during middle-age in a large, national sample. Participants were 482 men, ages 51–59, from the Vietnam Era Twin Study of Aging (VETSA). T1-weighted images were used in volumetric segmentation and cortical surface reconstruction methods to measure regional volume and thickness. Primary linear mixed effects models predicted structural measures with ApoE status (ε3/3, ε2/3, ε3/4) and control variables for effects of site, non-independence of twin data, age, and average cranial vault or cortical thickness. Relative to the ε3/3 group, the ε3/4 group demonstrated significantly thinner cortex in superior frontal and left rostral and right caudal midfrontal regions; there were no significant effects of ε4 status on any temporal lobe measures. The ε2/3 group demonstrated significantly thicker right parahippocampal cortex relative to the ε3/3 group. The ApoE ε4 allele may influence cortical thickness in frontal areas, which are later developing regions thought to be more susceptible to the natural aging process. Previous conflicting findings for mesial temporal regions may be driven by the inclusion of older individuals, who may evidence preclinical manifestations of disease, and by unexamined moderators of ε4-related effects. The presence of the ε2 allele was related to thicker cortex, supporting a protective role. Ongoing follow-up of the VETSA sample may shed light on the potential for age- and disease-related mediation of the influence of ApoE allele status.