This chapter argues that although the health literacy field has many frameworks and conceptual models, it lacks core components necessary to address health literacy as both a pressing public health and population health matter. Health literacy stakeholders need to develop three sets of activities that can provide the infrastructure for large-scale health literacy improvement. First, the field needs an ‘epidemiology of health literacy’ so we have a sustained and robust mechanism to collect and report data on multiple health literacy dimensions. Second, stakeholders should agree on standards to educate and train health professionals; develop health materials; and conduct organizational assessments. Third, the field should adopt a Health Literacy in all Policies stance and prepare to conduct health literacy impact assessments so previously hidden or minimized health literacy effects from health, education, and social polices become visible and measurable.