Detailed Grading Methodology

The methodology for each of the nine grading categories is described below.

1. Academic Achievement

The score for each state was calculated by averaging together into a "NAEP index" the percentage of 4th and 8th grade students scoring at or above the proficient level on math and reading on the 2005 National Assessment of Educational Progress (NAEP). Known as "the nation's report card," NAEP is the only nationally representative assessment of what America's students know and can do in various subject areas. NAEP assessments are conducted in mathematics and reading at grades 4 and 8. Because final counts on the number of students taking each NAEP exam were not available, we did not create a weighted index in this category or in others that used NAEP data.

We then ranked the states on their index scores using a quintile curve: The top 10 states received As, the next 10 received Bs, the next 11 received Cs, the next 10 received Ds, and the bottom 10 received Fs. States earning a given letter grade are listed alphabetically online and are ranked from highest to lowest according to their relative performance in the print edition.

SOURCE:
U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress, 2005.

2. Academic Achievement of Low-Income and Minority Students

The score for each state was calculated by first creating several NAEP subgroup indices. We did this by averaging the percentage of 4th and 8th grade students scoring at or above the proficient level on math and reading on the 2005 NAEP for the following subgroups: African-American, Hispanic, and low-income. Low-income is defined as students eligible for free and reduced-price lunch. Because final counts on the number of students taking each NAEP exam were not available, we did not create a weighted index.

We then averaged these three indices to create a ranking and graded the states on a quintile curve: The top 9 states received As, the next 9 received Bs, the next 9 received Cs, the next 9 received Ds, and the bottom 9 received Fs. States earning a given letter grade are listed alphabetically online and are ranked from highest to lowest according to their relative performance in the print edition.

Every state reported sufficient data for its low-income students. States that reported enough data for either African-Americans or Hispanics to meet NAEP sampling requirements are included here; states that did not have adequate data for both subgroups did not receive a grade. As a result, Maine, Montana, New Hampshire, North Dakota, South Dakota, and Vermont were not graded. The percentage of African-American, Hispanic, and low-income students in every state was listed for informational purposes only-the data were not used to calculate the final grades.

Some states were missing data on certain indicators, necessitating the use of proxies. The U.S. Department of Education's Common Core of Data did not report the number of students eligible for free and reduced-price lunch in Kentucky in 2003 or 2004, so we used 2002 data. We also used 2003 data from the Common Core of Data for the number of students eligible for free and reduced-price lunch in New Jersey. The Common Core of Data has not reported the number of students eligible for free and reduced-price lunch for Tennessee for nearly a decade, so we used the number of students in Title I schools as reported by the state on its Web site. The data on the number of African-American and Hispanic students in Nevada are from 2003 from the Common Core of Data.

SOURCES:
Percentage of African-American/Hispanic/Low-Income Students Scoring at or Above the Proficient Level on NAEP: U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress, 2005.

Percentage of African-American/Hispanic/Low-Income Students: U.S. Department of Education, National Center for Education Statistics, Common Core of Data.

3. Return on Investment

To calculate each state's return on investment, we sought to measure the achievement that states are producing relative to their educational expenditures. To do this, we first averaged together into a NAEP index the percentage of students scoring at or above the proficient level on the 2003 NAEP in 4th and 8th grade reading and in 4th and 8th grade math. We then divided this result by the state's current total expenditures after controlling for student poverty, the percentage of students with special needs, and the cost of living.

The expenditure data came from the 2003–2004 school year, the most recent year for which data are available. We used 2003 NAEP data because it was closest to the instructional year of the expenditure data. We also used current expenditures rather than total expenditures because current expenditures are the preferred metric among educational leaders and policymakers, who argue that it better reflects the amount of money spent on the education of students. Current expenditures include salaries, services, and supplies. They exclude capital expenses, which tend to have dramatic increases from year to year, and thus are unreliable for state-by-state comparisons.

The National Center for Education Statistics (NCES) expenditure data include money from all revenue sources: federal, state, and local.

To adjust for cost-of-living differences across states, we used the Comparable Wage Index (CWI), which is a measure of regional variations in the salaries of college graduates who are not educators. Lori L. Taylor at Texas A&M University and William J. Fowler at the NCES developed the CWI to help researchers fine-tune education finance data to make better comparisons across geographic areas.

Because it costs more to educate low-income and special education students than their peers, the expenditure data are adjusted accordingly. For every low-income student in a state, we adjusted the data by 1.2 and for special education students by 1.9. In other words, we counted each low-income student as costing 20% more than a non-low-income student to educate and each special education student as 90% more than a non-special-education student to educate. These adjustment figures were derived from a 1995 NCES report by education finance expert Thomas Parish.¹ They have been used by the newspaper Education Week and others.

We then ranked states on return on investment using a quintile curve: The top 10 states received As, the next 10 received Bs, the next 11 received Cs, the next 10 received Ds, and the bottom 10 received Fs. States earning a given letter grade are listed alphabetically online and are ranked from highest to lowest according to their relative performance in the print edition.

A few states were missing data on certain indicators, so we used proxies in those cases. The U.S. Department of Education's Common Core of Data did not report the number of students eligible for free and reduced-price lunch in Kentucky in 2003 or 2004, so we used 2002 data for the state. For New York, we used the number of students eligible for free and reduced-price lunch from 2005 from the Common Core of Data and students with individualized education plans from 2003 from the Common Core of Data. Researchers typically use data on individualized education plans to identify the number of special education students in a state. The Common Core of Data has not reported the number of students eligible for free and reduced-price lunch for Tennessee for nearly a decade, so we used the number of students in Title I schools as reported by the state on its Web site.

Given the likelihood that the relationship between achievement and spending is not linear, and that additional improvement in achievement may be more costly than initial achievement gains, this metric is obviously an imperfect measure of educational efficiency. The U.S. Chamber of Commerce and its partners are aware of this possibility and consider it worthy of examination. However, at this time, we view the relatively straightforward calculation displayed here as the most useful measure given existing data and knowledge.

SOURCES:
U.S. Department of Education, National Center for Education Statistics, National Assessment of Educational Progress.

U.S. Department of Education, National Center for Education Statistics, Current Expenditures for Public Elementary and Secondary Education: School Year 2003–04.

U.S. Department of Education, National Center for Education Statistics, Common Core of Data.

Lori L. Taylor and William J. Fowler, Jr., A Comparable Wage Approach to Geographic Cost Adjustment. U.S. Department of Education, National Center for Education Statistics, June 6, 2006.

4. Truth in Advertising About Student Proficiency

The comparison of national and state student proficiency standards grades comes from Keeping an Eye on State Standards by Paul E. Peterson and Frederick M. Hess. The authors of this report calculated a score based on the difference between the percentage of students deemed proficient by the state in 2005 and the percentage identified as proficient on the NAEP in 2005. The authors acknowledge that this is an imperfect measure of state transparency because there is some debate about using NAEP alone to benchmark state tests. However, this method is currently the only one available when comparing the transparency of reporting from one state to the next.

Minnesota, New Hampshire, and Vermont did not test their students in the 4th or 8th grades in 2005 and were not graded. The authors provided us with updated data for Washington, DC, on September 22, 2006; that information is included in the report.

We also removed the pluses and minuses that had accompanied each state's grade in the original report. States are listed alphabetically online and are ranked from highest to lowest according to their relative performance in the print edition.

SOURCE:
Paul E. Peterson and Frederick M. Hess, "Keeping an Eye on State Standards: A Race to the Bottom," Education Next, Summer (2006); 28-29. The authors provided updated data on September 22, 2006.

5. Rigor of Standards

To calculate a letter grade for this category, we used five indicators, with each indicator accounting for one-fifth of the total grade. For the first, second, and third indicators, we relied on the work of the Thomas B. Fordham Foundation, a Washington-based think tank, which evaluated the quality and rigor of each state's English, math, and science standards. To calculate grades, we converted Fordham's letter grades into numerical ones (F = 59 points; D = 65 points; C = 75 points; B = 85 points; A = 100 points).

For the fourth indicator-looking at how well states align their graduation requirements with college and workplace expectations-we gave a state an A, or 100 points, if it had gone through the process. If a state did not align its requirements, it received an F, or 59 points. In its survey, Achieve Inc., a Washington-based education and research organization, asked the states whether their high school English and mathematics standards have been aligned with postsecondary expectations and whether the business and postsecondary communities have confirmed that the high school standards are in alignment. Achieve did not confirm the responses with the higher education and business communities. While Achieve noted that some states are in the process of adopting standards, we only gave credit to those states that already had standards in place when the Achieve study was completed in February 2006.

For the final indicator-whether or not a state requires students to pass a test to graduate from high school-we gave a state an A, or 100 points, if it mandated an exit exam and an F, or 59 points, if it did not. We did not give states credit if they planned to implement this policy but have not yet done so. The data were collected in 2005 but published in 2006.

After averaging the indicators together, we then assigned letter grades based on the following scale: 90 to 100 = A; 80 to 89 = B; 70 to 79 = C; 60 to 69 = D; below 60 = F. We did not use a curve.

Note that Achieve did not have data on Vermont or Washington, D.C.'s alignment policies, so we did not grade them. The Fordham Foundation did not evaluate Iowa's state standards; we did not grade that state either.

SOURCES:
State has high-quality science, math and English standards: Chester E. Finn Jr., Michael J. Petrilli, and Liam Julian, The State of State Standards 2006. Thomas B. Fordham Foundation, August 2006.

State aligned high school graduation requirements with college and workplace expectations: Achieve, Inc., Closing the Expectations Gap, February 2006.

Graduation contingent on performance on statewide exit or end-of-course exams at 10th grade level: Editorial Projects in Education, Quality Counts 2006, January 2006.

6. Postsecondary and Workforce Readiness

To calculate a letter grade for this category, we used three indicators: Each indicator accounted for one-third of the total grade.

For the first indicator on Advanced Placement, we calculated an "AP quotient" for the following exams: AP Biology, AP Calculus AB, AP English Language, and AP U.S. History. For each test, we divided the number of students who scored a 3 or above on the exam in 2005 by the total number of public school 11th and 12th graders in each state during the 2004-2005 school year. We then averaged the four indicators into a single "AP quotient," weighted for the number of students who took each exam, and ranked the results on a quintile curve: The top 10 states received 100 points; the next 10 received 85 points; the next 10 received 75 points; the next 10 received 65 points; and the bottom 10 received 59 points.

The second indicator on graduation rates is an enrollment-based estimate that is widely used by education researchers.

The last indicator, students' chances for college attendance by age 19, measures students' persistence through high school and enrollment in college. It is a widely accepted measure of student readiness for college.

We then averaged the indicators together and distributed grades based on a curve: The top 10 states received As, the next 10 received Bs, and so forth. States earning a given letter grade are listed alphabetically online and are ranked from highest to lowest according to their relative performance in the print edition. There was insufficient data in this category to give the District of Columbia a grade.

SOURCES:
AP quotient: Students passing core AP tests divided by high school 11th and 12th graders: U.S. Chamber of Commerce, unpublished tabulations from the College Board and U.S. Department of Education, National Center for Education Statistics, Common Core of Data.

Percentage of students graduating from high school in four years with a regular diploma: Editorial Projects in Education, Diploma Counts 2006, June 2006.

Percentage of 9th Graders Chances for College by 19: Thomas Mortensen, Postsecondary Education Opportunity, 2004. The author provided updated data on November 20, 2006.

7. 21st Century Teaching Force

This category examines whether states require incoming teachers to pass basic skills tests; whether they require high school teachers to pass subject knowledge tests; whether states have alternative certification programs; and whether these programs require alternative route participants to demonstrate subject matter expertise. If a state had all four policies, it received an A. If the state had three policies in place, it received a B. If it had two policies, it received a C. If it had one policy, it received a D. If it had none of the policies, it received an F. States that had pilots or future programs planned did not receive any credit. Education Week, which collected this data, gives credit to some states that require alternative route participants to demonstrate subject matter expertise either by passing a subject matter exam or by completing course work. Those states include California, Florida, Georgia, North Carolina, Ohio, South Dakota, and Texas.

SOURCE:
Editorial Projects in Education, Quality Counts 2006. January 2006.

8. Flexibility in Management and Policy

To calculate a letter grade for this category, we used a formula that produced a numerical value based upon four indicators, with each indicator counting for one-fourth of the total grade. We then assigned letter grades on the resulting numerical value using the following scale: 90 to 100 = A; 80 to 89 = B; 70 to 79 = C; 60 to 69 = D; below 60 = F. We did not use a curve.

For the first indicator, on the strength of a state's charter laws, we used data from the Center for Education Reform (CER), converting the CER grades into numerical values (F = 59 points; D = 65 points; C = 75 points; B = 85 points; A = 100 points). We assigned states that did not have a charter school law a failing score (59 points). The states without charter school laws are Alabama, Kentucky, Maine, Montana, Nebraska, North Dakota, South Dakota, Vermont, Washington state, and West Virginia.

For the second and third indicators, regarding the amount of influence that principals reported having over school budgets and teacher hiring, we contracted with Richard Ingersoll of the University of Pennsylvania to conduct an analysis of the Schools and Staff Survey, 2003–2004 (the most recent year for which the data are available). He conducted the work in October 2006. The information comes from the principal questionnaire, which asks "How much actual influence do you think each group or person has on decisions concerning the following activities?" Then it lists two items-"hiring new full-time teachers" and "deciding how your school budget will be spent"-and asks principals to rank them along a scale: minor influence, moderate influence, major influence, and no influence. We explicitly focused on resources and hiring rather than on how much control principals have over pedagogy and curriculum; we believe that principals should control the former, but there is considerable debate over the latter, and we have remained deliberately noncommittal.

On the last indicator, a state received an A (100 points) if it had a virtual school and an F (59 points) if it did not. Education Week, which collected this data, gives credit to states allowing individual districts to provide their own online courses.

SOURCES:
Strength of charter school law: Center for Education Reform, Charter School Laws Across the States; Ranking and Scorecard, February 2006.

Percentage of principals who report a major amount of influence over how school budgets will be spent and on new teacher hiring: U.S. Department of Education, National Center for Education Statistics, Schools and Staffing Survey 2003-2004. Unpublished tabulations by Richard Ingersoll, University of Pennsylvania, October 2006.

State has established a virtual school: Editorial Projects in Education, Technology Counts 2006, May 2006.

9. Data Quality

To grade the quality of each state's data system, we created an index using the framework "10 Essential Elements of a Longitudinal Data System" established by the Data Quality Campaign (DQC). The DQC, managed by the National Center for Educational Accountability, is a national effort to encourage states to implement longitudinal data systems to improve student achievement. Our index measures how many of the following 10 policies each state had in place:

• State uses unique statewide student identifier.
  A unique statewide student identifier is a single number that is assigned to a student and can help a state track the student through his or her education career from kindergarten through 12th grade.
• State has student-level enrollment, demographic, and program participation information.
  Accurate information on student enrollment, demographics, and program participation is necessary to evaluate the effects of schools and programs, and to assess the impact of student mobility and continuous enrollment on learning.
• State has the ability to match individual students' test records from year to year to measure academic growth.
  A statewide database of individual student performance on state exams (and state-mandated local exams) should be maintained-with the ability to disaggregate the results by individual item and objective-to provide good diagnostic information to teachers.
• State has information on untested students.
  Establishing greater transparency encourages states to review their practices and prevents schools from excluding low-performing students from assessment programs to boost test scores.
• State has a teacher-identifier system with the ability to match teachers to students.
  While many states collect data on teachers, they often do not connect the database to student-level performance and enrollment records.
• State has student-level transcript information, including information on courses completed and grades earned.
  Many states are encouraging students, particularly low-income and minority students, to take rigorous courses in high school so that they are better prepared for success in postsecondary education and the job market. In most states course taking data is not collected at the state level, making it impossible to monitor the impact of these policies.
• State has student-level college readiness test scores.
  Only nine states currently collect this data and are able to report student performance on ACT and SAT exams.
• State has student-level graduation and dropout data.
  States should create data collection systems that calculate student-level graduation data and track each student through graduation.
• State has the ability to match student records between the pre-K-12 and higher education systems.
  To know if students are college- and job-ready, states need to collect data on students after they leave high school. To gain the full power of this data, states need to be able to connect the records back to the students' K-12 experience and report on how schools and programs helped, or did not help, prepare students for college and the workplace.
• State has a state data-audit system assessing data quality, validity, and reliability.
  To keep the confidence of parents, educators, and the public, states need to make sure that the education system's data are sound.

Each indicator accounted for one-tenth of the index. In other words, if a state had all 10 policies in place, we gave it a 10; if it had 9 policies in place, we gave it a 9; and so on. After calculating each state's score on the index, we distributed grades based on a broad curve. If states had the same score on the index, we gave them the same grade. Because the District of Columbia did not participate in the DQC's 2006 survey, we used data from the 2005 DQC report. We received the state policy information from the DQC in December 2006.

We selected DQC's framework because these measures provide the infrastructure for a 21st century information system, although we do not believe that data alone are sufficient for school improvement.

SOURCE:
Data Quality Campaign, Data Quality Index, 2006.

Footnote
1. T.B. Parish, C.S. Matsumoto and W.J. Fowler, Disparities in Public School District Spending (NCES 95-300). U.S. Department of Education, National Center for Education Statistics, (1995): Washington, DC: U.S. Government Printing Office.