|
Programs > Competitive Workforce > Report Card
|
Frequently Asked Questions
General
- Why didn't the Chamber create an overall grade for each state?
- Evaluating each state's educational effectiveness is an enormously complex task, and giving one summative grade would have masked much of the variation in education delivery and outcomes among the states.
- Why didn't the Chamber use pluses and minuses with the grades?
- We used whole grades because we wanted to be clear about each state's standing.
- Why did the Chamber grade on a curve, and why was a quintile curve used instead of a specific threshold for states to meet?
- We wanted to highlight the vast differences in state performance, so we graded on a curve in six categories: Academic Achievement; Academic Achievement of Low-Income and Minority Students; Return on Investment; Truth in Advertising About Student Proficiency; Postsecondary and Workforce Readiness; and Data Quality.
When we used a curve, we largely employed a quintile approach: Broadly speaking, we gave the top 10 states an A, the next 10 a B, the next 11 a C, the next 10 a D, and the bottom 10 an F. We believe that grading on a quintile curve is a fair and reliable way to rank the states.
- Why didn't the Chamber grade all of the categories in the same way? Why weren't all the categories graded on a curve?
- Three of the categories—Rigor of Standards, 21st Century Teaching Force, and Flexibility in Management and Policy—were not graded on a curve because the data used reflect the presence or absence of state-level policies that in most cases are not associated with a numerical value. This enabled the existence of ties and did not lend itself to a curved grade distribution. While we would have liked to have graded all the states against a standard, there would have been far too many states with Ds and Fs.
- Why does my state get ranked lower than other states with the same grade?
- In some categories, all states with the same grade have the same numerical score; in these instances, states are listed alphabetically. In other categories states with the same grade have different numerical scores. In these cases, the states are ranked from highest to lowest depending on how well they performed on that measure.
- How often are you going to do this report?
- We will consider doing this report every two to three years. We do not think that significant changes will occur from year to year that will warrant doing this report annually. Many of the sources that we used are other studies that are not issued annually. However, we plan to do in-depth follow-up work that looks closely at the individual indicators in this report.
- Are you going to do more in-depth state reports?
- We will work with state leaders to share what is in this report, to identify what lagging states can learn from their high-performing peers, and how leaders can continue to improve.
- What is the National Assessment of Educational Progress (NAEP)?
- Known as "the nation's report card," NAEP examines students' academic ability and is the only source of comparable state-by-state performance data. NAEP is overseen by the federal government and assesses student performance in multiple subjects including reading, math, science, and writing. The tests are administered at regular intervals.
The two most widely referenced NAEP assessments are reading and math, which are administered in every state and Washington, DC, to 4th and 8th graders at least once every two years. Many NAEP tests are also administered in the 12th grade, but the results are not provided on a state-by-state basis. While NAEP is not an ideal metric for student performance—some critics have questioned its design and some states have complained that it does not reflect their own standards and curricula—it is the best tool we have today for measuring student achievement across the states.
- How are NAEP results reported?
- NAEP results are reported as either scale scores (a numeric value) or achievement levels: below basic, basic, proficient, and advanced. Researchers and policymakers frequently use NAEP achievement levels to report on students' knowledge and skills.
- When using NAEP data, why did the Chamber use the percentage of students at or above the proficient level rather than students at or above the basic level?
- We looked at the percentage of students scoring at or above the proficient level because this level indicates that the students have a firm grasp of the knowledge and skills needed to succeed at grade level. Students scoring at the basic level have only partially mastered the necessary knowledge and skills.
About the Categories:
Academic Achievement
- In the Academic Achievement category, my state posts average NAEP scores on some of the 4th and 8th grade math and reading assessments, but it receives a below average grade. How is that possible?
- We calculated the score for each state by averaging together into a "NAEP index" the percentage of 4th and 8th grade students scoring at or above the proficient level on math and reading on the 2005 NAEP. We then ranked the states on their index scores using a quintile curve: the top 10 states received an A, the next 10 received a B, the next 11 received a C, the next 10 a D, and the bottom 10 an F. While states may have scored above average on one or two NAEP assessments, we used the overall index score to grade the states.
- Our state academic assessment tests show that our kids are doing really well. Why did the Chamber give my state an F on Academic Achievement?
- Under federal law, each state sets its own proficiency level on state exams. While some states grade their students against high proficiency standards, others use lower standards that make the performance of their students appear more impressive. For the report card, we relied on the achievement results from NAEP, which is the only source of comparable performance data among the 50 states and Washington, DC.
Academic Achievement of Low-Income and Minority Students
- Why did some relatively racially and economically homogeneous states score so high on the Academic Achievement of Low-Income and Minority Students category?
- If a state served its low-income and minority students well, it earned high grades in this category—whether or not it is typically thought to be a "poor" or "minority" state. Moreover, although states like Wyoming are often portrayed in popular accounts as relatively homogeneous, the evidence shows that they have significant populations of low-income students and may have sizeable populations of minority students as well. The results for this category were determined solely by how well such states were educating these populations of children.
- Did the Chamber look at achievement gaps in the Academic Achievement of
- Low-Income and Minority Students category?
- We did not look at achievement gaps between subgroups. We believe that the most important question in judging the performance of minority and low-income students is what percentage score at or above the proficient level, not how much distance there is between African-American, Hispanic, low-income, and other subgroups.
- In the Academic Achievement of Low-Income and Minority Students table, the Chamber listed the percentage of minority and low-income students in each state. Did the Chamber grade a state more favorably if it had higher minority or low-income enrollment?
- The information on the percentage of African American, Hispanic, and low-income students displayed in the tables was not used to grade the states. It is provided for informational purposes only.
Return on Investment
- What is Return on Investment?
- In the business world, return on investment is a measure of the amount of benefit received, relative to the amount of expenditures, or investment. For the report card, we sought to measure the achievement that various states are producing relative to their educational expenditures, while controlling for student poverty, the percentage of students with special needs, and cost of living.
- If my state gets an A on Achievement, how could it get a D on Return on Investment?
- A state received a high grade in this category if it had high achievement and low spending. States that had high achievement and high spending tended to receive lower grades as did states with low achievement and high spending.
Truth in Advertising About Student Proficiency
- If my state gets an A on Truth in Advertising About Student Proficiency, how could it get a D on Achievement?
- States with a low grade in the Academic Achievement category earned a high grade in the Truth in Advertising category if state proficiency results closely reflected NAEP proficiency results. Again, as noted, NAEP may not be the ideal benchmark for student achievement—but it is the best benchmark that is currently available. When a state's results more closely reflect its NAEP results, it suggests that the state is accurately reporting what students know and can do. Conversely, some states report very high levels of student achievement, but compared to the NAEP, this data appears to be largely a product of grade inflation.
Note: This category does not evaluate state tests nor does it grade states on the performance of their students. Instead, the evaluation looks at how truthfully a state reports student proficiency.
Rigor of Standards
- How did the Chamber decide which states have better academic standards than others?
- To evaluate state academic standards, we relied on the work of the Thomas B. Fordham Foundation, a Washington-based think tank, which evaluated the quality and rigor of each state's science, math, and English standards. Fordham gives letter grades to each state's academic standards, and we converted those grades into numerical ones to calculate our grades.
- Achieve Inc.'s study Closing the Expectation Gap, which the Chamber used in constructing the indicator "alignment of graduation requirements with college," makes a distinction between "policy in place" and "plans to implement." It appears that the Chamber graded the former more favorably than the latter. Why?
- Since there was no way to ensure that the state would implement a policy, we did not give states credit if they reported that they planned to execute policy but had not yet done so.
- Education Week's study "Quality Counts," which the Chamber used in constructing the indicator "graduation contingent on performance on statewide exit or end-of-course exams at 10th grade level," makes a distinction between current and future policies. It appears that the Chamber graded the former more favorably than the latter. Why?
- Since there was no way to ensure that the state would implement a policy, we did not give states credit if they reported that they planned to execute policy a but had not yet done so.
Postsecondary and Workforce Readiness
- In grading Postsecondary and Workforce Readiness, what are the skills necessary to make a student both college and workplace ready?
- The business community is clear about what students should know and be able to do. The American Diploma Project, for instance, has clear benchmarks that define the knowledge needed for a student to graduate high school. In short, students must possess a combination of academic and applied skills including the ability to read, write, communicate (in English and another language), calculate, problem solve, and work in groups. Still, we do not know nearly enough about how well students are doing on these standards. Also, the lack of reliable and available data on state performance is alarming and created serious challenges in evaluating results on a state-by-state basis.
- My state's Web site shows that more than 90% of our students are graduating high school. Why does your report card provide a much lower figure?
- Official state graduation rate data are notoriously unreliable. So we relied on an estimated four-year cohort graduation rate measure created by Christopher Swanson, the research director of Education Week. This estimate is widely used and relies on grade-by-grade enrollment counts from the Common Core of Data, a data set captured by the U.S. Department of Education's National Center for Education Statistics, to approximate how many 9th graders graduate four years later. To improve the quality of graduation rate data, the National Governors Association has established the Graduation Counts Compact with all 50 states and Washington, DC.
- How was the Advanced Placement (AP) quotient calculated in the Postsecondary and Workforce Readiness category?
- We created a quotient for each of the following exams, AP Biology, AP Calculus AB,
- AP English Language, and AP U.S. History, by dividing the number of students who scored a 3 or above on the exam in 2005 by the total number of public school 11th and 12th graders in each state during the 2004–2005 school year. (The AP exam is reported on a 5-point scale with a score of 3 or above indicating that the student passed and is qualified to receive college credit.) We then averaged the four indicators into a single indicator, weighting for the number of students who took each exam.
21st Century Teaching Force
- Why doesn't the Chamber include measures of teacher certification in the 21st Century Teaching Force category?
- We reject the premise that credentialing is a reliable measure of teacher quality. To raise the capacity of the teaching force, we believe that states need to open up additional pathways into the profession while maintaining checks on quality.
- How can a state receive an F in Academic Achievement and an F in Academic Achievement of Low-Income and Minority Students, but receive a high grade in the 21st Century Teaching Force category?
- Student achievement is the primary indicator of academic success, and the achievement-based categories are among the cornerstones of this report card. At the same time, states control the policies and conditions that influence student achievement, and we graded the states on the policies that we believe are central to establishing an effective delivery system and increasing student outcomes. If a state earns a high grade in a single category, it does not necessarily mean that the state will show improvement in student achievement this year or next. State education systems are immensely complex, and we approached this project knowing that there was no magic school reform bullet.
- The description of the 21st Century Teaching Force category addresses knowledge workers. What is a knowledge worker?
- Author Peter Drucker coined the term knowledge worker in the 1950s, and it means a person who works primarily with information and uses knowledge and problem solving skills in his or her work. According to some estimates, knowledge workers outnumber other workers by 4-to-1.
- Education Week's study "Quality Counts," which the Chamber used in constructing its 21st Century Teaching Force indicators, makes a distinction between current and future policies and current programs and pilot programs. It appears that the Chamber graded the current policies and programs more favorably than the future or pilot ones. Why?
- Since there was no way to ensure that the state would implement a policy or scale up a pilot, we did not give states credit if they reported that they had a pilot program or that they planned to execute policy but had not yet done so.
Flexibility in Management and Policy
- In the Flexibility in Management and Policy category, how did the Chamber measure how much flexibility principals have?
- To examine principal control and flexibility, we relied on data from the Schools and Staff Survey, a research initiative overseen by the U.S. Department of Education's National Center for Education Statistics. We used data from the principal questionnaire, which asks "How much actual influence do you think each group or person has on decisions concerning the following activities?" Then it lists two items—"hiring new full-time teachers" and "deciding how your school budget will be spent"—and asks principals to rank the items along a scale: minor influence, moderate influence, major influence, and no influence. We looked only at principals who reported a major degree of influence.
- The Flexibility in Management and Policy category refers to virtual schools. What is a virtual school?
- A virtual school is a school that provides either some or all of its academic instruction over the Internet.
Data Quality
- What is a teacher identifier system as referred to in the Data Quality category?
- A teacher identifier system allows state officials to match teachers to their students' achievement and is a key tool in understanding teacher effectiveness and student performance.
- What is a unique statewide student identifier as referred to in the Data Quality category?
- A unique statewide student identifier is a single number that is assigned to a student and stays with that student as he or she progresses through each grade level of their education. While providing for student privacy, the number allows state officials to follow students as they move through school and helps educators track dropout and graduation rates as well as student achievement for individual students. In an effective school system, this identifier would be used to individualize support for student learning from year to year and is not simply used as a tool for state reporting of student-level information.
|
|
|
Share This Page