Measuring Performance in Higher Education

K-12 education has been, in recent years, the subject of a massive amount of data-gathering and analysis. From individual classrooms to state boards of education, data is being used to evaluate, to tailor instruction, and to set goals and track performance. Through an examination of the data, we are asking questions: Are we producing the results we want to see? Are we investing public money wisely?

More recently, some of that attention has shifted to higher education.

Last month saw the release of a new College Scorecard by the U.S. Department of Education, a web-based, user-friendly presentation of data on institutions of higher learning.

Parents, students, and state taxpayers all have an interest in evaluating their investment in higher education. And the federal government is increasingly interested in the performance of higher education institutions since the entire apparatus of higher education relies heavily on federal financing, through grants and loans.

Many of the Scorecard’s statistics have long been available. In Alabama, the Alabama Commission on Higher Education (ACHE) is the centralized source of higher education data-gathering. ACHE produces regular reports on a wide variety of metrics and publishes them on its website. Collected nationally, these statistics have long been used by news organizations or other publications to rank colleges.

The new federal site makes the data more accessible, and it adds financial information about the earnings of graduates. Aggregated earnings and loan repayment information are drawn from tax returns filed with the IRS and matched with students who used financial aid to help pay for college.

As with any statistics, it is important to acknowledge the limitations of the data and also to consider it in context. The Department of Education plans to continue refining the data and its presentation. And this first publication is receiving both praise and criticism, as well as suggestions for how it might be improved in the future.

Here are a few of the limitations:

A) Earnings data is based only on students who used federal loans to help pay for college.

B) Average salary data mixes students who pursue a wide variety of professions, some highly paid and others that aren’t.

C) The socio-economic status of students entering colleges is highly correlated with student status after leaving college, thus any comparisons between schools should take the composition of the student body into account.

To further explain that last point: the effects of poverty that we see in K-12 education data don’t disappear in higher education. For example, we find lower graduation rates at schools where the majority of students come from low-income households. At those schools, students tend to enter with lower levels of college readiness, as reflected in lower average ACT scores.

Those students have often come from under-resourced or academically challenged schools. All public institutions, and, in particular, historically black colleges and universities make it a point to provide access to such students. That access is vital, but it’s equally important that students succeed once they are admitted.

Lower-income college students end up having to borrow more to finance a college education. They often face greater challenges staying in college and finishing on time because of financial pressures. The new figures show that schools that work with the most challenged students have lower rates of graduation and higher levels of debt on graduation. That is not particularly surprising, but the release of the new data is causing all colleges and universities to put more energy into keeping students in school and on track to graduate.

The new data also highlights the importance of Plan 2020, Alabama’s strategic plan for improving public education. Plan 2020 sets a goal of having all graduates prepared for college and career. All public school students now take the ACT college readiness test in the fall of the 11th grade. If a student fails to achieve a benchmark score on any of the ACT subjects, it indicates that student isn’t prepared for college-level work. Knowing that, a student and the high school can focus on those areas that need improvement.

If K-12 and colleges can cooperate in bringing these students up to proficiency before they graduate high school, those students would be less likely to need remediation once they reach college. They’d also be much more likely to graduate and move into higher earning professions.

Below is a presentation of results for Alabama schools drawn from the College Scorecard data. Use the tabs at the top of the chart to navigate through the data.


Alabama Statewide ACT Results for 2015

Alabama’s 2015 graduating class was the first in which all students took the well-known college readiness test, the ACT. New state-level reports on the 2015 results provide a wealth of information that can be useful to parents, students, and educators.

Before we take a closer look at Alabama’s results, it’s important to understand the benefits of having all students take the ACT. It’s also important to understand the consequences of that choice.

Students take the ACT to demonstrate they’re ready for college. Colleges use the results to help determine whether an applicant is admitted and whether he or she qualifies for a scholarship. Before public schools started offering the ACT, some students didn’t take it. For some, the cost was an issue. Some didn’t plan on going to college. Others might have had difficulty arranging to take the test outside of school and outside of school hours.

Now, all students have the opportunity to take the test in a convenient location at no cost to the student. This should increase college-going opportunity. And since the test is taken by high school juniors, students can focus during their senior year on areas of weakness, so that they can go to college better prepared to succeed. The ACT has four subject areas – English, reading, math and science. According to ACT, a student who scores at or above the readiness benchmark in the given subject has a 50 percent chance of making a B or better in a college-level class and a 75 percent chance of making a C or better.

The ACT is designed to test what a student has learned in school. Alabama has adopted another set of tests produced by ACT, the Aspire, to test children in grades 3-8 on what they’ve learned at each grade level. Aspire results are intended to predict roughly a student’s trajectory toward success on the ACT; students who exceed the readiness benchmark on the Aspire are expected to be on track to perform well on the ACT.

Alabama’s decision to give the ACT to all high school juniors makes a lot of sense. However, it significantly changes the pool of students taking the ACT in Alabama. Now, it’s not just the kids who plan to go to college taking the test; it’s everyone.

Alabama’s State Board of Education has set a goal of having 90 percent of Alabama high school students graduate and graduate prepared for college and career. The ACT is just one measure of whether those students are college and career ready. The ACT is designed to predict success at a four-year undergraduate university. Not every high school graduate is headed for a university. Some career-ready students will head straight into the workforce, into technical training, or the military. So, ACT results are not the only measure of career readiness. That being said, it’s a measure the state needs to see improve. Only 16 percent of the 2015 graduates were college-ready in all four subjects.

 

Since the 2015 results are the first in which all Alabama graduates took the ACT, these scores serve as a baseline from which to build. Alabama’s 2015 results shouldn’t be compared with results from previous years when the only students taking the ACT were those planning to go to college. However, the general pattern of results is similar: a higher percentage of Alabama students beat the benchmark in English and reading; while fewer are college ready in math and science.

Another caution on comparisons, comparing Alabama’s 2015 scores to the national average is not completely fair. At the national level, the pool of students taking the ACT is more heavily weighted with students who are college bound.

But Alabama’s new scores can be compared with those of 12 other states that give the ACT to all of their students. (For the record, ACT reports that 100 percent of Mississippi graduates took the test. However, according to the Mississippi Department of Education 2015 was the first year that all that state’s juniors took the ACT. That would indicate Mississippi results won’t be 100 percent of students until next year). In the chart below, you can look through the results subject-by-subject. You can sort the results by selecting the bottom axis and choosing the icon for ascending or descending results. There are two tabs on the top of the chart. The first allows you to look at the percentage of students who scored at or above the ACT benchmark for college readiness. The other tab allows you to look at the average composite score for each state in each subject area. You’ll notice some familiar patterns in the results and some surprises. Alabama outperforms Mississippi across the board while Kentucky and Tennessee consistently outperform Alabama. Somewhat surprisingly, considering North Carolina’s historic leadership in public education improvement, that state lags behind Alabama on some measures.

Giving the test to all was a brave step in honest reporting of education measures. Regardless of comparisons and statistical nuance, it is clear Alabama has significant room to grow in producing high school graduates who are ready for success in college. Fortunately, Alabama K-12 education does have a plan in place for improvement. Plan 2020 has strategies and goals that start with Pre-K and continue through to college and career success. Hopefully, as a result of these concerted efforts being made across the educational spectrum, we’ll see Alabama ACT scores rise over time.


Alabama Strategies for National Problem of Teacher Shortages

PARCA’s new Teachers Matter report, commissioned by the Business Education Alliance, identifies Alabama-specific strategies for recruiting and retaining teachers. The report’s release comes in the midst of rising concern over a local and national shortage of teachers.

As children headed back to the classroom this month, several Alabama school systems reported difficulty filling vacancies. The Tuscaloosa City Schools System was offering $5,000 signing bonuses for math teachers. The Jefferson County School System was short 30 classroom teachers heading into the school year.

PARCA’s review of Alabama Department of Education data found that the shortages were concentrated in certain fields like math, science and special education. The shortages also tended to be concentrated in rural systems and certain urban districts. Those subjects and those geographies also tend to be where Alabama faces its toughest academic challenges. Teachers Matter suggests reviving and revising the Alabama Teacher Recruitment Incentive Program to provide scholarship support and other incentives to individuals willing to teach in high-need fields and hard-to-staff schools.

The report also recommends resurrecting the Alabama Teacher Mentoring Program, which provided a $1,000 stipend to veteran teachers who coached and supported teachers in their first year in the classroom. Teachers who’ve had mentoring support tend to be more successful and persist in the profession at a far higher rate. The report also calls for creating options other than administration for talented veteran teachers who want advance professionally but who want to stay in teaching.

The shortage problem is by no means restricted to Alabama. The New York Times and National Public Radio have recently taken a look at the problem.


PARCA Briefs BCA on Teacher Quality and Progress in Public Education

Alabama public schools have made substantial progress in raising high school graduation rates, but must continue to improve student performance on measures of college and career readiness.

To do that, the state must sharpen its focus on the quality of public school teachers. That means recruiting talent to the profession, supporting existing teachers in a quest to continuously improve their craft, and rewarding teachers for successes. Alabama’s local school systems spent over $3 billion in 2014 providing teachers for public school classrooms, which accounted for 53 percent of core operating expenses. Teachers are the state’s largest and most important educational investment.

Strategies for improving teacher quality are detailed in a new report, Teachers Matter: Rethinking How Public Education Recruits, Rewards, and Retains Great Educators, which was released today at the Business Council of Alabama’s Governmental Affairs Conference at The Marriott Grand Hotel in Point Clear.

PARCA prepared the report for the Business Education Alliance, a 501(c)(3) foundation devoted to helping Alabama have a well-prepared workforce and a robust economy through initiatives to improve education that are brought about by collaboration between educators and business/industry.

The release of the new report was accompanied by a presentation and publication of the 2015 BEA Progress Report, which updates progress made on the state’s educational strategic plan, Plan 2020.

In 2014, PARCA prepared and presented a comprehensive look at Plan 2020, Obstacles into Opportunities, also commissioned by the BEA. That report looked at the potential benefits to the state of reaching its educational goals: a 90 percent high school graduation rate, with every graduate prepared for college and a career.

The release of Teachers Matter comes the same week as the Alabama Board of Education approved new and higher standards for the state’s teacher preparation programs. This school year, the State Department of Education is leading local systems statewide through the first year of a two-year design process for a new performance evaluation system for teachers. The new system, Educate Alabama, should be in place for the 2017-2018 school year. For the first time, this teacher performance evaluation system will include as a factor student performance on benchmark tests. Test scores will be one of multiple measures, including student surveys, self-assessment and self-improvement plans made by the teacher, as well as other factors.

Teachers Matter recommends:

–  Reviving and revising a state scholarship and incentive program for individuals willing to teach in high-need fields and hard-to-staff schools.
–  Restoring the Alabama Teacher Mentoring Program, which paired first-year teachers with veteran teachers to support the transition to teaching.
–  Sponsoring pilot programs that create new roles for teachers, creating pathways to grow and advance in the profession.
–  Restoring funding for a Rewards program to recognize schools and faculties that make significant performance improvements or consistently deliver excellent results.


The Poverty vs. Performance Challenge

PARCA continues to explore the results of the state’s new standardized test, the Aspire, and is developing new ways of looking at the data.

Aspire was developed by the same testing company that offers the college readiness benchmark test, the ACT. From grades 3-8, Aspire measures student progress at each grade level toward an eventual goal of graduating from high school ready for college-level work. In the graphics below, we will look at the average percentage of students scoring “proficient” on the Aspire. A student scoring at or above proficient is considered “on track” in the journey to success in college.

Below are three new glimpses of Aspire results from 2014.

In each of the graphs, the circles represent school systems. A system’s horizontal position is determined by the percentage of students in that system qualifying for a federally-funded free or reduced lunch, a marker of poverty. Systems are arranged from 100% poverty, on the left, down to 0% poverty, on the right.

A system’s vertical position represents the percentage of test scores at or above proficient. The higher the circle the greater the percentage of students scoring proficient.

These three graphs within the chart below represent blended results for math and reading from grades 3 through 8. The first looks at the results for “all students” in a system. Then, using the menu at the top, you can also view the results for students from nonpoverty and poverty backgrounds. The red line through the middle represents the relationship between proficiency and poverty. In effect, this line shows the typical proficiency level for a school system, given the level of poverty among its students.

 

What do you see in the results? The first observation is the most obvious. When results from all students are considered, systems that have higher rates of poverty have a lower percentage of students reaching proficiency. This is not a new finding. Students coming from a background of poverty tend not to do as well on these test as students from nonpoverty households. This is consistent with results produced in Alabama and nationwide for many years. Closing this achievement gap is one of the great challenges of public education.

At the same time though, these graphs also indicate that schools and school systems make a difference. Those systems that are above the average line are outperforming expectations. There are lessons to be learned from those systems. It is also obvious from the wide span of performance among poverty students, that poverty status does not dictate results. In some systems, children from poverty backgrounds are succeeding at much higher rates than in others. It will take further study to isolate the secrets to success.  

There is even more promise when you look at results at the school level. In the graph below, you can see the wide variation in success rates. Identifying successful schools and learning from their approaches should be a priority for the state and for struggling systems that want to improve.


Setting the Course for Continuous Improvement in Tuscaloosa City and County Schools

Thanks to Plan 2020, the state’s strategic plan for improving the state’s public schools, schools and the communities they serve have increased access to data on school performance measures.

Local systems are examining how their students perform on the Aspire, the new benchmark test given to student in grades 3 through 8. Systems are also tracking graduation rates, remediation rates, and scores on the ACT, the widely-known college readiness test.  These performance measures, and others, can be used in goal-setting and evaluation. Through a careful analysis of the results, school systems can recognize areas of strength and areas of weakness, and then craft plans to improve.

In Tuscaloosa, The West Alabama Chamber of Commerce and the Schools Foundation, for the third year in a row, commissioned PARCA to take a close look at the Tuscaloosa City and County Schools Systems and present the results to city and county school leaders and to the public at large at their annual Education Summit, held this year on June 16.

PARCA’s report put the two systems’ results in context, describing the financial and personnel resources each system has to work with and the demographics of each system’s students. PARCA identifies similarly situated systems in the state, creating a peer group of schools, and then compares the Tuscaloosa City and Tuscaloosa County results with those of peer systems.

“We believe that every student can learn at higher levels, and that means every school can improve its results from year to year,” Williams told the audience. “There is no such thing as a perfect school or school system, or a perfect organization of any type for that matter. A school system has a number of goals, and it needs to focus on meeting all of them. Like any organization, it will do some things very well, other things pretty well, and still others not so well. So we don’t give school systems a single rank. Rather, we analyze the key indicators so that a system can build on its successes and at the same time improve where that is needed.”


How does your school compare on the Aspire?

Take a look at the 2014 ACT Aspire results for Heard Magnet Elementary school in Dothan.

The following images show the percentage of students at each grade level scoring proficient on the Aspire, the state’s standardized test. Students scoring proficient on the tests of math and reading are considered to be on track to advance through their academic career toward an ultimate goal of graduating from high school ready to succeed in college level coursework.

The charts show the results for All Students at the school and also display the results for students who qualify for the free or reduced lunch program (identified as “Poverty” in the chart), and also for the students who don’t qualify (“Nonpoverty”).

What do we see at Heard? Here are some important things to note:
1. High percentages of students are academically on track. The percentage of proficient students at Heard is far above the state average proficiency rate (The state average is represented by the red reference line).
2. The gaps in performance between students from poverty households and students from nonpoverty households are relatively small. In some grades, the success rate for poverty and nonpoverty students is practically the same.

Heard’s results are an exception. In general, across Alabama, there is about a 30-point gap between the success rate of nonpoverty and poverty students on these benchmark tests. That’s true in both reading and math. However, Heard’s results are proof that the performance gap is not inevitable.  Similar results can be found in schools around the state.

It should also be noted that Heard has some advantages on these performance measures. It is a magnet school. It is drawing students and parents that have high expectations for academic success, which may mean that it isn’t working with the children who have the highest degree of disadvantage. Also, Heard’s student population doesn’t have the same concentration of poverty as other schools in the state. About 36 percent of Heard’s students are in the poverty subgroup. Statewide almost 60 percent of children are in that poverty subgroup. At some schools, virtually all the children come from poverty households. When we look at results statewide and across subjects and grades, a lower percentage of students achieve the proficiency benchmark in schools where poverty is more concentrated. All these considerations should be kept in mind when viewing and comparing school results.

Previously, we launched a tool for viewing 2014 ACT Aspire results that allowed you to look at your local system’s results in comparison to the state average. The default view in that tool showed separate results by grade for All Students, for Nonpoverty students, and for Poverty students. Below, we’re making a new tool available, one that allows you to look at the 2014 ACT Aspire results at the school level.

In the table below, you can look at 2014 ACT Aspire results, school-by-school, for grades 3-8. As at the system level, this tool will allow you to look at results for the various demographic subgroups available.

You’ll notice that in this view we’ve included a blue reference line labeled “Poverty.”

This is the statewide percentage of students from the poverty subgroup who scored proficient on the Aspire test in the featured grade level and subject. We include that reference line because in some cases, the state did not publish a Nonpoverty or a Poverty result. The state withheld those results in cases where the number of students in one of the groups was small enough that the reported scores might be linked to individuals.

In cases where the Poverty and Nonpoverty results are withheld, only an “All Students” result will appear. That scenario tends to occur most often in high-poverty schools and school systems. That is why we included the Poverty reference line, so that the All Students result can be compared to the statewide proficiency percentage for students for poverty households.

In subsequent posts, we plan to make available additional capabilities, like the ability to compare multiple schools or school systems on individual measures.


PARCA's school analysis for Dothan


Each year, PARCA travels to Dothan to present data on the city school system to educators, parents, and the public.

The PARCA analysis, sponsored by the Dothan-based Wiregrass Foundation, presents Dothan City School enrollment, financial data, and test scores results as they compare to peer school systems and to state and national averages. How does the system stack up over all? How do students from poverty backgrounds perform compared to similarly-situated students statewide? How do nonpoverty students perform compared to peers elsewhere? What is the system’s ultimate success rate in preparing students for college as measured by the ACT (the college readiness assessment) and by the rate at which Dothan students have to take remedial education courses after enrolling in college?

The deep dive into the data is part of the Dothan City School System’s commitment to continuously improve. The analysis provides a basis for understanding the test results. The system takes ownership of the results, providing an opportunity to celebrate successes, but also to identify areas in need of improvement. Strategies are developed to counter weaknesses. Resources are devoted to those strategies, and future results are monitored.

It’s an approach that has led to success. In 2005, over half of Dothan’s system-level results on the Alabama Reading and Math Test were below the state average. By the 2012-2013 school year, all the system’s results were at or above the state average.

Starting in 2014, Alabama changed its statewide assessment test to the ACT Aspire, and it was the results of the new test that were presented this week, along with financial data and college readiness results from ACT. Dothan’s baseline results showed that on the new tests, Dothan, as a system, bested the state average on most measures. However, when the results are viewed at the school and grade level, it is clear there is room for improvement. Like school systems across the state and nation, Dothan sees a wide disparity in results between poverty and nonpoverty students. Dothan schools with high percentages of poverty in the student body generally showed results that trailed far behind state averages. On the other hand, on several measures, poverty and nonpoverty students at Dothan magnet schools scored proficient at almost the same rate, erasing that stubborn achievement gap. In fact, in some grades and subjects, poverty students at Dothan magnets beat the proficiency rate of non-poverty students statewide.


How does your school system compare with state averages?

PARCA has been working on a new system for presenting information about Alabama’s public schools.

In the next few posts, we will be rolling out some tools that we hope educators, parents, students and communities will find useful in understanding how their local schools stack up: where they are succeeding and where they might improve.

Our first offering is a way of looking at ACT Aspire results from 2014. The display below compares the performance of Alabama systems with the state average result for students in grades 3-8 in either reading or math. The results represent the percentage of students who scored at or above the level the state considers proficient at that grade level. The state average result for all students is represented by the red reference line in the graphics.

You can explore the results by picking which system you want to compare with the state.

You can look at the results for all grades or choose to focus on one.

You can also choose to look at the results by demographic subgroup. By default, the results displayed are for the following subgroups of students within the system: “All Students,” “Nonpoverty,” and “Poverty.” You can choose to add other subgroups to the comparison or focus on a specific one.”

Across the state, you’ll note gaps in the results between various subgroups.  Students from poverty households tend, as a group, to perform less well on these tests than students from nonpoverty backgrounds. It’s true here and it’s true across the country. It’s one of the great challenges of public education. It’s also true though that kids from poverty backgrounds perform better in some systems than in others. This creates the potential to learn from those successes and replicate best practices.

These were tests administered in the spring of 2014. Students across the state are currently taking the ACT Aspire for 2015. 2014 was the first year the ACT Aspire was taken statewide. Because it’s the first year of a new test, results were slow to be published. It is hoped that results from the testing going on right now will be available much more quickly. And we plan to be in a better position to analyze and publish them quickly. The Aspire tests are one of the cornerstone measures in the state’s strategic plan for educational improvement, Plan 2020.  In future posts, we’ll present additional avenues for looking at Aspire data and will explore the results of other measures that are part of Plan 2020.


2014 Remedial Education Rates by High School

Alabama public high schools produced more graduates in 2014 than they produced the previous year, and more of them enrolled in Alabama public two-year and four-year colleges and universities, according to the latest High School Feedback report from the Alabama Commission on Higher Education (ACHE).

However, a slightly higher percentage of those entering freshman, 32.1 percent, had to take remedial courses in English or math to prepare them for college-level work, according to the ACHE report. (To explore results for schools and systems scroll to the bottom of the page). Plan 2020, the state’s strategic plan for educational improvement, calls for increasing the high school graduation rate and decreasing the need for those graduates to enroll in remedial courses.
Statewide Remediation Rate and GoalsHigh School Graduation Rate and Goals
Each year, ACHE works with the Alabama Department of Education and the state’s colleges and universities to account for how many graduates from Alabama high schools enrolled in Alabama public colleges and universities.

The report also gathers data on how many of the entering freshmen required remedial courses in college. These are courses designed to bring entering freshman up to the level of competency required to perform at the college level.

Ideally, all students graduating from high school and enrolling in college would be prepared by their high school to be college-ready.

Students who need remediation are taking courses that don’t count toward college completion. Remedial courses are a cost to the students, who are paying for non-credit courses, and a cost to colleges and universities that provide the courses.

The State Department of Education has set goals for increasing the percentage of students that graduate from high school and goals for increasing the percentage of those students who are college ready, work and who are not in need of remediation.

The latest report from ACHE tracked students who graduated from high school and enrolled in college in 2014.

Before getting too deep in the details of the data, it is important to note some limitations.

First, the only students who show up in the survey are those who graduated from Alabama public high schools and who then enrolled in Alabama public colleges and universities, both two-year and four-year institutions. Students who attended private schools for high school or who entered private colleges and universities are not counted. Neither are students who chose to enroll in out-of-state colleges and universities. So, the statistics explored don’t capture a true college-going rate for the state or for individual schools.

And the remediation rate calculated for any high school only applies to the graduates of that high school who enrolled in an Alabama public higher education institution.

As a final caveat, there is not a common definition across Alabama public higher education institutions for who qualifies for remediation. At some schools, the bar may be higher; at some, it might be lower.

So to the data:
Alabama public high schools produced 1,007 more graduates in 2014 than they did in 2013 and sent an additional 500 graduates to Alabama colleges and universities (22,872 in 2013 vs. 23,379 in 2014).

In 2014, 51 percent of graduates went on to a public two-year or four-year college in Alabama, the same percentage as the year before.

Of those recent high school graduates going to college, 11 percent needed both remedial math and remedial English; 15 percent needed only remedial math, and 6 percent needed only remedial English. In total, 32.1 percent took either remedial math, remedial English or both. That is a slightly higher percentage than in 2013 when 31.8 percent enrolled in remedial courses.

The dashboard below allows you to explore detailed results for schools. There are two tabs, one focusing on remedial data, the other on in-state public college going rates. Bear in mind that the percentage of high school graduates needing remedial math and the percentage of graduates needing remedial English won’t add up to the total percentage of graduates in need of remediation. Some of those graduates needed remediation in both subjects. The total percentage figure avoids double-counting the students in need of remediation in both subjects.