A Better Way to Measure Student Success
Graduation Rates. The beacon of institutional success. The metric that everyone references as indicative of an institution’s ultimate goal. The one used to justify funding and support. And to decide which institutions are better than others. The metric that ignores differences in institutional and student characteristics. The one that fails to take into account external factors and gives no insight into HOW to improve. The metric that doesn’t tell the whole story. In this multi-chapter resource we discuss a better way to measure student success.
Administrators, staff, and students at colleges and universities across the country work hard to be successful. Institutions implement initiatives to reach out to and help improve the performance of under-performing student groups. Each institution is a unique combination of student and institutional characteristics; no two are exactly the same.
One-Sized Metric Doesn’t Fit All
But when it comes time to evaluate institutional performance, higher education treats institutions as if they are all the same. They lump them together and compare their performance on metrics such as acceptance, retention and completion rates. Key stakeholders make judgments about how well colleges and universities serve their students based on how they perform on these metrics as compared to other institutions. Someone assumes that a university with a 75% graduation rate is better than a college with a 50% graduation rate.
All colleges and universities would love to see 100% of their students graduate. But what does it mean when an institution’s completion rates are less than 100%? Did the institution fail? Of course not.
The Whole Story and Nothing But the Whole Story
The reality is that comparing your institution’s completion rate to that of others doesn’t tell your whole story to potential students, accreditors, regulators, governing boards, alumni, and other stakeholders.
Metrics like completion rates are important measures of performance. But comparing your completion rates for only a subset of your students and/or to those of the institution across town or others around the state ignores the fact that there are more than just first-time full time students at your campus and those institutions are different from yours.
At IEHE, we know there is a better way to tell the stories of the work your administrators, staff and students are doing: by assessing the institution’s student success against the perfect comparator—itself — and for ALL undergraduate students.
The IEHE’s RealityCheck Report analyzes completion rates against expected (RealityCheck) completion rates. We take into account two of the most important aspects that make a college or university unique – their institutional characteristics and student characteristics. An institution’s customized RealityCheck Report analyzes graduation rates for each of its degree levels, and explores subgroups of full-/part-time, first-time/transfer, and Pell/non-Pell students. Additionally, the RealityCheck Rate provides an unbiased analysis of an institution’s performance.
RealityCheck compares an institution’s actual completion rate for each student group with the calculated RealityCheck Rate – the predicted rate based on the institution’s unique characteristics. In this way, the institution can identify students who are succeeding at higher rates than expected, such as the full-time transfer students receiving Pell grants in the example on the left. The actual completion rate for this group is 68%, which is 3.0 percentage points higher than the RealityCheck Rate, giving the institution something to celebrate. The institution can look at what supports are given to this group that may be contributing to their stronger than expected success rate. Perhaps those initiatives could be offered to other student groups who are not performing as well.
Did you ever wonder why there’s so much emphasis on First-time, Full-time (FTFT) student success? Why do first-timers get so much more attention than transfers? What about part-time students? Adult learners? Online students?
Not paying equal attention to these other student groups doesn’t make a whole lot of sense. Every student’s success is equally important. Almost daily, we hear something about the declining number of high school graduates. Fewer high school grads means fewer students available to enter institutions as First-time, Full-time students. So, many institutions are turning their attention to non-traditional students (adults, returning students, online learners). This effort has the added benefit of serving the employer community by training a broader range of candidates with the skills and credentials to meet local workforce needs.
#1: Good Graduation Rates Don’t Actually Sound Good
People deeply entrenched in college student success know that the national average for graduation rates is around 50%. And it has been for decades. A graduation rate of 75% is what some colleges dream about and what others have come to expect. But for non-higher ed people (like most of your Board), a 75% graduation rate sounds like a solid C letter grade. Certainly nothing to brag about.
Boards may need context for understanding the rates. We can provide the comparison rates of ‘peer’ institutions, but we are quick to point out the differences between our institutions and those of our peers.
Why not speak to your Board in a language they understand? Wouldn’t it be more effective to discuss the resources directed to different groups of students, and the impact those financial and personnel decisions had on success? Using analyses that show the over- and under-performance by student group enables a Board to see precisely where the institution has been successful. Just as importantly, it highlights areas that could benefit from more of their strategic thinking.
#2 Public Graduation Rates Do Not Count All Students
The public graduation rate that is typically offered on government and third-party websites only counts those students who had never been to college before (not transfers) and attend full-time (rather than part-time). That essentially limits the population to only students who went to college right after high school. For the majority of colleges and universities – this graduation rate does NOT count all students. Not even close. How many folks outside higher ed would ever think that the graduation rates they hear about don’t consider all students?
At some institutions, this group of first-time college goers represents less than half of the student body. The concept of NOT counting all of your students toward such an important metric is absurd to most businesspeople. ESPECIALLY a metric that is paramount to the institutional mission the way student success is. The equivalent to businesspeople would be counting just a portion of sales, products, revenue, or expenses. Counting a portion of a metric simply isn’t done in other industries. After all, how would a business look to its investors if its reported profits only included weekend sales?
The solution is an understandable graduation rates that count all students. That way, internal and external audiences gets a complete understanding of student success. RealityCheck is the tool that addresses these issues by counting all students, disaggregating them into multiple groups, and comparing actual rates with the predicted rate for each. This allows institutions to better understand student success while accounting for unique student and institutional characteristics. The RealityCheck Report highlights the degree to which each group is over- or underperforming so the focus is on continuous improvement.
Why Has Higher Education Focused on First-Time Full-Time Students?
First, it’s important to understand how we got here. In the early 1990s, the NCAA introduced graduation rates for athletes. But how would one know if, for example, a 42% athletic graduation rate is something to boast about? You need a comparison group, of course. So, who do you compare 18-21 year-old athletes to?
Answer: Other 18-21 year olds at the SAME institution.
So, higher education began collecting data on all First-time, Full-time students in order to better understand athlete graduation rates. Who would’ve thought?
As a result, First-time, Full-time graduation rates were flawed from the get-go. They were relevant for some institutions (those that have athletes) and didn’t make much sense for others (those that have no or small populations of FTFT students).
And if the FTFT graduation metric doesn’t accurately reflect the success rates of your institution’s students, why would you use it as your primary measure of student success?
How Is RealityCheck Complete and Different?
Fast forward 30 years, and we finally have IPEDS-OM data that accounts for multiple student groups (First-time/Transfer, Full-time/Part-time, Pell/Non-Pell). IEHE has taken this data to the next level with RealityCheck. Now, you can account for institutional and student characteristics—and all your students—to give your institution a far better picture of student success.
The 12 RealityCheck Groupings are shown below.
With anything new, there is a little bit of a learning curve – but in this case – well worth it. The learning curve is short and the payoff is huge. In this section, we show how to explain graduation rates with RealityCheck.
Nearly a decade ago (in 2011), the Higher Education Research Institute (HERI) released the higher education expected graduation rate calculator. The calculator allows an institution to enter some basic summary information about their student demographics. From that, they could quickly obtain an expected graduation rate.
There is just one problem. The calculator only works for a limited number of colleges and universities. And that number is getting smaller.
In order for the calculator to work, SAT/ACT information is required. But more than 1,000 four-year institutions do not require the SAT or ACT for admissions. So, the calculator doesn’t work for these institutions. And it really, really doesn’t work for those with open admissions. This includes many community colleges, as well as some four-year institutions.
RealityCheck solves this problem by NOT using standardized test scores (such as SAT/ACT) in the data analytics. As a result – ALL institutions that serve undergraduate students can be included in the data. ALL institutions can have expected graduation rates calculated on each of 12 different student groups. And for all of the undergraduate degree levels they offer – Certificate, Associate, and Bachelor’s.
A CASE IN POINT
Consider the following sample gauge from a fictional institution (we’ll call it Notable State College). Full-time students who transfer into Notable State College (NSC) and receive Pell grant funding have a 68% completion rate. But this rate is 3.0 percentage points higher than predicted for this group. These students are performing better than would be predicted. Something for NSC to be very proud of.
Unfortunately, NSC’s part-time transfer students who don’t receive Pell funding aren’t faring as well. These non-Pell students have a 14% graduation rate. That is 4.0 percentage points below the RealityCheck rate. This student group could definitely use some extra support.
As you can see RealityCheck’s data visuals make it easy for the institution’s stakeholders to understand which groups are over- or under-performing. This can be a powerful way to identify and address the student groups that could use special attention and support to help them succeed.
The RealityCheck Report is more than a bunch of numbers. The report includes action steps for the institution to use the data. This is much more than a “nice to have.” There are a number of ways that RealityCheck can be of value to a college or university.
Tell a Better Institutional Story
One of the most underutilized IPEDS reports is Outcome Measures. First collected in the 2015-16 reporting year, IPEDS Outcome Measures (IPEDS-OM for short) collects student success data on ALL of your undergraduate students. And institutions are submitting their IPEDS-OM data in the winter IPEDS data collection.
With the addition of the IPEDS-OM survey, the long-time complaint that only first-time, full-time students are counted in IPEDS is resolved. And – it gets better. The data are disaggregated by first-time vs. transfer, full-time vs part-time, and Pell vs. Non-Pell. So – when accreditors ask for disaggregated data – guess what? The institution already has the work done! Just use IPEDS-OM data. And since all institutions submit these data, every institution can benchmark to others.
IEHE helps institutions take their IPEDS-OM data further – giving them the opportunity to develop usable, data-driven action steps to improve their completion rates through reliance on the RealityCheck report.
With an institution’s RealityCheck Report, leadership can better understand what the institution is getting right, and identify areas for improvement. Where is the campus team making a difference in student success? The reusable graphics make it easy to show others the evidence. RealityCheck gives colleges and universities the tools to better tell the institution’s student success story to both internal and external stakeholders (including accreditors).
Provide Extra Supports for At-Risk Students
Prospective students take the time to find the college that is right for them. Students no longer limit themselves to nearby institutions, and there are plenty of online options. But still, not every student starts their college journey at the institution they will graduate from. Higher education doesn’t always lose these students entirely when they drop-out. Some transfer. And today’s higher education system is purposefully designed to make the transfer process as easy as possible.
States, governing boards, and higher ed groups have spent decades developing a ‘seamless process.’ Common course numbering across state institutions (e.g., English 101 is English 101 at all state institutions) and articulation agreements enable students to transfer as many credits as possible. As a result – today’s institutions have a fair number of transfer students. And many are increasing the size of their transfer-in student cohorts.
Institutions need ways to measure the success of the growing number of transfer students.
- So, how do we judge the success of those transfer students?
- Should we treat them as a cohort, like first-time, full-time students (FTFT)?
That hardly seems reasonable. Transfer students enter with differing numbers of credits. They are starting at different distances from the finish line. Not only from FTFT students, but also from each other. This may explain why no one has created a metric designed specifically to measure the success of transfer students. Until RealityCheck.
The ability to drill down and examine how graduation rates differ for full-time vs. part-time or Pell vs. non-Pell students is an invaluable tool for an institution seeking to identify students who can benefit most from additional support.With RealityCheck, an institution can examine how its transfer students are faring compared with how they are expected to perform. Institutions already have the IPEDS-OM data. They just have to take that data and use it to drive improvement. With RealityCheck, we’ve done the analysis for you. This tool allows you to look deeper and identify answers to your most challenging question: “How do we help more of our students succeed?”
InsideHigherEd published an Op Ed of predictions for what the 2020s have in store for higher education (Will Higher Education Roar in the 2020s by John Katzman). In his opinion piece, Mr. Katzman makes a series of interesting observations and predictions. At the IEHE, we’re particularly interested in the very first point he makes regarding the rise of the nontraditional learner:
“Depending on how you define the term, close to 75% of today’s undergraduate students are ‘nontraditional’ learners. This could mean they hold a full-time job while enrolled, they are financially independent from their parents or they are a primary caregiver themselves. Perhaps they have a GED rather than a traditional high school diploma. Further, 37% of undergraduates are now part-time. It’s time for us to let go of our outdated understanding of bachelor’s students and restructure the educational experience around their new reality.”
At the IEHE, we’ve noticed that the data analytics and metrics aren’t keeping pace with the growth in nontraditional student populations. For example, until recently, the only student success indicator for part-time students in IPEDS was retention rates. And this rate is limited to first-time students; that is, those students entering the institution with no prior college experience. The introduction of IPEDS-OM now has institutions tracking part-time students’ graduation rates, both those who are first-time students and those who have transferred-in to the institution. But given the newness of the measure and the (expected) lower completion rates compared to their full-time counterparts, the metric isn’t especially useful on its own.
It is fantastic that there are finally national data collected on all part-time undergraduate students. However, little is known about part-time students, especially in comparison to full-time. For example, what is a good eight-year completion rate for part-time students? No one knows because it hasn’t been researched sufficiently (yet!).
Clearly, part-time rates should NOT be compared to full-time rates. Attending part-time is generally out of necessity (working full-time, care-giver full-time, etc.). Generally, if a student can attend full-time, they do. Part-time attendance is rarely a preference.
RealityCheck is the only metric available that provides context for part-time graduation rates – for both first-time and transfer-in students. Rather than consider the rates in a vacuum—or worse, compare them to full-time completion rates—the RealityCheck Rate analysis determines what these student success rates should have been. Thus, when the RealityCheck Rate is compared to the reported graduation rate, say for First-time, Part-time students, we get a sense of whether or not the student group is over or under performing.
RealityCheck provides much-needed context for understanding part-time student success. It helps us identify whether or not we are meeting student success expectations for part-time students. Additionally, using data analytics such as RealityCheck Rates, allows for comparison across the institution’s student groups. When explored in the context of the success of other student groups at the institution, as well as institutional programming and initiatives, we can begin to identify programs and services that might encourage more Part-time, students to persist to graduation, thereby closing the much-needed skills gap for the workforce.
Support Accreditation Efforts
There are nearly 100 accrediting bodies recognized by the Council for Higher Education Accreditation (CHEA) or the U.S. Department of Education – including regional, faith-related, career, and programmatic entities. Colleges and universities often elect to participate in multiple accrediting bodies, both regional and programmatic. The regional accreditor may be WSCUC, HLC, SACS, Middle States, or NECHE. Depending on its degree offerings, the institution may also use programmatic accreditors (in the case of education or business). It’s no wonder it feels like there is ALWAYS an accreditation effort beginning, underway, or just ending.
Long gone are the days where an accreditation visit is something undertaken once every 10 years. Most accreditors require an annual report, as well as interim updates – perhaps every three to five years – that include student success and learning. If institutions report on key issues more frequently, (it’s assumed) they will notice and address problems earlier. Sounds like a reasonable approach, right?
Sure, college and university leaders are digging into the data more frequently to monitor and respond to concerns. Naturally, you’d expect graduation rates would be going up. But nationally, they are nearly flat. Why?
There are two primary reasons for this:
- The graduation rate that gets the most publicity takes into account only a portion of students (for most institutions) – specifically, First-time, Full-time undergraduate students.
- The rates do not account for institutional and student characteristics. Thus, colleges and universities are chasing an elusive 100% completion rate.
Higher education leaders, faculty, and staff work tirelessly to improve a rate that accounts for a small subset of students and often heavily influenced by external forces (e.g., job market, employment rates, federal financial aid policies) – all things that colleges and universities have little to no control over. Remember, RealityCheck accounts for these issues by measuring the institution against the perfect comparator – itself – and disaggregates the data into multiple student groups.
What’s the Solution?
Clearly, efforts to improve overall graduation rates that only consider First-time, Full-time students haven’t been enough. IPEDS-OM allows us to examine the success rates of each student group at a more granular level, and IEHE’s RealityCheck Report can help you identify the student groups
on your campus that are doing better than expected, and those that could benefit most from more targeted supports.
Where Can We Make the Biggest Difference?
Consider an institution whose overall graduation rates are low, but the results of the RealityCheck Report indicate that it is primarily the institution’s Part-time, transfer students who are performing at a far lower rate than expected. First, this knowledge alone is a valuable asset to leadership with an upcoming accreditation visit. It helps explain why the institution isn’t where it wants to be in terms of its overall graduation rate.
In addition, it provides a guide for where to focus improvement efforts. Accrediting agencies insist on a plan for continuous improvement. And a thorough examination of what is (and isn’t) being done to support this particular student group could yield powerful information to help address concerns about the success of those students. If the institution can improve success for these Part-time, Transfer-in students, the overall completion rate will improve as well. As they say, “A rising tide lifts all boats.”
Address Institutional Weaknesses
We all know it’s hard to address your own weaknesses. But it is even harder to address weaknesses you aren’t even aware that you have.
Institutions have weaknesses too. Some are obvious, and some are not. And, like personal weaknesses, some leaders may not even recognize them. But as is the case with personal weaknesses … institutional weaknesses exist whether we address them or not. It’s better to recognize your own institutional weaknesses (and address them) before others begin to point them out. Like anything – weaknesses that are neglected continue to fester.
There is a certain sense of “spring cleaning” that can come from getting the topic out in the open and beginning the discussion. There are several advantages to getting the conversation underway:
- Create awareness of the issue (remember – not everyone has noticed this particular weakness).
- Identify severity of the issue (discussion may reveal that it isn’t as bad as one might have thought).
- Find the bright side (maybe the weakness can be turned into a strength).
Student success is often one of those topics where weaknesses aren’t always obvious. Most institutions do a better job of graduating some students than others. But which ones? It’s important to figure out your student success weak spots. RealityCheck helps institutions figure out which student groups they are serving well so they can focus on improving the student success of the ones they aren’t.
College and university foundation and advancement offices and supporting professional associations, such as CASE do important work. Really important work. The majority of people on campus don’t realize that special events, programs, scholarships, and endowed chairs are often funded by donations to the institution.
To many, there is one “type” of money – (green) and that’s it. However – multiple types of money – state money, tuition money, fee money, donations – are needed for the institution to function and flourish. And even within the donation category, there are sub-categories. Restricted funds are intended for a special purpose like scholarships, buildings, or student support services. While unrestricted funds allow the institution to ‘do good things’ of their choosing.
What most people don’t realize is that getting donations – of any type – is strategic work that requires a well-organized and coordinated effort. A significant amount of data and information is culled, analyzed and utilized in creating capital campaigns, campus-wide fundraising, and specific donor asks. This is not a haphazard approach. For example, asking a donor for $50 may result in an eager contribution. However, it could actually be a lost opportunity. What if that donor would be interested in making a much larger donation for a specific purpose that is near to their heart? Doing the work to ask the right questions to the right person can take longer but may bear better fruit.
One of the most common donation types is scholarships. Who doesn’t want to contribute to the success of a rising star? And everyone wants to ensure their money is used well. So, if a donor is considering giving money to a scholarship, they may want information about your record of student success – including the completion rates of other scholarship recipients, since past performance is a strong predictor of future performance. So, if scholarship recipients did well in the past, chances are the institution will be able to make good use of the donation.
Consider the following story. Sarah, a development officer at Extraordinary University carefully reaches out to a number of alumni with the potential of giving a large gift. Hazel is intrigued by making a difference in the lives of students at the institution she credits as significantly impacting her career. So, she’s considering making a sizable scholarship donation. She thinks about it and does her research. As part of her research, she googles Extraordinary U and finds credible government and third-party websites that show graduation rates at 75%. Recalling that a 75% is a C, she is surprised that her beloved alma mater isn’t having the impact that it had on her and her classmates. When the Development officer follows up on Hazel’s decision to make a gift, Hazel asks about the 75% — is it accurate? Are graduation rates really that low?
Sarah gets this question a lot as a Development Officer, actually. But Extraordinary University has their RealityCheck report, allowing Sarah to answer Hazel’s question without a lot of confusing higher ed jargon. Armed with her RealityCheck’s analysis, Sarah can say, “That’s a great question, Hazel. We saw that data too and dug in, because we want all of our students to be successful. We know some of our students graduate successfully, while others don’t. For example, the data show that our full-time transfer in students graduate at a higher rate than expected – four points higher, actually. That’s in large part due to scholarships and student support resources – just like the donation you are considering. We also know that some of our other student groups are under-performing. Those are exactly the students you would be helping with your donation. As you know, Extraordinary University has always been an innovator in education. We believe we can help all groups of students be successful. But we need to provide the right resources and support to those students – and that’s why we are focusing our scholarship efforts on students who have the greatest chance to benefit.
Sarah used her institution’s RealityCheck Report created by the Institute for Effectiveness in Higher Education to answer this potential donor’s question. She was able to speak to Hazel in language that resonates with her (over and under performance) rather than higher education jargon and statistics.
Without the RealityCheck Report the Development Officer might explain how the public rate is based only on a small set of students, and is not reflective of the entire population. Worse yet, the Sarah might unintentionally end up down a rabbit hole describing what a first-time full-time student is. (Hint – IEHE answers that question in another blog and how it came to be in the first place.). Alternatively, Sarah might indicate that a 75% completion rate is actually good, compared to Extraordinary’s peers. Still. Hazel might be left thinking the bar is too low if Extraordinary is pleased with a 75% graduation rate.
RealityCheck gives Sarah the narrative she needs to secure the donation which will allow Extraordinary University to support students with the greatest need, who can benefit the more. Sarah meets her goals as a Development Officer, and helps the institution achieve its goal of stronger completion rates for all students. Win-Win!
Analyze Declining Graduation Rates to Develop Actionable Solutions
When an institution’s overall graduation rates declines, everyone wants to know why. Campus leadership will want to know more, not to mention the Board and the Accreditor that is scheduled for an upcoming visit.
Increasing graduation rates and student success is part of many institutions’ strategic plan. We’ve infused thousands of dollars into multiple student success efforts that departments and ‘best practices’ say will work. After all of all of this hard work from an entire campus and directed funding, how can graduation rates decline?
The answer is complex but manageable. The single completion rate metric likely does not reflect that the multiple student groups on campus – transfer students, part-time students, and Pell students. So, one single number—THE Graduation Rate—doesn’t tell anyone which groups of students a campus was successful with and which they are not reaching.
An institution’s RealityCheck Report will demonstrate how to highlight where the college has exceeded expectations and identify where to focus future strategic efforts. The groundbreaking RealityCheck Rate analyzes reported completion rates against RealityCheck (expected) completion rates. RealityCheck Rates are based on the institution’s unique characteristics, and is an unbiased review of student success. The results can be used to concentrate faculty and staff efforts on student groups where there are gains to be made.