The latest PISA results are about to be released to the public. How will the media spin the latest results? What outrage will surface? What drastic policy shifts will happen as a result? What pressure will be put on front line educators because of how their jurisdiction fares in the rankings?
Why are we obsessed with quantifying and ranking student performance? According to Jerry Z. Muller, in his book “Tyranny of Metrics“, there are two problems with this kind of thinking.
First, it isn’t true that anything that can be measured can be improved. Learning is a complex outcome, yet standardized tests like PISA measure only the relatively simple. So many important learnings are not being measured, like innovation, initiative, creativity, cooperation and mentoring.
In Respect the Power of Large-Scale Assessments: What They Cannot Measure, September 30, 2019 by Zhenguo Yuan, & Yong Zhao we get a sense of what is not done well in Large Scale Assessments.
According to Yuan and Zhao, what they cannot measure includes the following.
- They can’t measure what they are not intended to measure. In other words, we can’t overgeneralize the results – “the results of PISA have been generalized to reflect the quality of educational systems although they simply reflect how well 15-year-olds in different systems perform on the PISA assessments in math, reading, and science.”
- They can’t measure what we don’t know or understand. Since Large Scale Assessments don’t predict success, there must be aspects of success we are not measuring with tests.
- They cannot measure exceptionalities beyond the limits of the test.
- Skills and learning exist within contexts and cultures. They are fluid and dynamic and situation-dependent.
- Every individual has a jagged profile of abilities. They are not the sum of a few data points.
It is particularly troubling when the people making up the test are also the people profiting from the sales of the remediation solution when the jurisdiction fails to achieve the desired success. When we strip away the context for the learning, results can appear to be more certain than they really are.
Secondly, people don’t always respond to incentives in the way we think they will. Ranking scores presents a distorted view of human motivation. Most people are already trying to do well. Pressure to do better on a test can lead to cheating, creaming, omission and distraction along with other undesired effects.
Our “Metric Fixation” diminishes the humanity of the organizations that are the very FUTURE of that humanity. We begin to believe that these standardized measurements can replace human judgement.
Big data has its place. It allows us to uncover patterns, counteract bias and uncover incompetence. But we have to use expert human judgement along with data collection.
Featured image:https://unsplash.com/photos/_zsL306fDck
Resources:
The research of Sam Sellar
Yuan and Zhoa, Respect the Power of Large-Scale Assessments: What they cannot measure.
The Manitoba Teachers Society has released documents for teachers and parents to help bring a more balanced and informed view to the PISA process.
Hi Donna, this was interesting and I new about PISA but had not thought about it in some time. You info will help put it all into perspective and calm the waters. Like Ontario’s EQAO, this wide range testing can be done useful in a very broad way but at the school level, not so much. Enjoy your day!
Thank you for this insightful summary. Ever since I learned of the real profiteers of large-scale, standardized testing, I have become much more critical of how and why we collect data … and what we do with it. The trickle-down effect of large-scale testing is often negative, for students and teachers. It can be challenging to keep the focus on learning and not on testing.
This certainly gives us lots to think about. There is so much we can deconstruct and so few who have the desire to do so.