The latest PISA results are about to be released to the public. How will the media spin the latest results? What outrage will surface? What drastic policy shifts will happen as a result? What pressure will be put on front line educators because of how their jurisdiction fares in the rankings?
Why are we obsessed with quantifying and ranking student performance? According to Jerry Z. Muller, in his book “Tyranny of Metrics“, there are two problems with this kind of thinking.
First, it isn’t true that anything that can be measured can be improved. Learning is a complex outcome, yet standardized tests like PISA measure only the relatively simple. So many important learnings are not being measured, like innovation, initiative, creativity, cooperation and mentoring.
According to Yuan and Zhao, what they cannot measure includes the following.
- They can’t measure what they are not intended to measure. In other words, we can’t overgeneralize the results – “the results of PISA have been generalized to reflect the quality of educational systems although they simply reflect how well 15-year-olds in different systems perform on the PISA assessments in math, reading, and science.”
- They can’t measure what we don’t know or understand. Since Large Scale Assessments don’t predict success, there must be aspects of success we are not measuring with tests.
- They cannot measure exceptionalities beyond the limits of the test.
- Skills and learning exist within contexts and cultures. They are fluid and dynamic and situation-dependent.
- Every individual has a jagged profile of abilities. They are not the sum of a few data points.
It is particularly troubling when the people making up the test are also the people profiting from the sales of the remediation solution when the jurisdiction fails to achieve the desired success. When we strip away the context for the learning, results can appear to be more certain than they really are.
Secondly, people don’t always respond to incentives in the way we think they will. Ranking scores presents a distorted view of human motivation. Most people are already trying to do well. Pressure to do better on a test can lead to cheating, creaming, omission and distraction along with other undesired effects.
Our “Metric Fixation” diminishes the humanity of the organizations that are the very FUTURE of that humanity. We begin to believe that these standardized measurements can replace human judgement.
Big data has its place. It allows us to uncover patterns, counteract bias and uncover incompetence. But we have to use expert human judgement along with data collection.
The research of Sam Sellar
The Manitoba Teachers Society has released documents for teachers and parents to help bring a more balanced and informed view to the PISA process.