|  ElCiudadano.cl |  Noticias

El Ciudadano

Original article: Lo que PISA dice y lo que le hacemos decir

By Dr. Jorge Miranda Ossandón, Faculty of Education, U. Católica de Temuco

Every three years, just two days after the results are released, the phone rings. On the other end is a friendly journalist with little time and a single question: «Professor, what place did we finish in?»

The honest answer is uncomfortable because the rank is the least reliable figure in the entire report.

It is important to understand what a PISA score represents before analyzing whether it has risen or fallen. No student answers the complete test. The items are distributed across rotated booklets, and each student responds to a different fraction (never the same as their seatmate), and what is eventually published is not actual scores but statistical estimates of how the total population would perform if everyone had answered everything.

In fact, a Chilean student’s score does not exist. There is only a probabilistic description of fifteen-year-olds in Chile as a group. It is a tool designed to depict countries, not individuals, and it works well as long as that is what is being asked.

The problem arises when we ask for precision down to the decimal. In 2014, Svend Kreiner, a statistician from the University of Copenhagen, demonstrated that the model PISA uses to convert responses into scores does not fully fit reading data, allowing for different subsets of questions, all valid from a technical standpoint, resulting in a country potentially shifting dozens of places in the ranking.

Subsequent discussions ensued. John Jerrim and his team refined this finding in 2018: averages resist changes in the model rather well, although they are not entirely immune to it. The practical conclusion from that academic debate is what never makes the headlines. The average holds steady. The rank does not.

Chile placed 37th out of 90 in reading, 43rd out of 91 in science, and 61st out of 90 in mathematics. These three numbers circulated widely this week, and all three are among the most fragile in the report.

Between Chile and Uruguay, there are two points in mathematics and three in science—differences that easily fit within the margin of error. Saying one outperforms the other is merely reading noise. In reading, however, the Chilean advantage holds; there is a difference of thirteen points.

There is something worse than the fragility of the rank, which is the trap of regional comparison.

PISA evaluates fifteen-year-olds who are still in the school system. It does not measure those who have left. In the 2022 cycle, Chile had about 86% of its adolescents in that age group within the sample; Uruguay had 85%; Argentina, 84%. Guatemala included 48%. Panama had 58%. Mexico had 64%.

When we claim that Chile leads Latin America, we are comparing a group that nearly all participated against several groups where half were absent, and where those who weren’t present are predictably the ones who would have performed the worst. I say this with discomfort because I too have shown that data in a presentation, with a map of the region and Chile on top, without mentioning coverage. It’s the kind of omission one commits in good faith that still distorts the conversation.

Then there’s what PISA simply cannot answer by design. It does not allow for regional comparisons: the Chilean sample for this cycle comprised 6,574 students from 222 schools, a number intended to represent the entire country and not to compare regions like La Araucanía with Antofagasta.

Any regional table circulating has such wide margins of error that differences dissolve. It also reveals nothing about a specific school, a particular student, or an individual teacher.

And there’s an absence that is particularly painful in this region. PISA does not record affiliation with indigenous peoples in Chile. A large portion of the school enrollment in La Araucanía is Mapuche (58,610 students, around 28.5%), and regarding them, the country’s most cited international tool has absolutely nothing to say.

It’s not that the results are bad or good; it’s that those students, for the purposes of this measurement, are statistically invisible.

To that, we must add everything the test does not measure because it was never intended to. Democratic coexistence, citizenship education, artistic expression, intercultural competence, proficiency in an indigenous language, the ability to carry out a project—none of this generates comparable figures every three years, which is why none of it occupies space in the public discussion about educational quality.

We end up talking about what can be counted instead of what we agreed was important and was expressed in various institutional educational projects.

Nevertheless, it would be a mistake to use these objections to dismiss the tool. PISA is the best technically comparative tool we have, and there are things it states firmly without relying on the scaling model or the rank in the table. There are three main points, and none of these were headline news.

First: 59% of fifteen-year-old Chilean youth do not reach the minimum level in mathematics. In other words, in a class of forty, twenty-four do not achieve the proposed learning outcomes. This percentage does not change even if one alters the subset of items or adjusts for coverage; it is too large to be a mere measurement artifact.

Second, the floor has widened. Since 2015, the proportion below that threshold grew ten percentage points in mathematics and another ten in reading. This is a sustained trend over three cycles, not just a post-pandemic stumble.

The third point is perhaps the most serious and the least mentioned. Between 1% and 2% of Chilean students reach the highest performance levels in each area, compared to 7% or 8% in OECD countries. In mathematics, the Chilean figure is practically zero.

We do not have a problem of lagging behind with a group of excellence pulling the average up. We have an entire distribution shifted downwards, flattened, with no upper tail. A country obsessively discussing its average yet never its ceiling is looking at only half of the chart.

Therein lies, I believe, the difference between using the data and obeying it. The rank in the table is an anxious figure that lends itself to headlines and does not support any decision-making. The shape of the distribution is hard, uncomfortable, and actionable information, which has been available for years without anyone converting it into effective policy for improving classroom learning.

I suspect part of the problem is that the first question fits into a headline while the second does not. But I also think that asking about the rank allows for a concluding conversation in thirty seconds, whereas asking about the shape forces one to stay engaged.

The next time the phone rings, I’ll try to answer differently. I have the impression I won’t get published.

(*) Dr. Jorge Miranda Ossandón is the lead researcher of the Fondecyt Initiation project 11240073

La entrada Understanding PISA Results: What They Really Mean for Chilean Education se publicó primero en El Ciudadano.