PISA Problem

So I’ve just started working with PISA this semester and it only took me one round of analysis to realize that something is wrong with the data when sorted by region. I kept getting results that showed astounding performance and regional differences that did not make any sense (my study focuses on Internet adoption patterns). I could not figure out why living in Eastern Asia was such a strong predictor of Internet use at times when I knew the West had generally a much better penetration rate (I am not trying to be insensitive or anything here, I just know the numbers and know that there was no way that China had 70% Internet penetration in 2000). My main concern was this: Maybe the data out of Eastern Asia is not representative, thus confounding the study. Of course, I searched think tanks to see if anyone reported on this. Hit number 1 tells the story:

PISA’s China Problem

So I was right. The data only reflect certain areas or provinces, particularly in China. I’ve always read in the media that “China performs near the top” on PISA scores. This article says it all: Only certain areas of China are tested. I wonder what the US PISA scores would look like if we disaggregated the scores by state or region. I’ve heard this argument before, but never had first-hand experience with the data. Journalists have to do a better job writing headlines and reporting what PISA scores actually show us. I know that in my study I am going to have to make explicit the issues that exist in the data when considering regional differences. -b

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s