Data Mongering (6): OU's Bad Data... Bad Bad Data, Very Bad

This is my sixth round-up; you can see them all here: Data-Mongering Round-Ups. I'm also annotating as I go in Diigo, so you can the Diigo files here: Data-Mongering articles. (The editorial comments there in Diigo are just copied-and-pasted from the blog posts.)

If the data can be faked, they will be faked

Last week, I shared a picture of a device to game the steps-data on a cellphone that's being monitored by your employer and/or health insurance company. So, in that same spirit of fakery, my lead story this week is about my very own school, the University of Oklahoma: Oklahoma Gave False Data for Years to 'U.S. News,' Loses Ranking in Inside Higher Ed, University of Oklahoma gave false data to U.S. News college rankings for 20 years in CNN, etc. Here's the graphic from Inside Higher Ed:

Also, in a bit of Google Search hilarity, one of the first-page search results for "University of Oklahoma fake data" is... OU's own Academic Integrity Council. And, seriously, it's a hard sell telling students they should not game their test grades and other life-determining numbers when they see OU doing the same.

Related: check out Peter Greene's blog post earlier this year on Goodhart's Law And The BS Test: quote Goodhart was an economist and critic of Margaret Thatcher's policies, which led him to this observation circa 1975: Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. In 1997, Strathern, an anthropologist, translated that from economistese into punchy English: When a measure becomes a target, it ceases to be a good measure.

Also, there's a great write-up on the whole US News rankings scheme in Cathy O'Neil's Weapons of Math Destruction. Yes, the rankings meet her definition of a WMD: If you look at this development from the perspective of a university president, it’s actually quite sad. Most of these people no doubt cherished their own college experience—that’s part of what motivated them to climb the academic ladder. Yet here they were at the summit of their careers dedicating enormous energy toward boosting performance in fifteen areas defined by a group of journalists at a second-tier newsmagazine. They were almost like students again, angling for good grades from a taskmaster. In fact, they were trapped by a rigid model, a WMD.

Bill Gates strikes again!

Can You Measure the Value of College? New Gates Foundation Commission Will Try by Jeff Young for EdSurge. Yes, the man who gave us VAM is back with another datamongering plan: instead of just ranking schools as in the US News scam, now we are going to measure the value of college. I'm sure there is a WMD coming our way from this initiative! quote The Bill & Melinda Gates Foundation has been funding higher education projects at colleges for more than a decade, with mixed success, but this week it announced a fresh approach, with the formation of a new commission charged with creating a framework for measuring the value of each college in the U.S. Their goal is answering the age-old question: What’s a college degree really worth? For a warning about the perils of such a project, the article quotes Phil Hill: quote Foundations, particularly Gates, are so wedded to hard data and the usage of that data that they have blind spots to things that may not fit into their view because they don’t have data on it,” he said in an interview. That, he added, could result in a “simplistic framework.”

For a sense of what this kind of data might look like, check out Kevin Carey's latest in the New York Times: Can Data Ward Off College Debt? There’s a Push to Show Which Majors Pay Off. This article is about Department of Education data based on tracking graduates by program, not just by school. Yes, instead of just using data to pit one school against another, we can now pit one program against another at the same school! For example: quote "data published by the state’s higher education coordinating body reveals large differences within the university. Some University of Virginia majors earn more than $70,000 or $80,000 three years after graduating, while others are in the $35,000 to $50,000 range. University of Virginia systems engineers, for example, make almost double what environmental science majors earn."

Sadly, with my adjunct wages, I am making UC Berkeley look bad. The "value" of my Ph.D. is barely equal to the average value of a freshly minted B.A. from George Mason University. Is that really true? The numbers say so, so it must be true. Alas, poor Berkeley!

LMSes chasing big data

But hey, it's not just about schools and their programs; the LMSes have got their own data too... and lots of it, now that they are pooling data across schools in pursuit of AI/ML breakthroughs (Soylent Canvas). So, here's some important reporting by Justin Menard: EdTech Companies With The Most Student Data. Of the LMSes, Instructure is definitely out in front... but it's interesting to think about Google and Microsoft in this regard, providing email and other services as they do (my school is a Microsoft school):

I keep hoping for some kind of data policy statement from Instructure to clarify the issues that are naturally arising as they pursue these new AI/ML projects, and I've been told there might be something in the Instructure blog or maybe the Canvas blog next week.

Don't forget the publishers...

Textbook publishers are making themselves into serious players in the datamongering business also, and in their pursuit of scale and big data for data-mining, of course they want to strike campus-wide deals; you can read about that here: Colleges Are Striking Bulk Deals With Textbook Publishers. Critics Say There Are Many Downsides by Jeff Young again at EdSurge. There are lots of concerns expressed from different perspectives, including this statement from SPARC: quote That’s one argument made by the Scholarly Publishing and Academic Resources Coalition, or SPARC, which advocates for lower textbook costs and OER. “There is ... no mechanism preventing publishers from resuming their historical rate of price increases once the model becomes widespread,” the group said in a recent analysis of the changing landscape for course materials.

Although it's not about data-mongering, I also want to mention here an OER-related discussion I've been following about Knowledge Unlatched. You could call it OER-mongering. For a good summary of the situation, see these statements from Open Book Publishers and OPERAS. Plus a Twitter thread from Punctum Books:

And for a closing image...

Check out this item from Tim Maughan at Twitter: how creepy is that? Seriously creepy.

And here's something really cool: I had snagged this tweet from Tim Maughan not knowing who he was, only to find out that he is the author of Infinite Detail, a book that was recommended to me by multiple people in multiple conversations at Twitter over the past week. That is clearly a sign that I need to read his book... and no, I didn't need badges and an engagement algorithm to reach this conclusion. :-)