Pages

May 26, 2019

Data Mongering (6): OU's Bad Data... Bad Bad Data, Very Bad

This is my sixth round-up; you can see them all here: Data-Mongering Round-Ups. I'm also annotating as I go in Diigo, so you can the Diigo files here: Data-Mongering articles. (The editorial comments there in Diigo are just copied-and-pasted from the blog posts.)

If the data can be faked, they will be faked

Last week, I shared a picture of a device to game the steps-data on a cellphone that's being monitored by your employer and/or health insurance company. So, in that same spirit of fakery, my lead story this week is about my very own school, the University of Oklahoma: Oklahoma Gave False Data for Years to 'U.S. News,' Loses Ranking in Inside Higher Ed, University of Oklahoma gave false data to U.S. News college rankings for 20 years in CNN, etc. Here's the graphic from Inside Higher Ed:



Also, in a bit of Google Search hilarity, one of the first-page search results for "University of Oklahoma fake data" is... OU's own Academic Integrity Council. And, seriously, it's a hard sell telling students they should not game their test grades and other life-determining numbers when they see OU doing the same.


Related: check out Peter Greene's blog post earlier this year on Goodhart's Law And The BS Test: quote Goodhart was an economist and critic of Margaret Thatcher's policies, which led him to this observation circa 1975: Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. In 1997, Strathern, an anthropologist, translated that from economistese into punchy English: When a measure becomes a target, it ceases to be a good measure.


Also, there's a great write-up on the whole US News rankings scheme in Cathy O'Neil's Weapons of Math Destruction. Yes, the rankings meet her definition of a WMD: If you look at this development from the perspective of a university president, it’s actually quite sad. Most of these people no doubt cherished their own college experience—that’s part of what motivated them to climb the academic ladder. Yet here they were at the summit of their careers dedicating enormous energy toward boosting performance in fifteen areas defined by a group of journalists at a second-tier newsmagazine. They were almost like students again, angling for good grades from a taskmaster. In fact, they were trapped by a rigid model, a WMD.

Bill Gates strikes again!

Can You Measure the Value of College? New Gates Foundation Commission Will Try by Jeff Young for EdSurge. Yes, the man who gave us VAM is back with another datamongering plan: instead of just ranking schools as in the US News scam, now we are going to measure the value of college. I'm sure there is a WMD coming our way from this initiative! quote The Bill & Melinda Gates Foundation has been funding higher education projects at colleges for more than a decade, with mixed success, but this week it announced a fresh approach, with the formation of a new commission charged with creating a framework for measuring the value of each college in the U.S. Their goal is answering the age-old question: What’s a college degree really worth? For a warning about the perils of such a project, the article quotes Phil Hill: quote Foundations, particularly Gates, are so wedded to hard data and the usage of that data that they have blind spots to things that may not fit into their view because they don’t have data on it,” he said in an interview. That, he added, could result in a “simplistic framework.”

For a sense of what this kind of data might look like, check out Kevin Carey's latest in the New York Times: Can Data Ward Off College Debt? There’s a Push to Show Which Majors Pay Off. This article is about Department of Education data based on tracking graduates by program, not just by school. Yes, instead of just using data to pit one school against another, we can now pit one program against another at the same school! For example: quote "data published by the state’s higher education coordinating body reveals large differences within the university. Some University of Virginia majors earn more than $70,000 or $80,000 three years after graduating, while others are in the $35,000 to $50,000 range. University of Virginia systems engineers, for example, make almost double what environmental science majors earn."

Sadly, with my adjunct wages, I am making UC Berkeley look bad. The "value" of my Ph.D. is barely equal to the average value of a freshly minted B.A. from George Mason University. Is that really true? The numbers say so, so it must be true. Alas, poor Berkeley!



LMSes chasing big data

But hey, it's not just about schools and their programs; the LMSes have got their own data too... and lots of it, now that they are pooling data across schools in pursuit of AI/ML breakthroughs (Soylent Canvas). So, here's some important reporting by Justin Menard: EdTech Companies With The Most Student Data. Of the LMSes, Instructure is definitely out in front... but it's interesting to think about Google and Microsoft in this regard, providing email and other services as they do (my school is a Microsoft school):



I keep hoping for some kind of data policy statement from Instructure to clarify the issues that are naturally arising as they pursue these new AI/ML projects, and I've been told there might be something in the Instructure blog or maybe the Canvas blog next week.

Don't forget the publishers...

Textbook publishers are making themselves into serious players in the datamongering business also, and in their pursuit of scale and big data for data-mining, of course they want to strike campus-wide deals; you can read about that here: Colleges Are Striking Bulk Deals With Textbook Publishers. Critics Say There Are Many Downsides by Jeff Young again at EdSurge. There are lots of concerns expressed from different perspectives, including this statement from SPARC: quote That’s one argument made by the Scholarly Publishing and Academic Resources Coalition, or SPARC, which advocates for lower textbook costs and OER. “There is ... no mechanism preventing publishers from resuming their historical rate of price increases once the model becomes widespread,” the group said in a recent analysis of the changing landscape for course materials.

Although it's not about data-mongering, I also want to mention here an OER-related discussion I've been following about Knowledge Unlatched. You could call it OER-mongering. For a good summary of the situation, see these statements from Open Book Publishers and OPERAS. Plus a Twitter thread from Punctum Books:



And for a closing image...

Check out this item from Tim Maughan at Twitter: how creepy is that? Seriously creepy.


And here's something really cool: I had snagged this tweet from Tim Maughan not knowing who he was, only to find out that he is the author of Infinite Detail, a book that was recommended to me by multiple people in multiple conversations at Twitter over the past week. That is clearly a sign that I need to read his book... and no, I didn't need badges and an engagement algorithm to reach this conclusion. :-)




May 18, 2019

Data Mongering (5): Dashboards Eat the World

This is my fifth round-up; you can see them all here: Data-Mongering Round-Ups. I'm also annotating as I go in Diigo, so you can the Diigo files here: Data-Mongering articles. (The editorial comments there in Diigo are just copied-and-pasted from the blog posts.)

DISQUANTIFIED: Higher Education in the Age of Metrics. There was an amazing speaker line-up at this conference on May 16-17 in Santa Barbara, and I'm hoping we will see materials from these talks appearing online soon. See the detailed abstracts of the presentations at the website, along with lots of materials for a related Reading Group. In the Twitter traffic, I was intrigued by this screenshot from Ben Williamson's presentation, Student Counting: Assembling the Data Infrastructure of Higher Education. I'm calling this "dashboards eat the world." Thanks to Phil Hill for the photo:


Audrey Watters gave one of the keynotes at Disquantified, and she has been writing on datamongering in education for a long time now; if people don't know, she's collected all those fabulous keynotes into a series of books: The Monsters of Education Technology, The Revenge of the Monsters of Education Technology, The Curse of the Monsters of Education Technology, and The Monsters of Education Technology 4. With pigeons, of course:


And on the subject of dashboards eating the world, big news this week was the SAT's new adversity score: "The new adversity score is meant to be one such gauge. It is part of a larger rating system called the Environmental Context Dashboard that the College Board will include in test results it reports to schools" as reported in SAT’s New ‘Adversity Score’ Will Take Students’ Hardships Into Account by Anemona Hartocollis in the NYTimes plus in many other media outlets. This is a perfect example of datamongering since it is, ultimately, all about the SAT trying to stay competitive in the college admissions marketplace. One of the things that I find mind-boggling and scary about this new score is that it will be reported to colleges, and students will not even be able to see that data being reported to colleges about them. There are so many problems here, and I will not even try to unpack them all. Thomas Chatterton Williams's editorial in the NYTimes, The SAT’s Bogus ‘Adversity Score,’  focuses on the pseudoscientific faux accuracy of rating something on a scale of 1-100, for example. 

(Not directly on the subject of datamongering, but I was really intrigued by this remark about software engineering in this ASCD EducationUpdate piece, Are Grades Reliable? Lessons from a Century of Research by Susan M. Brookhart and Thomas R. Guskey: quote "The current resurgence of the 100-point percentage grade scale has been attributed to the use of computerized grading programs that typically are developed by software engineers who have scant knowledge of this history of their unreliability.")

The AI Supply Chain Runs on Ignorance by Sidney Fussell in the Atlantic. You can read all about Ever photo service here, along with the larger problems of surveillance and data mongering; quote "Companies can mine data to be scraped and used later, with the original user base having no clue what the ultimate purpose down the line is. In fact, companies themselves may not actually know who’ll buy the data later, and for what purpose, so they bake vague permissions into their terms of service." This expresses my concerns about what could happen to our Instructure data down the line. It's not just about privacy (i.e. Instructure removing personally identifiable information), but instead about the products that will be created with that data. I have strong personal beliefs about what is and is not ethical in education, but after Instructure obtains data about me from surveilling my use of their software, I have no control over what kinds of products they will create, or products they will create in conjunction with their partners. Those products might include plagiarism police, robograders, tutorbots, etc., and I do not want to support creation of such products with my data or data harvested from my students. So, I keep hoping for a data re-use opt-out (more on that here).

How Much Artificial Intelligence Should There Be in the Classroom? by Betsy Corcoran and Jeffrey R. Young in EdSurge reporting on a conference about AI in education organized by a Chinese company called Squirrel AI, including an interview with Derek Li, one of Squirrel's co-founders. This quote from pretty much sums it up: "And he believes that having AI-driven tutors or instructors will help them each get the individual approach they need. He closed his remarks by saying that he “hopes to provide each child with a super-power AI teacher that is the combination of Einstein and Socrates, that way it feels like she has more than 100 teachers in front of her.”" Yeah, right. There is nothing in the article that makes it seem like Squirrel AI actually understands the world of teaching and learning, and it was disappointing to see it written up in EdSurge without any actual facts or details that would lend some credibility to this pie-in-the-sky AI hype. Instead, for a really detailed report on a different company in China and the Chinese government's endorsement of massive AI experiments in education, see this important article: Camera Above the Classroom by Yujie Xue.

And speaking of pie-in-the-sky hype, let's not forget about Knewton. Here's some reporting on its demise: End of the Line for Much-Hyped Tech Company by Lindsay McKenzie in InsideHigherEd. quote "The educational technology company, which famously boasted about the power of its adaptive learning platform to “semi-read” students’ minds, has been acquired by publisher Wiley Education." And a quote from Phil Hill in the article: "It’s a fire sale. In the press release, they don’t say they’re buying Knewton the company, they say they’re acquiring its assets -- they don’t even try and sugarcoat it." See also Tony Wan's write-up in EdSurge: Wiley to Acquire Knewton’s Assets, Marking an End to an Expensive Startup Journey with Michael Feldstein's "snake-oil" description of Knewton's hype, as reported a few years ago in NPR: Meet The Mind-Reading Robo Tutor In The Sky: quote "But wait. Learning is a lot more complicated than algorithmic reading of multiple-choice responses. What about a student's level of motivation or persistence or, say, the empathy, passion and insight of a good teacher or dozens of other factors? "He is overselling the kind of power that Knewton can do," says Michael Feldstein, a digital education consultant and co-founder of the ed blog e-Literate. "I would go so far as to say that he is selling snake oil." Claims about the wonders of tech to revolutionize learning, Feldstein argues, vastly oversimplify the complexity, beauty and mystery of how humans learn."

And for more on algorithms and the companies who sell them, Civitas Learning has a new CEO: Chris Hester. Although the website claims the company is "built by educators for education," Hester is not an educator. He comes to Civitas from the health care industry, "twenty years of experience leading companies that create social impact through the application of technology and analytics." My school is a Civitas customer (or so I've read), but faculty have never received any information about that partnership, so I actually have no idea what data services they provide for us, or at what cost.

Finally, for the "if-it-can-be-gamed-it-will-be-gamed" files (and in the field of health care surveillance in fact), here's some data gaming for you (via Twitter):