AI Overreach in Education: Teachers and Students Must Speak Out

Two weeks ago I wrote here about the comments by Instructure's CEO Dan Goldsmith in which we learned that Instructure is now going to be mining student data and using machine learning to create intrusive algorithms, telling students what and when and how to learn: My Soylent-Green Moment: Instructure and the Future of Canvas. Even if Instructure could make good on that promise, I still think it is wrong for them to take and use our data for that project. Just as importantly, I also think that project is doomed to fail, and that it will be squandering valuable resources in the process.

So, in this blog post, I want to say something about both of those topics: first, I will explain why I have zero expectation that Instructure will be able to build useful algorithms; then, I will share an article I read this week about student data-gathering in China that shows what could happen if teachers and students do not speak out now to protest these moves by Instructure and other ed-tech harvesters of student data.

Canvas and the Assumption of Sameness

Anyone who's used Canvas knows that it is committed to an assumption of sameness. You cannot even modify your course menus without installing an external tool that then "redirects" menu items, even when the "redirect" is going to a page in your own course. Moreover, you have to install a new instance of the redirect tool app from the app store for every change you make. 

I have seven separate instances of the redirect tool installed in my Canvas courses (you can see here: Myth.MythFolklore.net), and no matter how much I want to rename the "Grades" menu item there is nothing I can do in order to change its name. When we used D2L I was able to change the name to "Declarations" (which suits my class better), but Canvas adamantly refuses to let me change the menu labels.

Why must everything be the same, named the same, look the same, etc.? Apparently students will be "confused" if things are not exactly the same, or so I've been told again and again by Canvas enthusiasts. That paternalistic, condescending assumption is something that has always bothered me about Canvas; I think it should be up to students and teachers to make choices about what works and what doesn't. Based on my 20 years of teaching experience, I don't think students are so easily confused as Canvas assumes that they are. Learning how to navigate difference is something students need to do in school, and minimizing difference is not necessarily doing anybody a favor. In any case, both students and teachers need freedom to customize their own learning spaces in the ways they think are best. 

But Canvas is all about minimizing difference. Sure, you can use the redirect tool if you want, and you can use other plug-ins and tools to bring external content into Canvas (that is the method I rely on most), but there is an abiding assumption of sameness in Canvas, and there always has been. It's not a bug; it's a feature.

Now, of course, as Instructure launches its data mining and machine learning efforts, it is going to be all the more important to keep things the same. Not so that the students will not be confused, but so that the computer will not be confused. Because if you are going to try to bring together all the data, across schools and across curriculums as Dan Goldsmith claims, then you really need to make sure things are the same across those schools and curriculums. That's what will allow Instructure to combine the data to create "the most comprehensive database on the educational experience in the globe" (again quoting Goldsmith).

But here's the thing: not all courses have the same design. Not all teachers have the same approach. Not all students learn the same way; they are not white lab rats genetically engineered to respond in the exact same way to the exact same stimuli in the exact same conditions, as a computer does. Nothing in human learning is ever exactly the same.

Canvas, however, wants to make everything the same. Sometimes the results of that sameness are just annoying, like having to add the redirect tool app again and again to a course. At other times, though, the results are dangerous, like when Canvas decided to start labeling my students' work, incorrectly, with big red labels in the Gradebook. LATE said Canvas, and MISSING said Canvas, even though my students' assignments are not late and not missing. Here's a real screenshot from a real student's Gradebook display... and none, not one, of these labels is appropriate:

Canvas started adding these labels as part of the new Gradebook. I documented the problem extensively at the Canvas Community back in Fall 2017, at which point they rolled the feature back, but then, like a good zombie, it returned from the dead again this semester when my school made everyone switch to the new Gradebook. Nightmare. I'm not going to repeat all the details here, but you can see my posts with the #red ink tag documenting the story at the Canvas Community (that last post about the future of Canvas is the one that prompted the Community Managers to prohibit my future blogging at the Community).

Suffice to say, Canvas has no idea how my courses are designed, and they have no right to be labeling my students' work that way in the Gradebook. There is no place for punitive, deficit-driven grading in my classes (more about that here), and that means there is no place for Canvas's red labels. Those labels are just wrong, completely and absolutely wrong. And my students were, understandably, upset when the red ink missing and late labels showed up out of nowhere. I don't mind apologizing to students for mistakes that I make, but I sure don't like apologizing to students for mistakes that the computer software is making. Especially when it's computer software that I am required to use by my school for grading purposes because of FERPA.

This is just one example from one teacher, but I know that we could document thousands upon thousands of examples of ways in which individual teachers and students are doing work that Instructure's "machine" simply cannot understand. And because the machine cannot grasp all the different ways we do things, that means it cannot actually learn from what we are doing. Even worse, it will learn wrongly and do the wrong things, like putting all the wrong labels on my students.

In short, it seems to me that anyone who believes in Goldsmith's claims about "the most comprehensive database on the educational experience in the globe" does not really understand what educational experiences (PLURAL) are all about.

From Bad to Worse: Really Big Data in China

If you have not read this horrifying/fascinating report about student surveillance and data-gathering in China, then you need to do that now; yes, it's long, and it's worth reading from start to finish: Camera Above the Classroom. You can find the author at Twitter: @YujieXuett.

Power of hashtags: I love the way the investigation begins with a hashtag: #ThankGodIGraduatedAlready. A hashtag, and also a government plan:

In July 2017, China’s highest governmental body, the State Council, released an ambitious policy initiative called the Next Generation Artificial Intelligence Development Plan (NGAIDP). The 20,000-word blueprint outlines China’s strategy to become the leading AI power in both research and deployment by 2030 by building a domestic AI industry worth nearly $150 billion. It advocates incorporating AI in virtually all aspects of life, including medicine, law, transportation, environmental protection, and what it calls “intelligent education.”

So, in the great data-arms race to have "the most comprehensive database on the educational experience in the globe," Instructure is really going to have to up their game. Just clicks and eyeballs are not going to be enough. Now they are going to need our faces too.

Much like the MOOC-boosters of yesteryear, this data collection is presented as something that is really for the students, for their benefit, for their own good:

“Do you know the two types of students teachers pay the most attention to?” Zhang asks. “The smartest and the naughtiest.” Hanwang’s CCS technology was born from the desire to care for every kid in the classroom, even the “often-ignored, average students,” he adds.

The desire to care. Uh-huh. "Class Care" is the name of the system, CCS. But is this how students want to be cared for? As the article documents, the students are not happy about this surveillance:

Back in the classroom, my questions about the cameras evoke curiosity among the boys. Jason tells them everything he knows. There is a gasp, followed by silence. “I want to smash it,” one boy says. “Shhh!” Another boy shakes a warning glance at Hanwang’s camera behind us. “What if the camera just captured everything?”

Some students are indeed disabling the cameras in their classrooms, despite their justified fears. Brave students: bravo!

Privacy? The Chinese ed-tech vendor Hanwang is using the same excuse that no doubt Instructure will use: privacy is being respected because data is only being used "in-house" and not shared with a third party:

CCS doesn’t violate the students’ privacy. We don’t share the reports with third parties, and you see that on the in-class pictures we send to the parents, all the faces other than their child’s are blurred out.

It begins innocently enough, as you can see in this exchange with a principal at a school that has installed the system:

Niulanshan’s principal, Wang Peidong, who has over 40 years of teaching experience, is also dismissive of CCS. “It’s not very useful,” he says. “You think a teacher standing on a podium needs AI to tell her if a student is sleeping in class?”
“Then why is it still used in your classrooms?” I ask.
“Zhang Haopeng is an alumnus of our school. He wants to do experiments here, so we let him. We don’t need to pay for it anyway,” he says, already walking away to get to his next meeting.

Don't think it can happen here? Think again: check out this eerily similar graphic for a rah-rah-data article in the Chronicle of Higher Education:

Labels in the Gradebook. Labels in the classroom. 

Wrong labels. Punitive labels. 

My sense of despair here is enormous, and I am glad that I will be retired from teaching before we have handed everything of importance over to our new data-driven robo-overlords.

Don't get me wrong. I believe in using data: small data, data that fits the educational experience: The Value of SMALL Data and Microassignments. We need to let the educational experience drive the data first. Otherwise, it's just wrong data. And big wrong data is dangerous data. You cannot let that data drive our education.

For more, read Audrey Watters. And Cathy O'Neil's Weapons of Math Destruction: How big data increases inequality and threatens democracy.

And speak out. Money talks, and money is ultimately what is driving all of these conversations. So, we cannot just talk: we have to shout! I'm going to keep on shouting. I admire the students in China who are disconnecting the surveillance cameras in their classrooms in protest, and I am going to keep on protesting the overreach of AI in Canvas and their (mis)use of my students' data (even if that conversation cannot happen in the Canvas Community itself).

More to come.

The Paradox of Canvas's "Big Data" and Lack of Search

If you haven't read Phil Hill's piece about Instructure's new pursuit of machine learning, start there:
Instructure: Plans to expand beyond Canvas LMS into machine learning and AI

In particular, note Goldsmith's claim about the Instructure database: he says it is "the most comprehensive SaaS database on the educational experience." I'll set aside for now the very depressing view of "educational experience" which Goldsmith is promoting here (but more on that later), and I'll also set aside the claim about "most comprehensive" except to note that it sounds scarily similar to the boasting of Jose Ferreira, Knewton’s founder and former CEO: “We literally know everything about what you know and how you learn best, everything” (more at Marketplace).

What I want to emphasize here is the stark contrast between what Instructure is doing with data for its own purposes and the data it denies to teachers and students who are trying to use Canvas LMS: specifically, the fact that you cannot search the content of a Canvas course.

That's weird, right? 

It's very weird. Read on to find out more.

Seek and Ye Shall Not Find: No Search in Canvas

One of the biggest advantages of digital content is that it is searchable. So, you would think that if teachers go to the trouble of using Canvas tools to create content, then they would be able to search the content they create, and the students would also be able to search the content.

But... you cannot search your content in Canvas. You can create Pages, sure, and you can use the "rich content editor" in order to do that. But you cannot search that content, and your students cannot search the content either. 

Here's what the Pages look like to a student. No search box. No searching.

Here's what the Pages look like to the teacher: again, no search box. Add a new Page? Yes. Edit? Yes. Delete? Yes. Duplicate? Yes. But... Search? Nope. Thou shalt not search.

You have to remember everything yourself. Memorize. With your own brain. Because the Canvas database, big though it may be, is not going to help you here.

Given all the other problems with Canvas Pages (no folders, broken links if you try to rename a page, etc. etc.), I cannot imagine why anyone would actually choose to use Pages to develop content, and I feel really badly for all the teachers who, because of school policies, are required to use Pages for their content.

Back when Canvas was just a scrappy LMS built on a shoestring budget, sure, I guess it sort of maybe kind of made sense that they skipped the search feature of the Pages area. Although even that is still very strange IMO. 

But Instructure's CEO is now making claims about how big their database is and all the data mining they are going to do... while teachers and students still cannot even search their own course content.

Project Khaki Did Not Deliver

This problem was supposed to be fixed after the Project Khaki back in 2017 when search was one of the features voted up and supposedly a priority for engineering resources. (Disclosure: I participated in Project Khaki that year.)

But that Project Khaki commitment led nowhere. The engineers scoped the project as "global search," and then they decided that they did not have the resources available to implement global search. Did they rescope the project? Like maybe the ability to search course content in Canvas Pages? Nope. They did not rescope the search project. It's just... deferred. And is there any timeline about the availability of search coming soon? So far as I know, there is no such timeline. Which means it is now the year 2019, and neither teachers nor students can search their own Canvas Pages to find something they are looking for.

Of course, if you can pay...

Given that search is a real need, perhaps it is not surprising that, yes, there is a third-party vendor, Atomic Jolt, that is willing to provide a Canvas search feature for you. Just look at all the happy Atomic searchers: laughing and smiling. Atomic Search users can search for content, they can save time... except they have to pay more money to do that. 

The existence of this paid service does prove one thing: schools really want to have a search option. So much so that they are willing to pony up additional money to pay another company simply in order to be able to search their course content and allow their students to do the same. (My school does not pay for this service.)

The Google Work-Around

The irony is that you can use a Google work-around: just open up your Canvas course — and, yes, this is my favorite thing about Canvas: you really can open up your course with real URLs, linkable and searchable. Once you do that, Google will be glad to index your content and return results via Google search. So, because I keep all my Canvas spaces fully open, I can use Google as my search engine even if Canvas will not let me search. 

Each course has its own distinctive subdirectory, so I just need to add site:canvas.ou.edu/courses/54178/ to search my Widget Warehouse course site for any term. For example, if I am looking for cats, I can search like this:

cats site:canvas.ou.edu/courses/54178/

You can also do this across a school. So, for example, adding the search delimiter site:canvas.ou.edu/courses to a Google search will search all the open course content at my school. The problem is that very few instructors (how many? I don't know) choose to open up their courses. All the cat results you get this way are still my cats, except that now these are results for all the cats in my Canvas Pages across all my OU Canvas course spaces:

cats site:canvas.ou.edu/courses/

Yes, Google is also extracting its own value from this search to build its advertising empire, but at least it is also returning some value back to me in the ability to perform my own searches.

Instructure, meanwhile, is extracting value from our course content as part of its machine-learning pipe dream, but they are not even letting us perform our own searches of that data.

Plus, Instructure is also missing out on one of the best possible data sources as a result: if they did let us search, they could learn a lot and share what they learn back with us. But instead of letting students navigate-by-search, it is all just click-click-click following the pathways predefined by the instructor: click here, click here next, and so on.

LMS: Undermining Digital Literacy

One last point: I've argued before that the LMS is bad for digital literacy, and this lack of search is a perfect example. For students to become skilled users of digital tools, they need to use real tools, and the lack of search in Canvas shows how it fails as a real web tool. Search is one of the key components of digital literacy, but Canvas doesn't allow students to search, which further means that Canvas does not help students to learn how to search well.

So, while Instructure is busy mining our data supposedly in order to further our education, it is at the same time depriving us of one of the key educational tools that we need.

The year 2018 went by without a search feature in Canvas Pages.

I wonder where we will find ourselves at the end of 2019...?

I promise to update this post if/when news is available.

Meanwhile, you can learn lots more about web literacy, and about search in particular, at this nifty resource from the Mozilla Foundation: Web Literacy: Search. 

Plus there's a Latin LOLCat who knows all about the power of search... and, yes, I found this cat at my Latin LOLCats blog by using the search feature there. :-) 

Quaerendo invenietis.

By seeking you will find.

My Soylent-Green Moment: Instructure and the Future of Canvas

This past week ranks as one of the worst weeks of my professional life: I learned that Instructure is going to be using (is already using?) the data collected about students in Canvas for machine learning and algorithms. I'm still completely shocked. If you haven't read the statements by Instucture CEO Dan Goldsmith in this report by Phil Hill, here is the article:
Instructure: Plans to expand beyond Canvas LMS into machine learning and AI

It's a kind of "Soylent Green" moment for me, realizing that a company and a product in which I had put a lot of faith and trust is going to be pursuing an agenda which I cannot endorse and in which I will not participate.

In this blog post, I'll explain my understanding of the situation, and then close with three main concerns that I have. There will be many more posts to come, and I hope those who know more than I do about machine learning in education will chime in and help me further my own education about this grim topic.

The Now: Canvas Data for Classes and Schools

I've not been impressed by the current Instructure data analytics since their approach is based only on surface behaviors, with no attempt to ask students the "why" for those behaviors (for example, short time spent on content page: because the student is bored? because they are confused? because it was the wrong page? because they have limited time available? because they got distracted by something else? etc.). Yes, Instructure collects a lot of data from students (all those eyeballs! all those clicks!), but just because they have a lot of data does not make it meaningful or useful. Speaking for myself, I get no benefit of any kind from the "Analytics" page for each student in my class that the Canvas LMS wants to show me:

I know that some schools also use the data from Canvas on an institutional level, but that's not something I know a lot about, and I also know there are commercial products, like Dropout Detective, that help schools extend their use of the data in Canvas. Just how a school tracks and uses the data it gathers about its students is for each school to decide.

At my school, for example, there is a strong presumption of student privacy when it comes to enrollment and grading data, as you would expect from FERPA. As an instructor, I use my students' ID numbers to report the students' grades (I am required to do that at the end of the semester, and I am urged to report midsemester grades, but not required), and that is all I can do. I cannot find out what other courses a student is enrolled in or has enrolled in, nor can I find out a student's grades or GPA.

And that is how it should be: it is not my business. Yes, that data exists. And yes, in some cases that data might also be helpful to me in working with a student. But just because the data exists and might be helpful does not mean that I can use it. The student starts with a fundamental right to privacy about their enrollment and grades, and it is up to the school to make decisions about how that data is shared beyond the classroom, like when advisors are able to look at a student's courses and grades overall, or aggregate analysis, like the way the university publicly reports on the aggregate GPA of student athletes, for example.

The Future: Instructure Robo-Tutor in the Sky

So, while my students' performance in their other classes is not my business, Instructure has decided to make it their business. In fact, they have decided to make it the future of their business. Goldsmith is emphatic: the Instructure database is no longer about data reports shared with instructors and with schools. Instead, it is about AI and machine learning. Instructure is going to be using my students' data (my students, your students, all the students) in order to teach its machine to predict what students will do, and then the system will act on those predictions. Quoting Instructure CEO Dan Goldsmith (from Phil's article, and yes, if they do have "the most comprehensive database on the educational experience in the globe," well, that's because we gave them all our data):

Welcome to your worst education nightmare: they are going to lump together all the data across all the schools, predict outcomes, and modify our behavior accordingly... thus sayeth Dan Goldsmith:

In future posts, I'll write in more detail about why this is bound to fail. The hubris here is really alarming; it's as if the executive team at Instructure learned nothing from the costly failures of other edtech machine-learning solutionists during the late, not-great era of the MOOCs. Back in February 2019, Michael Feldstein had speculated that this kind of hype might be subsiding (Is Ed Tech Hype in Remission?), but here we are just a few weeks later, and the hype is strong. Very strong.

Three Concerns

For now, I have three concerns I want to focus on:

1. What exactly did I agree to? To my shame, I put a lot of trust in Instructure, so it is indeed true that I clicked a checkbox somewhere at some point without reading the privacy policy and related legal policies. My students clicked such a checkbox too. At the Instructure website there is a Privacy Policy that relates to personal identifying information (you can access that from the Canvas Dashboard), and once you get to the Instructure website, you can also find an Acceptable Use Policy, but it seems primarily focused on indemnifying Instructure from wrongdoing by users (illegal content, objectionable content, etc.). I'm not a lawyer, but I guess it all hinges on this: "Instructure reserves all rights not granted in the AUP Guidelines." That sounds like they can use all the non-personally-identifying data as delimited in the separate privacy policy in any way they want, is that right?

They do state that they "respect the intellectual property of others and ask that you do too," but it's not clear at all if they regard all the content we create inside the system (assignments submitted, quizzes created and taken, discussion board posts, etc.) as our intellectual property that they should respect and not exploit without our permission. Hopefully someone who knows more than me can figure out how this AUP compares to the kind of terms-of-service that are being used by a company like, say, Coursera, which from the start was committed to machine learning and exploitation of user content in the system.

I don't know what the Coursera terms-of-service looks like now, but back when they first got started, they were very explicit about reusing our content to build their machine-learning system, as I wrote about when I took a first-generation Coursera course back in 2012: Coursera TOS: All your essay are belong to us. See that blog post for language like this: "you grant Coursera and the Participating Institutions a fully transferable, worldwide, perpetual, royalty-free and non-exclusive license to use, distribute, sublicense, reproduce, modify, adapt, publicly perform and publicly display such User Content," etc. I didn't see that kind of language in the Instructure policies, but I'm honestly not sure where to look.

Instructure does have a "Privacy Portal" with a cutesy graphic (visit the page to see the curtain being drawn and clouds of steam arising from behind the shower curtain). I thought the text in bold beside the graphic would be links leading to more information, but they are not links. There's a privacy policy, an acceptable use policy, and a data processing policy linked across the top of the page, but I don't see something labeled "terms of service" like what Coursera had in place. The shower curtain is labeled "privacy shield." Yeah, right.

2. What about opting out? Without an opt-out, Instructure is putting us in an impossible situation, way worse than with TurnItIn, for example. If a student insists that they will not use TurnItIn (as I think every student should do: just say no!), then it's easy to find work-arounds; teachers would just have to read the student's work for themselves without robo-assistance. But if a student says, no, they will not use Canvas because they do not want their data to be exploited for corporate profit, then that puts the teacher in a really awkward position. If you put all your content and course activities and assessments inside Canvas and a student does not want Instructure to use their data, what can the teacher do? It seems to me that Instructure needs, at a minimum, an opt-out for people who do not want their data to be used in this way by our new corporate overlords. Even better: it could all be opt-in, so that instead of assuming students and teachers all want to give their data to Instructure without compensation, you start with the assumption that we do not want to do that, and then Instructure can persuade us to opt in after all.

3. What about FERPA? Right now instructors at my school can put grades in Canvas for institutional reporting purposes (although I actually put mine directly into the SIS instead because the Canvas grading schemes can't accommodate my course design). My school then controls very strictly how that grade data is used, as I explained above. Now, however, it looks like that grade data is something that Instructure is going to be mining, at the course level and at the assignment level, so that its machine-learning engine will track a student's performance both within classes and also from course to course, analyzing their grades and their related data to create the algorithms. To me, that seems like a violation of privacy. In legal terms, perhaps it is not a problem because they are anonymizing the data, but just because it is legal does not make it right. We are apparently giving Instructure extraordinary freedom to take our students' grades and supporting work in order to exploit that not just beyond courses at an institutional level but, as Goldsmith stated (see above), across institutions in ways that will be totally beyond our control. It's like TurnItIn profiting from our students' work (to the tune of 1.7 billion dollars, also in this week's news) without any form of compensation to the students, but way worse. WAY worse. It's not just the students' essays now. It's... everything. Every eyeball. Every click. Teachers and students alike.

Of course, I know Instructure, just like TurnItin, will hire the lawyers they need to make sure they can get away with this. But how sad is that? I never thought I would write a sentence that says "Instructure, just like TurnItIn" ... and yes, I'm angry about it. Angry at Instructure for squandering money, time, and people's trust on what will turn out to be hype rather than reality (but more on that in a separate post). I'm also angry at myself for having put so much trust in Instructure. When I expressed my anger at the Canvas Community this week, I was told that my opinions violated the Community Guidelines which require that everything we post there be "uplifting," so that is why I am back blogging here again after blogging for a couple of years at the Community. I have nothing uplifting to say about the new turn Instructure is taking, and I need a blog space where I am free to say that I am angry about this.

Human Learning

But every cloud (including a SaaS cloud) has a silver lining. I am now going to take my casual layperson's knowledge of machine learning and predictive algorithms in education (mostly gleaned from reading about robograding of student writing) and learn more about that. If the machines are learning, we better get to work on our own learning too! And hey, perfect timing: it's Spring Break and I'll be spending two days in airports. Which means two days of reading.

I'm going to start with Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy by Cathy O'Neil.

Then what should I read next? Let me know here or at Twitter (@OnlineCrsLady).

Update: More on Canvas AI, plus a new weekly datamongering round-up. :-)

Canvas: Monetizing Student Data?

I won't have time to write more about this until next week, but if what Phil Hill has reported about Instructure's latest plans is true, it changes everything about Canvas. My strongly negative opinion about this new development is not welcome at Canvas Community, but I will be free to blog about it here. It's one thing to despair about the limitations and downsides of the LMS as it affects individual teachers and students (as I do), but this is something new, and far worse. So, more to come, but for now, I hope everybody who has an interest in the future of Canvas will take a few minutes to read Phil's important post: Instructure: Plans to expand beyond Canvas LMS into machine learning and AI.

ICYMI: @PhilOnEdTech on Instructure's plans to monetize student data with new AI/machine-learning schemes. TurnItIn has massively profited from reusing student data that schools are supplying; is Canvas going the same way? IMO: it's alarminghttps://t.co/dicpmUzeFX @hypervisible pic.twitter.com/pyS3xvGIY6

— Laura Gibbs (@OnlineCrsLady) March 15, 2019