CEO Dan Goldsmith published a post at the Instructure blog on January 20 (online here), but that post fails to address the concerns resulting from his own claims about Instructure AI projects last March (online here). In the post, Goldsmith repeated an announcement made last week that Melissa Loble is now "Chief Customer Experience Officer" who will be the "executive sponsor" for "data usage and privacy." I think we can safely assume that this appointment is in reaction to customer complaints about Instructure's data policies, but you would not know that from reading Goldsmith's post. He refuses to acknowledge our complaints, much less engage in a dialogue about our concerns.
What are we to make of the fact that Goldsmith has stopped talking about DIG publicly after his hyperbolic claims last year? As I see it, Goldsmith's silence (and the ongoing silence of others at Instructure about DIG) is making things worse, not better. I suspect (just guessing) that Goldsmith realized speaking publicly about these plans would make research and development more difficult, as users were objecting to Instructure's unilateral appropriation of our data for their own commercial purposes. He said as much in his letter about the Thoma Bravo acquisition (online here): "Reflecting this year on our goals and path forward revealed that operating in the public spotlight wasn’t fueling innovation and was starting to get in the way of customer success." I have no idea what it means to claim that sharing information about data usage was getting in the way of "customer success," but then the rhetoric of "customer success" is really just marcomm-speak which is hindering, not helping, an honest dialogue about what Instructure is doing with our data.
When Jared Stein published a blog post about DIG last summer (online here), I replied here at my blog (online here), hoping for a dialogue. We never heard from Jared again with any further information about DIG, which is also disappointing. There is so much that needs to be discussed in order for all parties to have a clear understanding of each other's goals, constraints, concerns, etc.
Everyone knows I'm prone to long blog posts, and I made sure then to write a post that was no longer than Jared's post; I'm doing the same again here. I've got 295 words left to equal Goldsmith's post, so here's a quick recap of my three biggest concerns:
1. Data Opt-Out. In addition to privacy issues in play, there are serious ethical concerns about the development of AI products and predictive algorithms in education; for a good discussion, see Michael Feldstein on A/B testing and product development (online here). Those of us who do not want our data used for AI research and development need an opt-out.
2. FERPA. I still do not understand how it is not a violation of FERPA for Instructure to use student grade data and enrollment status (i.e. profiling across courses) without the students' express permission. As an instructor, I am not allowed to see my students' grades in other classes, nor their GPA. I suspect (?) such data is crucial to Goldsmith's predictive algorithm for student performance: "We can predict, to a pretty high accuracy, what a likely outcome for a student in a course is, even before they set foot in the classroom" (online here).
If they are predicting performance before students even begin work for a class, grades in other courses must surely (?) be a big part of that predictive algorithm. I suspect many students would object to having their performance predicted in this way based on Instructure's unilateral appropriation of their grade data. Update. If not the letter of FERPA, then the spirit; see Twitter convo.
3. Dialogue. If CEO Goldsmith can't/won't engage in dialogue with users, then I hope we will be hearing soon from others at Instructure who are willing and able to do that.
If you have questions and concerns, I would urge you to add them to the GoogleDoc that Cristina Colquhoun (@call_hoon) has created here: Questions for Instructure. I'm very grateful to Cristina for the excellent job she has done of organizing this latest effort to get Instructure to respond to our concerns.
I'm taking a Zuboff break this week because I need to document my ongoing battle with Canvas Gradebook labels. The new semester has started, and as of midnight Tuesday, I've been fighting with Canvas to get control of my Gradebook. I think the Gradebook is my space, but Canvas insists on intruding. Yes, it's the labels. If you don't know what I mean, they look like this:
Yep, that would be red ink all over the Gradebook. Here's the story:
Unlike other LMSes I have used, Canvas does not respect the Gradebook space as belonging to teachers and students. Instead, Canvas thinks it knows better than teachers and students what's going on in a class. "MISSING" says the Gradebook in angry red letters (even when the assignment was optional), and "LATE" says the Gradebook (even when the student turned the work in before the deadline). By means of these labels, Canvas is sending negative and incorrect messages to my students.
So, if anyone is curious why it is that I have zero trust in Instructure's use of student data for machine-learning and AI, this is why: Canvas is intruding into the Gradebook with wrong messages for my students... and sending wrong messages to students about grades is just about the worst thing that can happen in a class. It's hard work to turn the Gradebook into a positive, rather than a negative space (my approach: Grading.MythFolklore.net), and Canvas then pulls the rug right out from under me. I tell the students they are in control... but Canvas then tells them the opposite: MISSING shouts Canvas (even when the work is not missing) and LATE shouts Canvas (even when the work is not late).
Does Canvas have any positive messages to send my students? Nope. Nothing but red ink. MISSING. LATE. Over and over again. And I cannot stop it.
It's like waking up in the morning to find that someone has thrown garbage on your front lawn.
Luckily, James Jones (the API and scripting guru of the Canvas Community) has written a script that will go pick up the garbage; you can see how he did that here: Removing Missing and Late Labels. Because Canvas built the Gradebook without any course-level control over the labels, the script checks every single assignment item for every single student, adjusting the label data item by item, student by student. Because I use a microassignments approach, that means the script has to check 18,450 records each time, and it does so quickly. Yay for James! Yay for scripts!
But here's the thing: James's script cannot stop the LATE labels from appearing; the LATE labels show up no matter what, and I cannot stop the students from seeing those labels. So I apologize to the students for the incorrect LATE labels and ask them to just ignore them; then I run the script once a week to clear them out.
Picking up the trash off our lawn.
The trash that Canvas put there.
It's not like Instructure doesn't know about this problem. When they first rolled out the red labels in the Beta version of the new Gradebook in September 2017, I documented the problem in great detail at the Canvas Community; that link goes to my blog posts tagged "red ink" and the first one is called "Gradebook Dismay," dated September 9, 2017. I was not the only one who was upset to find Canvas putting labels on my students, and Instructure rolled back that Beta feature from the Gradebook. I was sure they would fix it when we were all forced to go to the new Gradebook, which at my school happened in Spring 2019.
But I was wrong.
When Spring 2019 began, there were the labels in the Gradebook, just like before. I contacted Canvas support and found out there was nothing I could do about it; I could not disable the labels at the Gradebook level. I could not disable the labels at the Assignment level. I could not change the wording of the labels or the color or alter the algorithm that assigned the labels. All I could do was click on the 18,450 items in my Gradebooks one by one.
So, as I said, James Jones came to my rescue and wrote a script.
But is that really a solution? My guess is that most Canvas users are not going to want to copy a script from GitHub, configure the variables manually in the script, and then run that script in the Javascript Console of their browser separately for each class. And to do that week after week. Yes, it's amazingly cool how it works, and I personally love to watch the network performance monitor go blip-blip-blip as it checks on thousands of records at lightning speed. But I'm a nerd, and you shouldn't have to be a nerd to stop Canvas from putting labels on your students. Especially when those labels are completely inaccurate.
And now, let's talk about why the labels are inaccurate, because that reveals a lot about how the people at Instructure view student learning: Instructure is applying an old-fashioned, deficit-driven approach to education, an approach that is exactly the opposite of what we need in the year 2019 IMO.
What is LATE? Before the new Gradebook, Canvas had a great approach to the late problem: they let you have a soft deadline and a hard deadline. This used to be one of my favorite features of Canvas. The soft deadline is what I tell my students to aim for; such-and-such is due on Tuesday (and I set the soft deadline to Tuesday midnight). But does it really matter if students are finishing up something at midnight as opposed to 2AM? No, that's silly. My students are not Cinderellas riding in pumpkin carriages; midnight is totally arbitrary. So I set up the soft deadline, and then I give everybody a 12-hour extension for every assignment, no questions asked; that is the hard deadline, and it is set for noon the next day (so, noon on Wednesday for an assignment due Tuesday). I call it the grace period. If students make the hard deadline, that is GREAT. That is the whole point; they got the assignment turned in my the deadline: yay! But Canvas does not think so: nope, Canvas thinks the work is late, putting that punitive red label on every assignment turned in during that grace period. I call it a no-questions-asked extension, but with those red LATE labels, Canvas undermines my message. Your teacher may tell you it's okay to use the grace period, but we at Canvas know better: a good student should not need the grace period, and you are not a good student; your work was LATE. I gave the students an extension on purpose, and I want them to use the extension if that helps them to get the work done. But Canvas doesn't care about what I want or what my students need. Canvas is just going to apply its algorithm, using the mind of a machine and trampling our humanity. LATE. LATE. LATE. LATE. As if the students who struggle with time don't already beat themselves up enough as it is, Canvas is going to beat them up some more. I say: students should be praised for getting the work turned in, not shamed.
What is MISSING? So, those Late labels are pretty bad, but brace yourselves: the MISSING labels are even worse. The way Canvas applies the MISSING label means that you cannot let students choose what assignments to do. I repeat: Canvas will not let you make assignments optional. So, if you think that student choice is important (I do!), and if you want to design your course so that students choose what assignments to complete (I do!), then you better learn how to run James's script because Canvas is going to label every assignment that your students choose not to complete as MISSING. And it is going to freak your students out, understandably. That is how I first found out about the labels back in September 2017; one afternoon I started getting panicked emails from students. "You told us that we could choose what assignments to do, but now Canvas is telling me I have to do them all!" I was baffled; how could Canvas tell my students what to do or what not to do? I didn't even understand what the students were talking about because I had no idea Canvas had started putting labels in my Gradebook. But it's true: Canvas really was telling my students that they had missed assignments. Even though the assignments were not required. Of course my students were upset. And here we are, almost two years later, and the Canvas Gradebook still wants to put MISSING labels on all those student assignments. The only thing that saves me is James's magic script.
What is the point of labels anyway? Even if these labels were correct (and they are not correct in my classes; every single label Canvas applies in my Gradebook is incorrect), these labels are still not going to help students. So, this is not just about Laura-and-her-weird-classes. This is instead about a wrong approach to feedback at Instructure. Students need encouraging, actionable feedback to motivate them to improve their performance. They need to know what they got right, and they also need to know what they can work on in order to do better for next time.
The Late label fails because it disregards the fact that the student DID turn in the work, which is actually good! But instead of praising the student for getting the work turned in, the red label conveys the message "no, you did bad." Negative messages like that are not how you encourage students to do better the next time.
And the Missing label is worse: it sends a negative message, and it is not even clear what the student is supposed to do next. Are they supposed to complete the missing work and turn it in anyway? Or not? Different teachers have different approaches to missing work (if the work really is missing), but Canvas doesn't know that. And Canvas doesn't care. If Canvas cared about that, they would let us configure the labels in our own way, based on our own algorithms, and conveying our own messages to our students.
About uplift. I'll add one last observation here, and that is about what it means to be "uplifting." I used to be an active member of the Canvas Community, and my last blog post at the Community was about the Gradebook labels, along with my criticisms of Instructure's claims about AI and predictive algorithms. If they can't get the Gradebook right, why should I trust them to get anything else right about student data? At the time, the Community Managers told me I could no longer write posts like that; all contributions to the Community must be uplifting in nature, so say the Community Guidelines. Fair enough: it's their space; they make the rules, and they don't want me complaining in their space. Because I was not willing to self-censor my posts in order to be uplifting all the time, I started blogging again here; that was back in March of this year.
So, what about the Gradebook? Who makes the rules there? Just like Canvas wants an uplifting Community, I want an uplifting Gradebook. Those punitive red labels are NOT uplifting to my students, and I want them out of my Gradebook; those are my Gradebook Guidelines, and Canvas should respect that. The Gradebook belongs to me and my students. It is our space, and we should be able to tell Canvas to get its negative messages out of our space.
And so, in 2000 words (tl;dr I know), that is why I have zero faith in Instructure's ability to do anything useful with data analytics. The devil is in the details, and the details about the Canvas Gradebook are not pretty.
That's it for this week, but I'll be back with more Zuboff again next time. And I'm glad to say that, aside from the Gradebook labels, my classes are going great! The blog network is up and running; I'll be writing about our adventures at Twitter: @OnlineCrsLady. Happy New Semester, everybody!
This week Jared Stein wrote a blog post about Canvas data, Power to the People with Canvas Data and Analytics (Can You Dig It?). I'm glad that a conversation is happening, and I have a lot to say in response, especially about an opt-out for those of us who don't want to be part of the AI/machine-learning project, and a shut-off so that we can stop intrusive profiling, labeling, and nudging in our Canvas classes. It's not clear from Jared's post just what kind of opt-out and shut-off control we will have, and I hope we will hear more about that in future posts. Also, since Jared does not detail any specific Dig projects, I am relying on Phil Hill's reporting from InstructureConwhich describes one such project: profiling a student across courses, including past courses, and using that comprehensive course data to predict and manage their behavior in a current course. (This use of grade data across courses without the student's express consent sure looks like a violation of FERPA to me, but I'll leave that to the lawyers.)
And now, some thoughts:
1. Not everyone digs it. I understand that some people see value in Dig predictive analytics, and maybe they are even passionate about it as Jared says in his post, but there are also people whose passions run in different directions. As I explain below, my passion is for data that emerges in actual dialogue with students, so it is imperative that I be able to stop intrusive, impersonal auto-nudges of the sort that Dig will apparently be generating. The punitive red labels in the Canvas Gradebook are already a big problem for me (my students' work is NOT missing, and it is NOT late, despite all the labels to the contrary). Based on the failure of the Gradebook algorithms in my classes, I do not want even more algorithms undermining the work I do to establish good communication and mutual trust. So, I really hope Instructure will learn a lesson from those Gradebook labels: instructors need to be able to turn off features that are unwelcome and inappropriate for their classes. Ideally, Instructure would give that power directly to the students, or let teachers choose to do so; that's what I would choose. My students voted by a large majority to turn off the labels (which I now do manually, week by week, using a javascript), although a few students would have wanted to keep the labels. I say: let the students decide. And for crying out loud, let them choose the color too; the labels don't need to be red, do they?
2. We need to target school deficits, not student deficits. I believe that Instructure's focus on at-risk students comes from good intentions, but that cannot be our only focus. Instead, we need data to help us focus on our own failures, the deficits in our own courses: deficits in design, content, activities, feedback, assessment, etc., along with data about obstacles that students face beyond the classroom. This is a huge and incredibly important topic, way too big for this blog post, so I hope everybody might take the time to read some more about the perils of deficit-driven thinking. A few places to start:
For a great example of what happens when you invite students to talk about the obstacles they face, see this item by Peg Grafwallner: How I Helped My Students Assess Their Own Writing. Applying that approach to Canvas: instead of labeling students with red ink in the Gradebook ("you messed up!") and then auto-nudging them based on those labels ("don't mess up again!"), the labels could be more like a "what happened?" button, prompting a dialogue where the student could let the instructor know the reason(s) why they missed an assignment or did poorly, etc., and the instructor could then work with the student to find a positive step forward, based on what the student has told them. That is the way I would like to see data-gathering happen: student-initiated, in context and in dialogue.
3. Dig is not just about privacy; it is about Instructure's unilateral appropriation of our data. Jared emphasizes in his post that Instructure is not selling or sharing our data, but there is more at stake here than just data privacy and data sharing. Instructure is using our data to engage in AI experiments, and they have not obtained our permission to do that; I have not consented, and would not give my consent if asked. Dan Goldsmith has stated that users "own" their data, and one of the data tenets announced at InstructureCon was "Empower People, Don't Define Them" (discussed here). Speaking for myself, as someone who does not want to be defined by Instructure's profiling and predictive algorithms, I need to be able to just opt out. In his post, Jared writes about Instructure being a "good partner" in education, "ensuring our client institutions are empowered to use data appropriately." Well, here's the thing about partnerships: they need to work both ways. It's not just about what Instructure empowers institutions to do; it is also about what we, both as institutions and as individuals, empower Instructure to do. By unilaterally appropriating our data for their own experiments, Instructure is not being a good partner to individual students and teachers. If the people at Instructure "see no value or benefit in technology that replaces teachers or sacrifices students' agency" as Jared says in his post, then I hope they will give us the agency we need to opt out of Dig data-mining and remove our courses from the Dig laboratory.
Okay, everybody knows my blog posts are always too long, so I'll stop here to keep this about the same length as Jared's post (and of course I've written a lot about these topics here at the blog already). I hope this conversation can continue, and I also hope that Instructure will explore the options for both data opt-out and feature shut-off as they proceed with the Dig rollout for 2020. Thanks for reading!
Last week, I did a round-up post focused on InstructureCon, summarizing my many concerns about Instructure's new AI experiments. Back in March, CEO Dan Goldsmith announced a big shift for Instructure: instead of just giving teachers and schools access to data for traditional statistics as in the past, Instructure itself would be analyzing our students, profiling them in order to create predictive algorithms for future business growth, while doubling their TAM as Goldsmith claimed:
InstructureCon updates on DIG
So, after InstructureCon we know a lot more about this AI project, called DIG. For example, Goldsmith now claims: We can predict, to a pretty high accuracy, what a likely outcome for a student in a course is, even before they set foot in the classroom.
Personally, I find this claim hard to believe, given that the only data Instructure has to work with is the isolated, low-level data they gather from Canvas activity: log-ins, page views, quizzes, gradebook, etc. Unizin schools add demographics to that Canvas data (which I find even more alarming), but it sounds like Goldsmith is making the claim about Canvas data itself.
In any case, speaking for myself, I do not want Instructure to tell me how to do my job ("we can make recommendations..."), prejudicing my views of students before I have even met them. My school currently does not share a student's GPA with me, and for good reason; as I see it, Instructure's labeling of students in this way is no different than sharing their GPA. In fact, I would suspect that past grade data is a very significant component in Instructure's prediction engine, perhaps even the most significant component. But hey, it's their proprietary AI; I'm just guessing how it might work, which is all we can do with corporate AI/ML experiments.
Notice also the slipperiness of the word "outcome" in Goldsmith's claims about predictive accuracy. When teachers think about outcomes, we are thinking about what students learn, i.e. the learning they can take away with them from the class (what comes out of the class), especially the learning that will be useful to them in their later lives. And that's very complex; there is a whole range of things that each student might learn, directly and indirectly, from a class, and at the time of the class there's no telling what direction their lives might take afterwards and what might turn out to be useful learning along that life path. But the LMS has no record of those real learning outcomes. In fact, the LMS has no real measures of learning at all; there are only measures of performance: performance on a quiz, performance on a test, attendance, etc. So when Goldsmith talks about predicting the "likely outcome" for a student, what I suspect he means is that Instructure is able to predict the likely final grade that the student will receive at the end of a class (which is why I suspect GPA would be a big component in that prediction). But the grade is not the learning, and it is not the only outcome of a class. In fact, I would argue that we should not be using grades at all, but that is a topic for a separate discussion.
What about a data opt-out?
So, now that we know more about the goals of DIG, what about opting out? There was no announcement about an opt-out, and no mention even of the possibility of an opt-out. Goldsmith even claimed in an interview that there hasn't been any request for an opt-out: "We haven’t had that request, honestly."
Well, that claim doesn't make sense as I myself had a long phone conversation with two VPs at Instructure about my opt-out request. What Goldsmith must mean, I suppose, is that they have not had a request at the institutional level for any campus-wide opt-outs, which is not surprising at all. While it would be great if we had some institutional support for our preferences as individual users, I would be very surprised if whole institutions decide to opt out. Predictive analytics serve the needs of institutions far more than they do the needs of individual teachers or students, and I can imagine that institutions might be eager to see how they can use predictive analytics to look at school-wide patterns that are otherwise hard to discern. Teachers can grok what is going on in their individual classrooms far more easily than provosts and deans can grok what is going on across hundreds and thousands of classrooms.
Yet... there is hope!
Yet I still have some hope for an opt-out, because I learned from that same Goldsmith interview that individuals OWN their data: One of our first and primary tenets is that the student, the individual and the institution own the data—that’s their asset.
This concession about data ownership really caught me by surprise, in a good way, and renewed my hope for an opt-out. If individuals own their data, then we should be able to take our data out of the Instructure cloud when a course is over if we choose to do so. In other words: a data opt-out, perhaps with the same procedure that Instructure already uses to sunset data from schools that terminate their Instructure contract.
In fact, in the context of ownership, it really sounds more like an opt-in is required. If Instructure wants to use my data — data about me, my behavior, my work, my OWN data — then they should ask me for my permission. They should ask for permission regarding specific timeframes (a year, or two years, or in perpetuity, etc.), and they should ask for permission regarding specific uses. For example, while I strongly object to AI/ML experiments, there might be other research to which I would not object, such as a study of the impact that OER has on student course completion. Not all data uses are the same, so different permissions would be required.
Of course, as I've said before, I am not optimistic that Instructure is going to implement an opt-in procedure — even though they should— but I am also not giving up hope for a data opt-out, especially given the newly announced Canvas data tenets.
Canvas Data Tenets
In addition to this surprising concession about data ownership, we learned about these new Canvas data tenets at InstructureCon. In the video interview cited above, Goldsmith promised a post about data tenets coming soon at the Instructure blog, and there was already this slide in circulation at InstructureCon, which I assume are the data tenets Goldsmith is referring to in the interview (strangely, even the Instructure staff keynotes were not livestreamed this year, so I am just relying on Twitter for this information). As you can see, one of those tenets is: Empower People, don't Define Them.
Now, the language here sounds more like marcomm-speak rather than the legal or technical language I would expect, but even so, I am going to take heart from this statement. If Instructure promises to empower me, then surely they will provide a data opt-out, right? It would not be empowering if Instructure were to take my Canvas data and use it for an experiment to which I do not consent, as is currently the case.
My Canvas Data Doubts
Meanwhile, that tension between empowering people, not defining them, is what I want to focus on in the final part of this blog post. I saw really mixed messages from InstructureCon this year, as the big keynotes from Malcolm Gladwell, Dan Heath, and Bettina Love were all about community, peak moments, love and creativity... with a corporate counterpoint of big data and a billion Canvas quizzes as I learned via Twitter:
See also the contradiction where Goldsmith claims in an interview that Instructure is all about "understanding the individuals, their paths, their passions, and what their interests are" and what we see in the data dashboards: there are no passions and interests on those dashboards (but I do know those red "missing" labels all too well):
Impersonal personalization
There's a single word that I think expresses this dangerous ambivalence in ed-tech generally, and at Instucture in particular; that word is personalization. On the one hand, personalization looks like it would be about persons (personal agency, personal interactions, personal passions) but personalization has also become a codeword for the automation of education. Both in terms of philosophy and pedagogy, automation sounds really bad... but personalization: ah, that sounds better, doesn't it?
So, for example, listen to what Dan Goldsmith says in this interview: it's technology inevitablism, literally. (video here) So when you think about adaptive and personalized learning I think it's inevitable that we as an educational community need to figure out ways of driving more personalized learning and personalized growth experiences.
I'm not going to rehash here all the problems with the rhetoric of personalization; Audrey Watters has done that for us, as in this keynote (among others): Pigeons and Personalization: The Histories of Personalized Learning. (A good all-purpose rule for thinking about ed tech: READ AUDREY.)
Instead, I will just focus here on the impersonality of Canvas data, listing five big reasons why I mistrust that data and Instructure's claims about it:
1. Canvas data measure behavior, not learning. Canvas is an environment that monitors student behavior: log on, log off; click here, click there; take this quiz, take that quiz; this this many words, download this many files, etc. If your educational philosophy is based on behaviorism, then you might find that data useful (but not necessarily; see next item in this list). If, however, your educational philosophy is instead founded on other principles, then this behavioral data is not going to be very useful. And consider the keynote speakers at InstructureCon: none of them was advocating behaviorism; just the opposite. Here's Bettina Love, for example, on liberation, not behaviorism (more on her great work below):
2. Canvas fails to gather data about the why. Even for purposes of behavior modification, that superficial Canvas data will not be enough; you need to know the "why" behind that behavior. If a student doesn't log on to Canvas for a week, you need to know why. If a student clicks on a page but spends very little time there, you need to know why. If a student does poorly on a quiz, you need to know why. For example, if a student got a poor score on a quiz because of a lack of sleep that is very different from getting a poor score because they did not understand the content, which is in turn very different from being bored, or being distracted by problems at home, etc. Just because students completed a billion quizzes in Canvas does not mean Instructure has all the data it needs for accurately profiling those students, much less for making predictions about them.
3. Canvas data are not human presence. The keynote speakers consistently emphasized the importance of people, presence, relationships, and community in learning, but numbers are not presence. Does this look like a person to you? This is how Canvas represents a student to me right now; the coming data dashboard (see above) uses the same numbers repackaged, because that is all that Canvas has to offer me: numbers turned into different kinds of visualizations.
Goldsmith claims that Instructure is different from other learning companies because they are all about people's passions and interests, but that claim does not fit with the views I get of my students in the Canvas Dashboard and the Canvas Gradebook: no passions, no interests; just numbers. I don't need percentage grades, much less the faux-precision of two decimal points. Instead, I need to know about students' passions and interests; that would be exactly the information that will help me do my job well, but Canvas data cannot provide that data.
4. Canvas data does not reflect student agency. The basic pedagogical design of Canvas is top-down and teacher-directed. Student choice is not a driving principle; in fact, it is really a struggle to build courses based on student choice (I will spare you the gory detail of my own struggles in that regard). Students cannot even ask questions in the form of search; yes, that's right: students cannot search the course content. The only access to the course content is through the click-here-click-there navigation paths pre-determined by the instructor. And, sad to say, there is apparently no fix in sight for this lack of search; as far as I could determine, there was no announcement regarding the deferred search project from Project Khaki back in 2017 (details here). Think about that lack of search for just a minute. It's no accident that Google started out as a search engine; the questions that people brought to Google, and people's choices in response to those answers, generated the behavioral surplus juggernaut that now powers Google AI. Netflix succeeds as a prediction engine precisely because it is driven by user choice: lots of options, lots of choices, and lots of data about those choices with which to build the prediction engine. The way that Canvas forestalls student choice, including the simple ability to initiate a search, is why I believe their AI project is going to fail. (Meanwhile, if I am wrong and there was an announcement about Canvas search at InstructureCon, let me know!)
And this last item is actually the most important:
5. Canvas data cannot measure obstacles to student learning. By focusing data collection on the students, Instructure runs the risk of neglecting the social, political, and economic contexts in which student learning takes place. Whether students succeed or fail in school is not simply the result of their own efforts; instead, there are opportunities and obstacles, not evenly distributed, which are crucially important. Does Canvas data record when students are hungry or homeless or without health insurance? Does Canvas data record that a course is taught by a poorly paid adjunct with no job security? As Dave Paunesku wrote in Ed Week this week, "When data reveal students' shortcomings without revealing the shortcomings of the systems intended to serve them, it becomes easier to treat students as deficient and harder to recognize how those systems must be changed to create more equitable opportunities." I hope everybody will take a few minutes to read the whole article: The Deficit Lens of the 'Achievement Gap' Needs to Be Flipped. Here's How. (Short answer: another billion quizzes is not how you flip the deficit lens.)
Of course, this is all a topic for a book, not a blog post, so I'll stop for now... but I'll be back next week to start a new approach to these datamongering round-ups: a commentary on Shoshana Zuboff's Surveillance Capitalism. Of all the concepts in play here, the one that is most important to me is what Zuboff's calls our "right to the future tense." So, I will work through her book chapter by chapter in the coming weeks, and hopefully that will make it more clear just why it is that I object so strongly to Instructure's predictive analytics experiment.
~ ~ ~
I want to close here with Bettina Love's TED talk; take a look/listen and see what you think: I think she is brilliant! More also at her website.
Speaking for myself, I'll take dance and lyrics over data analytics any day. So, keep on dancing, people! And I'll be back next week with Shoshana Zuboff's book and our right to the future tense. :-)
I began this #datamongering project back in March when I first learned of Instructure's plans to exploit existing user data to create predictive algorithms built with AI and machine learning: Soylent Canvas. I am still hoping that those of us who are opposed to the use of predictive algorithms in education will be able to OPT OUT so that Instructure will not be able to use our data to develop its algorithms and train its AI system (even better would be opt-in, but I don't actually have any hope for that one).
I submitted a question about data opt-out to the InstructureCon Engineering panel (my question). Coincidentally (?), Instructure then published a blog post about its privacy policy, so I reiterated that my question is not about privacy; it is about opting out of Instructure's plans to mine my data, all our data, for machine learning (my follow-up):
In talking to people about this, I've found that many educators are still not really sure just what AI and predictive algorithms mean for education, how LMS companies do data mining, what the difference is between machine learning and traditional statistical analysis, etc. etc. Over the past five months, I've been collecting online materials on these topics, so in this special "InstructureCon Edition" of my #datamongering round-ups, I've listed what I see as some of the most valuable resources people can use to learn more. Read on:
For more on ed tech companies and their data, see also: EdTech Companies With The Most Student Data by Justin Menard.
2. Despite Dan Goldsmith's claims about Instructure's database, there is nowhere near enough data in Canvas to model real learning by real students. What kind of surveillance will be required to get the actual data required? China has a Next Generation Artificial Intelligence Development Plan (NGAIDP) that is bringing full-scale student surveillance to the classroom; there is detailed reporting here from Xue Yujie: Camera Above the Classroom. If you are going to read just one article on AI in education, this is the one to read.
As you can see, if the LMS does not give students the opportunity to opt out, things get very complicated as Bryan learned when he opted out on his own. This is why Instructure needs to give individuals more control over who is allowed to use their data and for what purposes.
4. Increasing surveillance of students is an issue of great concern for both higher ed and for K-12. On K-12, see this important piece by Benjamin Herold in EdWeek: Schools Are Deploying Massive Digital Surveillance Systems. The Results Are Alarming.
And if anybody is at the Instructure Engineering panel at InstructureCon on Thursday (July 11) 4:20PM in the Long Beach Convention Center, GB-B, please let me know if they say anything about a data opt-out. I don't know if my question will make the cut or not... but I have not given up hope yet.
Two weeks ago I wrote here about the comments by Instructure's CEO Dan Goldsmith in which we learned that Instructure is now going to be mining student data and using machine learning to create intrusive algorithms, telling students what and when and how to learn: My Soylent-Green Moment: Instructure and the Future of Canvas. Even if Instructure could make good on that promise, I still think it is wrong for them to take and use our data for that project. Just as importantly, I also think that project is doomed to fail, and that it will be squandering valuable resources in the process.
So, in this blog post, I want to say something about both of those topics: first, I will explain why I have zero expectation that Instructure will be able to build useful algorithms; then, I will share an article I read this week about student data-gathering in China that shows what could happen if teachers and students do not speak out now to protest these moves by Instructure and other ed-tech harvesters of student data.
Canvas and the Assumption of Sameness
Anyone who's used Canvas knows that it is committed to an assumption of sameness. You cannot even modify your course menus without installing an external tool that then "redirects" menu items, even when the "redirect" is going to a page in your own course. Moreover, you have to install a new instance of the redirect tool app from the app store for every change you make.
I have seven separate instances of the redirect tool installed in my Canvas courses (you can see here: Myth.MythFolklore.net), and no matter how much I want to rename the "Grades" menu item there is nothing I can do in order to change its name. When we used D2L I was able to change the name to "Declarations" (which suits my class better), but Canvas adamantly refuses to let me change the menu labels.
Why must everything be the same, named the same, look the same, etc.? Apparently students will be "confused" if things are not exactly the same, or so I've been told again and again by Canvas enthusiasts. That paternalistic, condescending assumption is something that has always bothered me about Canvas; I think it should be up to students and teachers to make choices about what works and what doesn't. Based on my 20 years of teaching experience, I don't think students are so easily confused as Canvas assumes that they are. Learning how to navigate difference is something students need to do in school, and minimizing difference is not necessarily doing anybody a favor. In any case, both students and teachers need freedom to customize their own learning spaces in the ways they think are best.
But Canvas is all about minimizing difference. Sure, you can use the redirect tool if you want, and you can use other plug-ins and tools to bring external content into Canvas (that is the method I rely on most), but there is an abiding assumption of sameness in Canvas, and there always has been. It's not a bug; it's a feature.
Now, of course, as Instructure launches its data mining and machine learning efforts, it is going to be all the more important to keep things the same. Not so that the students will not be confused, but so that the computer will not be confused. Because if you are going to try to bring together all the data, across schools and across curriculums as Dan Goldsmith claims, then you really need to make sure things are the same across those schools and curriculums. That's what will allow Instructure to combine the data to create "the most comprehensive database on the educational experience in the globe" (again quoting Goldsmith).
But here's the thing: not all courses have the same design. Not all teachers have the same approach. Not all students learn the same way; they are not white lab rats genetically engineered to respond in the exact same way to the exact same stimuli in the exact same conditions, as a computer does. Nothing in human learning is ever exactly the same.
Canvas, however, wants to make everything the same. Sometimes the results of that sameness are just annoying, like having to add the redirect tool app again and again to a course. At other times, though, the results are dangerous, like when Canvas decided to start labeling my students' work, incorrectly, with big red labels in the Gradebook. LATE said Canvas, and MISSING said Canvas, even though my students' assignments are not late and not missing. Here's a real screenshot from a real student's Gradebook display... and none, not one, of these labels is appropriate:
Canvas started adding these labels as part of the new Gradebook. I documented the problem extensively at the Canvas Community back in Fall 2017, at which point they rolled the feature back, but then, like a good zombie, it returned from the dead again this semester when my school made everyone switch to the new Gradebook. Nightmare. I'm not going to repeat all the details here, but you can see my posts with the #red ink tag documenting the story at the Canvas Community (that last post about the future of Canvas is the one that prompted the Community Managers to prohibit my future blogging at the Community).
Suffice to say, Canvas has no idea how my courses are designed, and they have no right to be labeling my students' work that way in the Gradebook. There is no place for punitive, deficit-driven grading in my classes (more about that here), and that means there is no place for Canvas's red labels. Those labels are just wrong, completely and absolutely wrong. And my students were, understandably, upset when the red ink missing and late labels showed up out of nowhere. I don't mind apologizing to students for mistakes that I make, but I sure don't like apologizing to students for mistakes that the computer software is making. Especially when it's computer software that I am required to use by my school for grading purposes because of FERPA.
This is just one example from one teacher, but I know that we could document thousands upon thousands of examples of ways in which individual teachers and students are doing work that Instructure's "machine" simply cannot understand. And because the machine cannot grasp all the different ways we do things, that means it cannot actually learn from what we are doing. Even worse, it will learn wrongly and do the wrong things, like putting all the wrong labels on my students.
In short, it seems to me that anyone who believes in Goldsmith's claims about "the most comprehensive database on the educational experience in the globe" does not really understand what educational experiences (PLURAL) are all about.
From Bad to Worse: Really Big Data in China
If you have not read this horrifying/fascinating report about student surveillance and data-gathering in China, then you need to do that now; yes, it's long, and it's worth reading from start to finish: Camera Above the Classroom. You can find the author at Twitter:@YujieXuett.
Power of hashtags: I love the way the investigation begins with a hashtag: #ThankGodIGraduatedAlready. A hashtag, and also a government plan:
In July 2017, China’s highest governmental body, the State Council, released an ambitious policy initiative called the Next Generation Artificial Intelligence Development Plan (NGAIDP). The 20,000-word blueprint outlines China’s strategy to become the leading AI power in both research and deployment by 2030 by building a domestic AI industry worth nearly $150 billion. It advocates incorporating AI in virtually all aspects of life, including medicine, law, transportation, environmental protection, and what it calls “intelligent education.”
So, in the great data-arms race to have "the most comprehensive database on the educational experience in the globe," Instructure is really going to have to up their game. Just clicks and eyeballs are not going to be enough. Now they are going to need our faces too.
Much like the MOOC-boosters of yesteryear, this data collection is presented as something that is really for the students, for their benefit, for their own good:
“Do you know the two types of students teachers pay the most attention to?” Zhang asks. “The smartest and the naughtiest.” Hanwang’s CCS technology was born from the desire to care for every kid in the classroom, even the “often-ignored, average students,” he adds.
The desire to care. Uh-huh. "Class Care" is the name of the system, CCS. But is this how students want to be cared for? As the article documents, the students are not happy about this surveillance:
Back in the classroom, my questions about the cameras evoke curiosity among the boys. Jason tells them everything he knows. There is a gasp, followed by silence. “I want to smash it,” one boy says. “Shhh!” Another boy shakes a warning glance at Hanwang’s camera behind us. “What if the camera just captured everything?”
Some students are indeed disabling the cameras in their classrooms, despite their justified fears. Brave students: bravo!
Privacy? The Chinese ed-tech vendor Hanwang is using the same excuse that no doubt Instructure will use: privacy is being respected because data is only being used "in-house" and not shared with a third party:
CCS doesn’t violate the students’ privacy. We don’t share the reports with third parties, and you see that on the in-class pictures we send to the parents, all the faces other than their child’s are blurred out.
It begins innocently enough, as you can see in this exchange with a principal at a school that has installed the system:
Niulanshan’s principal, Wang Peidong, who has over 40 years of teaching experience, is also dismissive of CCS. “It’s not very useful,” he says. “You think a teacher standing on a podium needs AI to tell her if a student is sleeping in class?”
“Then why is it still used in your classrooms?” I ask.
“Zhang Haopeng is an alumnus of our school. He wants to do experiments here, so we let him. We don’t need to pay for it anyway,” he says, already walking away to get to his next meeting.
Don't think it can happen here? Think again: check out this eerily similar graphic for a rah-rah-data article in the Chronicle of Higher Education:
Labels in the Gradebook. Labels in the classroom.
Wrong labels. Punitive labels.
My sense of despair here is enormous, and I am glad that I will be retired from teaching before we have handed everything of importance over to our new data-driven robo-overlords.
Don't get me wrong. I believe in using data: small data, data that fits the educational experience: The Value of SMALL Data and Microassignments. We need to let the educational experience drive the data first. Otherwise, it's just wrong data. And big wrong data is dangerous data. You cannot let that data drive our education.
And speak out. Money talks, and money is ultimately what is driving all of these conversations. So, we cannot just talk: we have to shout! I'm going to keep on shouting. I admire the students in China who are disconnecting the surveillance cameras in their classrooms in protest, and I am going to keep on protesting the overreach of AI in Canvas and their (mis)use of my students' data (even if that conversation cannot happen in the Canvas Community itself).
In particular, note Goldsmith's claim about the Instructure database: he says it is "the most comprehensive SaaS database on the educational experience." I'll set aside for now the very depressing view of "educational experience" which Goldsmith is promoting here (but more on that later), and I'll also set aside the claim about "most comprehensive" except to note that it sounds scarily similar to the boasting of Jose Ferreira, Knewton’s founder and former CEO: “We literally know everything about what you know and how you learn best, everything” (more at Marketplace).
What I want to emphasize here is the stark contrast between what Instructure is doing with data for its own purposes and the data it denies to teachers and students who are trying to use Canvas LMS: specifically, the fact that you cannot search the content of a Canvas course.
That's weird, right?
It's very weird. Read on to find out more.
Seek and Ye Shall Not Find: No Search in Canvas
One of the biggest advantages of digital content is that it is searchable. So, you would think that if teachers go to the trouble of using Canvas tools to create content, then they would be able to search the content they create, and the students would also be able to search the content.
But... you cannot search your content in Canvas. You can create Pages, sure, and you can use the "rich content editor" in order to do that. But you cannot search that content, and your students cannot search the content either.
Here's what the Pages look like to a student. No search box. No searching.
Here's what the Pages look like to the teacher: again, no search box. Add a new Page? Yes. Edit? Yes. Delete? Yes. Duplicate? Yes. But... Search? Nope. Thou shalt not search.
You have to remember everything yourself. Memorize. With your own brain. Because the Canvas database, big though it may be, is not going to help you here.
Given all the other problems with Canvas Pages (no folders, broken links if you try to rename a page, etc. etc.), I cannot imagine why anyone would actually choose to use Pages to develop content, and I feel really badly for all the teachers who, because of school policies, are required to use Pages for their content.
Back when Canvas was just a scrappy LMS built on a shoestring budget, sure, I guess it sort of maybe kind of made sense that they skipped the search feature of the Pages area. Although even that is still very strange IMO.
But Instructure's CEO is now making claims about how big their database is and all the data mining they are going to do... while teachers and students still cannot even search their own course content.
Project Khaki Did Not Deliver
This problem was supposed to be fixed after the Project Khaki back in 2017 when search was one of the features voted up and supposedly a priority for engineering resources. (Disclosure: I participated in Project Khaki that year.)
But that Project Khaki commitment led nowhere. The engineers scoped the project as "global search," and then they decided that they did not have the resources available to implement global search. Did they rescope the project? Like maybe the ability to search course content in Canvas Pages? Nope. They did not rescope the search project. It's just... deferred. And is there any timeline about the availability of search coming soon? So far as I know, there is no such timeline. Which means it is now the year 2019, and neither teachers nor students can search their own Canvas Pages to find something they are looking for.
Of course, if you can pay...
Given that search is a real need, perhaps it is not surprising that, yes, there is a third-party vendor, Atomic Jolt, that is willing to provide a Canvas search feature for you. Just look at all the happy Atomic searchers: laughing and smiling. Atomic Search users can search for content, they can save time... except they have to pay more money to do that.
The existence of this paid service does prove one thing: schools really want to have a search option. So much so that they are willing to pony up additional money to pay another company simply in order to be able to search their course content and allow their students to do the same. (My school does not pay for this service.)
The Google Work-Around
The irony is that you can use a Google work-around: just open up your Canvas course — and, yes, this is my favorite thing about Canvas: you really can open up your course with real URLs, linkable and searchable. Once you do that, Google will be glad to index your content and return results via Google search. So, because I keep all my Canvas spaces fully open, I can use Google as my search engine even if Canvas will not let me search.
Each course has its own distinctive subdirectory, so I just need to add site:canvas.ou.edu/courses/54178/ to search my Widget Warehouse course site for any term. For example, if I am looking for cats, I can search like this:
You can also do this across a school. So, for example, adding the search delimiter site:canvas.ou.edu/courses to a Google search will search all the open course content at my school. The problem is that very few instructors (how many? I don't know) choose to open up their courses. All the cat results you get this way are still my cats, except that now these are results for all the cats in my Canvas Pages across all my OU Canvas course spaces:
Yes, Google is also extracting its own value from this search to build its advertising empire, but at least it is also returning some value back to me in the ability to perform my own searches.
Instructure, meanwhile, is extracting value from our course content as part of its machine-learning pipe dream, but they are not even letting us perform our own searches of that data.
Plus, Instructure is also missing out on one of the best possible data sources as a result: if they did let us search, they could learn a lot and share what they learn back with us. But instead of letting students navigate-by-search, it is all just click-click-click following the pathways predefined by the instructor: click here, click here next, and so on.
LMS: Undermining Digital Literacy
One last point: I've argued before that the LMS is bad for digital literacy, and this lack of search is a perfect example. For students to become skilled users of digital tools, they need to use real tools, and the lack of search in Canvas shows how it fails as a real web tool. Search is one of the key components of digital literacy, but Canvas doesn't allow students to search, which further means that Canvas does not help students to learn how to search well.
So, while Instructure is busy mining our data supposedly in order to further our education, it is at the same time depriving us of one of the key educational tools that we need.
The year 2018 went by without a search feature in Canvas Pages.
I wonder where we will find ourselves at the end of 2019...?
I promise to update this post if/when news is available.
Meanwhile, you can learn lots more about web literacy, and about search in particular, at this nifty resource from the Mozilla Foundation: Web Literacy: Search.
Plus there's a Latin LOLCat who knows all about the power of search... and, yes, I found this cat at my Latin LOLCats blog by using the search feature there. :-)
