Hello everyone. I'm Dan Hickey. I'm here with Joshua Quick from the Learning Sciences program. We're going to talk about a course feature and some analytics that's currently underway. First of course, I'm going to say thanks to the CLASS for supporting this work, and George Rehrey for all his support over the years. Say thanks to some awesome graduate students and colleagues, particularly Suraj Uttamchandani who was instrumental in the work we're going to be talking about today. So our work has been deeply informed by the work of the late RandI Engle, in key respects, we've really picked up where she left off when she passed away unfortunately in 2012. This notion of productive disciplinary engagement is probably one of the most useful and used or set of design principles to come out of situative theories of learning. More recently we've embraced, or the last thing she published before she died was this notion of expansive framing. It's about pushing students to find connections with people, places, topics, and times. The goal with all of our work is to position students as authors rather than consumers of disciplinary knowledge and to hold them accountable to disciplinary discourse. So the research context we're going to talk about is a course that I've been teaching for 10 years online. It's really been the test bed for many of the ideas in my work in this framework we call participatory learning and assessment. What are the key things about this course? Well the instructor is pretty busy, I travel a lot in the summer. Sometimes, and I often teach this course in a compressed schedule, which means it's hard for me sometimes to, to get to class for several days at a time. So it's very important that the class sort of almost run itself when it needs to. I spend almost all of my time providing public feedback to the students and very, very little time engaging in private grading and private work. The point here is that the amount of time I'm able to spend is spent positioning students as authors in the course. One of the real goals of a lot of this work goes is avoiding instructor burnout by minimizing grading and private feedback. We do this using the thing we call Wikifolios, these are public artifacts of the class. All of the work happens either on these artifacts and then unthreaded discussions and student pages in Canvas. This is actually a new discussion posts students post each week. So much of our work involves looking at the threaded comments that students post on their work. What we're going to focus on today is a new feature that we added, and I'll provide the rationale. Each assignment includes a private self-assessment. Each assignment concludes with public reflections, these are most important part of our talk today. These reflections are summative for engagement and we can use them for awarding completion points, but they're formative in that they shape engagement proleptically, because students know that they're going to have to reflect on the contextual, collaborative, consequential, and conceptual engagement. These serve to further position students as authors. These are posted publicly. I'm able to comment on those reflections. There's also an exam that automatically automated and time limited to speed things along even more. Here are the four reflection prompt that we had in 2018. Now, what I want to talk about today, and one of the things I want to talk about today is recent critique of the PDE framework by Agarwal and Sengupta-Irving. They can see the PDE might help group-based inequities, compared to traditional curriculum, like problematizing content from your own perspectives allows for cultural meaning, explanation of that content. But they said that doing so without attending to power and privilege probably won't make any difference because of the way the minority students get positioned out of discourse in classrooms. Particularly they argued the problematizing content in ways that challenge culturally dominant ways of knowing can actually lead to racialized controversies. So they advance this framework called CPDE and suggested that instructors should reposition minoritized students proceed to as low status, and they introduce these four new CPDE principles, and central to them is this idea that you use sociopolitical uncertainties to help problematize disciplinary knowledge in your courses. the way that we respond to this in P507 was in 2019, we added a cultural reflection to the four existing reflections. You can see it there. Now we thought that the modest nature of this reflection would be hopeful because it might appeal to adjuncts who have limited curricular control, and it might sidestep some of the pushback from more explicit approaches to equity, but I really was interested in can it catalyze larger changes via instructor positioning and repositioning via instructor comments. So I was very much looking for and encouraging students to use sociopolitical controversies, and I look for opportunities to reposition students who might be in the minority in the class. The way we studied this was we compared the 2018 and 2019 courses. We coded the weekly wikifolios for whether or not they use sociopolitical controversies, and we did thematic analysis of anonymous course evaluations across two courses. Then in the 2019 course, we did thematic analysis of the actual content of the cultural reflections and interpretive analyses of social learning analytics and instructor repositioning and that work is continuing. One of the things we found was that a remarkable increase in the number of sociopolitical controversies. Basically, there were a handful of them that you see, there were 11% out of 230. They were almost all associated with reliability and fairness and bias and standardized testing, those are the only chapters that introduced sociopolitical content. What we see with the addition of the reflections is this use of sociopolitical controversies. Again, we didn't code the, this is not including the content of the cultural reflection, this is everything else. So we were pleased to see that, that we had the desired impact there. The, the themes of the sociopolitical controversies, nearly half of them raise assessment bias, but what's interesting is half of the considerations of bias, were outside of this session of bias. That was really important because what it meant was students, for instance, during formative assessment, were appreciating that bias could be a problem there if they weren't careful. So we compared the course evaluations. What was interesting was that there was a politically conservative student in each of the sections, but the responses were very different. In 2018, the student complain quite bitterly about being silenced. And in 2019, the student self-identified and expressed appreciation of the ability to reflect on these issues. There you see the 2018, "way too philosophical and political, I had less and less trust in either the instructor... I will not take another course by this instructor." Really quite different than what we saw in 2019. "I introduced my views, I was met with objectivity and politeness." And as I'll show you from the comment, you'll see why I was encouraged to. "It was encouraging to me that my voice was a valuable contributor." I'm going to come back to that. So when we did the Content Analysis, 35, 18% of those really got at something that's really important to me was that they surface a sort of implicit bias that critical teacher educators have long pointed to as a source of inequity. So there you see one example on the reliability and fairness, but here you see another one on formative assessment is what I was getting at earlier. "Perhaps it is my own background as a straight white male to cause me to remember this only after I had completed most of the assignment." So this is what I mean by, right? And this is an example by the way, of prolapses not working, right? So that's some evidence saying that maybe we needed to be more explicit about it or that it that it kicked in at some point. That's debatable. This, the kind of thing that we're interested in studying much more carefully. And here you see the 2019. Now, I don't have time to really get into this, but what the student did was essentially reject the chapter's discussion of assessment by us and say that, you know, life is not fair. And here's my feedback, right? This is really hasty on my part in retrospect, it was a pretty egregious mistake. I said, you know, basically bravo to you for expressing this. I was sitting in a Refugio in the Alps with a line of people waiting for me to get to the satellite access, and really quite hasty. In the end, I was quite embarrassed that the way that I tacitly endorses students characterization of assessment bias as being politically correct. And really more importantly, we're really blistering critique from my colleagues with expertise in diversity in several tacit assumptions as a reflection in particular that I'm requiring someone rather than inviting them to speak for their group. There's potential for stereotype threat. I might have done more harm than good. We made some revisions to this. We added it later in the semester when we prepared to discuss it. I'm not going to talk about that today. Instead, I want to shift and turn this over to Joshua because we decided to hold off. He wanted, decided to use this data. It was an ideal dataset for his early inquiry project, he's currently in the process of revising his proposal. It's a fairly formal process into dissertations, level work and many other programs. So I'm going to turn this over to Joshua. Joshua taken away. Thank you Dan. So as Dan mentioned what we're really interested in here is how students are positioning themselves as authors and developing these sort of expansive and contextualized frames. So what we're going to be doing is developing a coded data set that articulates the extent to which each student did this within the wikifolios, the reflections and of course the discussions themselves. And we'll also be looking at how the instructor repositions students via discussion comments in that sense, and paying particular attention to the minority students, such as they are within these courses. Next slide. You can go ahead for them. As you can see from this graph, we do have some reason to think that there might be something to this process of reflection. So on the bottom of the x axis, or just the occurrence of sociopolitical controversy themes within a wikifolio. That's not the total themes that occurred, it's just whether they occurred or not across the students wikifolios. And on the y-axis we have proxy for engagement of how deep the students got into the discussions just by number of sentences that occurred. There was some correlation though it wasn't significant, but we would expect that the more engaged students are, the students that participated more effectively as experts and, and authors of content to engage in these controversies may also perform better on these assessments. So that's what we'll be looking at next. And I think you went back there, Dan. The way that we're going to do that, well, initially wanted to look at the units and data platform and we were intending to do that for this work. But what we found is that through the process of how it transforms data and cleans it. This course wasn't adjusted yet, so we had to go back to the older Canvas data, and do all the cleaning ourselves, so that put us behind a bit. But as I mentioned, we'll be using these codes and then looking at the associations between codes within a technique called epistemic network analysis. So on, on the right of the slide you can see an example of that, and this is just a student's first wikifolio. As you can see, they, they tended to have stronger associations between places and times pass. And then they, they held themselves and others held themselves accountable through that sort of articulation. What's neat about this is that we can actually develop a numeric representation of how the student contextualize through sort of the, the mean of these associative codes. We're going to be using that to do comparative work between how the learners performed over time as well as within the exams themselves. So hopefully we'll have more to report on that soon and I'll turn it back to Dan. Thank you Joshua. And so one of the things we're really excited about is we do have exam data in this course. One of the things I'm focusing on in this work quite a bit is looking as Joshua, the analyses Joshua is doing. We're going to basically look at all of these different factors and see how they relate to exam performance, because our exams were very carefully constructed to be, we never teach to them, right? So they're arguably they're estimates of the transferability of that knowledge to subsequent setting. So we think that that's going to be, that we also have a survey that we've developed with the support of the subtle grant that we're also looking at right now and we're proposing that as an alternative to the more widely used community of inquiry survey, which has never been shown to be related to learning outcomes. So with that, we'll stop and ask for, we'll take questions. I'm gonna leave this slide up while we take questions, but for now, we're just going to give it a wrap on this talk. So thanks for coming, everybody.