7th Annual Learning Analytics Fellows Colloquium
Below you will find recordings from the 7th Annual Learning Analytics Fellows Colloquium, which featured highly interactive virtual sessions in which Learning Analytics Fellows discussed the results of their individual and group projects, shared new questions or insights that may have arisen during their research, and outlined any future plans they may have for making use of the results, including the continued use of analytical data to improve the college experience for all students. The Learning Analytics Fellows program is supported by Indiana University’s Center for Learning Analytics and Student Success (CLASS).
Satisfactory Grade Option: Choice Determinants and Consequences - Michael Kaganovich
Learn more about Michael KaganovichDescription of the video:
Michael. Thank you very much. Yeah. You guessed correctly. And somehow because I was not supposed to be the one speaking tonight. It was supposed to be early. But wherever. Last minute change in cash. The primary cancer, let's replace the secondary ones. That's a piece of bad news. Rule, rule. Last minute situation which cost us. But it will be my pleasure to welcome project. Especially because really the lion's share of the work. So it's some, my part is easier talking about this. So let me tell you what, what this is about. So this addresses what economists and modern economists call a natural experiment. A unique event happened in the spring of 2020. Well, besides the fact that endemics, yet a consequence of that was them, a lot of classes had to switch a modality, midstream. And students, obviously, outside of IU and I, you were hit with a major disruption and that caused the USU policy chef in grading. So all students, we're given an oxygen. And that was announced in April. And to choose a grade S satisfactory to place their hand on the grade that they otherwise class. And so our purpose is to study, start studying the impact of this unique natural experiment is when we call it that way. So, so one result, when one outcome of this is that everyone who acts. And an important elegant that was, that that policy was that students didn't have described right away in April, but could made their choice up until May 8th and possibly even laid some. May 8th was the mandated universal deadline. The choice, amazed, by the way, habit to be the last exam. The exams, that's an astronaut. But also the University encourage instructors to give students more leeway and give them an opportunity to learn the actual performance before, before making that decision, the choice of anoxia, great option. And so in order to figure out, so we need, we needed an additional piece of information in order to address these questions. In addition to the wonderful systematic data to now, we are getting from our concepts from Greece. So for this one source was candles. Because unlike millimeter assessment and research which tells us the outcomes, great outcomes, cameras can potentially tell us. The student progress within Canvas could tell us, in particular, and so helped could tell us how students are doing by the time they made. And I'll tell you in a second why why this turned out to be not, not so easy. And what we were able to accomplish with less a nation that we're hoping to get so bad. I would like to thank both momentum and research and learning. And the Latvian latter organization is collecting the data from Kant's. So this was a major undertaking because it entailed merging two data sets. Furthermore, merging it in such way that way. Compromise students privacy. So, so the, the two datasets have identified, the students. And the merger was accomplished in such way that we couldn't identify students who question. But we could just be assured that the bar data and the data about students, communities, and Canvas will not work properly matched. And soaps will be the questions that we wanted to address. The address understanding in the first place, understanding the factors and students choices. Well, one might just assume that it's easy that if a student is performing badly, then they're more likely to choose that stands out, is not all that simple, but there are other factors, such as students for months, students demographic characteristics, students year. You could show you one of those results. And in our longer term Go, which is true, It's too early this time around because we've labeled it took us a while to get all the approvals and go get F-measure yesterday. I don't know if you can get a little closer to your bank maybe or aiding you're fading in and out a little bit. That's all right. That's how it yeah. That's better. Thank you. Yeah. Sounds sounds in the distance. Yeah. So so a longer term go is to study the concept of choosing units. Because the choice of s effectively lowered the bar for many students to get a passing grade and to progress to higher level classes. So for instance, someone who would be otherwise getting a C minus re-explain a short time ago, then they wouldn't be able to progress to other other classes. An account. Now the he in that particular semester they could and so and they couldn't. And then the question is what happened as a result? Was that a 104 instruments to pick a pig's the semicolon on that subject matter. It wasn't that important for the subsequent progress. Maybe those students who didn't make it. Well, we did that using, using a time. Maybe they could have done just fine after that. So some, the s option is, is your experiment because it allowed basically everyone to progress. So that's our goal, but who hopefully will help to be able to pursue this next year. Once more information comes then to bar about student performance style throughout. So I very much and spoke to the radius. So we can skip the instrument, which has given us a second to take a look at this slide. And now I'm going to train him. It wasn't as much 20 nights when the policy was managed. Very very soon after the switch on the LED or your resulting from pandemic. So now we try to do, think of the situation. Imagine designation, STM design when making this decision. Should I stick with my grade that have otherwise getting or should I choose x? And sum? When economists talk about trade-offs, they meet costs and benefits. What is a cost to me are making this decision? And what is the benefit? And students performance in the course as well, is obviously relevant to the benefit, right? So, so if my performance was bad, no instance, then it would be beneficial. Um, but my pride performance awesome, That's great deal. Because let's say I used to be an, a student and now I'm getting a B minus and that's terror. Whereas if I was a student all along the spine while it actually connects cells. So these are some of the questions or would like to explore now. But there are also some, some costs may not come to mind immediately. So not all students are always. And I think all of us taught classes and, uh, you know that a lot of students don't know how they're doing. With Canvas or not, how they're doing it in a course in any given moment. So they need to undertake, to take an effort in order to explore that. They need to calculate that gray canvas may be confusing them. You need to interact with the instructor. And so sometimes decisions or to take, as I'm not a tech has maybe predicated on that. One student's willingness to invest in finding out how they're doing. And then thinking, he said, is that for me, it's not good for me hands. And the final fact is, it is what economists would call behavioral. That's, that's what we find. The most interesting, one of the most interesting ones to think. So. So if a student decided to take an S option, declared that aerator, the instructor or not, has decided no matter what. I have the Snopes, I guess. Let's see at the last minute. Well, now I'll actually permitted but I have this now. I know I hadn't stopped. And now that, that option potentially allows me to slack off. And so economists call this type of behavioral hazard, moral hazard. And so this used to be an inside term among economists. But you may recognize it because it was made, made famous during financial crisis of 2008. And let me remind you of the context. Snow. So there was a big prospect of failing companies being bailed out. And sounds for the problem and the Communists and non-economic like in the media spoke about the problem. Bailouts. Bailouts is moral, has that if you know you're going to be bailed out, why would we maintain this? So this is this is a similar situation. So if if i now that I have an option, maybe a SledDog and 10 was gives us a tool to to figure that out. But again, distance, that's something that we're soon. Very soon, but we don't have results yet. But there is this grade on canvas can tell us whether students continued to login to the class into Canvas itself, like or how much time they spent then? Yes, it is one lane. You have about five minutes left here of it. Great. Okay. So in that case, let's move on. Cameras date I already alluded to are that as snow. It details and the the instructor about all the assignments then and tells us children the same thing and grades received ants. And Portugal were found in numerous problems. With data through Canvas, calculates the braid itself, by itself, the score. And so our purpose, one main purpose in using Canvas is trying to infer students actual performed before deciding, deciding which rate data. And so, so this, this is one selection in, in thread column you see the grade that stood natural, ended up choosing move, obviously getting, so obviously these are students who chose not to DNS and then get these blacks close out. The scores that Canvas. We're telling students that we're getting corks in the app. And then you'll see that there is non-marital. So this is just a snapshot of this axle outward, but it goes all the way to two low grades. And we see that there are surprising non-white students with some, some students but a, a plus with the 75 and other students and be managed with this query. So there are some consistencies and inconsistencies in the way Canvas and science course. They, they either wait. All the assignments equally well as instructors have an option to override this. But not all instructions use instructions. So as a result, we, we could only get luminary zones. And so we decided for now not to publicize results based on that Canvas grade. And so we don't weigh album sulfate now, presenting sounds. Without tendency. So strictly using bar state, right? Yeah. And so to form a logical progression, where likelihood of choosing S as opposed to actual rate is independent parent. And so, so here are some, some results. So the higher the price cumulative GPA, the lower the likelihood that the student chose x as a grain. In the first approximation, this may seem obvious. Obviously data, This was high GPAs students. These are better students. So they're probably more likely to have performed better in the course. But remember that these choices are rather, and so this result tells us that the choices are not, not all that well, not all that for relic, that with higher GPA students still, regardless how the new grant related yet they face now Hi Barnard, match. And yet the likelihood of choosing ask is Islam. But so being male increased one month probability of choosing S, controlling for prior UP another, another characteristics. And so in the second specification, generates interacted with pride and willing to pay. And so this result sounds that the higher GPA, the greater the gap between men and women. Okay. So at high grade levels, men more likely than women to choose as risks. With me. Now, some data for raceway ASM has a benchmark and so sounds that white students controlling these are the prior GPA. It's parables are less likely to choose S, for instance, and not much. So receiving Pell Grant being first generation doesn't make much difference in terms of choosing S control for other variables. But residency. So Indiana residents are less than one, less than equals x. Now this does somewhat interesting. So the higher the year and I, you, the more likely students will choose yes. To both specifications come these results and signal. So again, controlling for performance. So the benchmarks here gets a first-year student. And so the higher the year, the more, the higher the incidence of choosing as neutrons, which has entries. And so finally, among these questions, so we broke down, we broke down students into five major academic categories. Education, of course, it's the slowest, but it couldn't be combined with anything else. Other professional schools, scam and social sciences and humanities and both business higher education is a reference point. And so students in stem controlling flow characteristics were less likely to choose S than business tunes. And the same is true for science and humanities. So business Jones, as matter of fact, most by teachers S. So maybe they wanted to progress so bad from that A1 100, two to the rest of their curricula. So it's going to other professional schools. Or the CountSketch were the least likely next step. Now, as I mentioned, and so now hopefully appropriate right now. So our plan going forward is first of all, to try to actually get hold them so better understand the Canvas grade data and then analyze the impact of that on the Canvas grade on students and essential to choose, ask, then to test for moral hazard. So was chosen gets hosing. And identifiable, has questions slacking off after after March 20 nights. And then a beggar and pursued after that is looking at consequences of choosing us. How did how did that affect the progress with students afternoon? Michael. Thank you. Yeah. I'm sorry to have to go. So I just want to open it up for a few minutes before we go over as we take one or two comments or questions, if anybody has anything they'd like to say about this.The Factors that Influence Entry, Exit and Progress along Students' Pathways to Success: An Examination of A100 - Leslie Hodder, Bree Josefy, and Jie Li
Learn more about Leslie Hodder, Bree Josefy, and Jie LiDescription of the video:
Next is going to be R. Kelly folks. So I don't know how you all plan on doing this, but I'd like to give, you know, I was giving chase the 10 minute, five-minute, two-minute kind of warning so you could keep on track. Who who should I send it to in your group sensors, three of you presenting had not defending that, had Bree. Okay. I'd just be like Okay, presented as bytes. Okay. Thanks, Leslie. Maybe introduce yourselves to again since we do have people from outside the Bloomington campus. So thank you again. All right. Thank here though, I'm lovely, hotter and I'm a Professor of Accounting at the Kelley School of Business. And this study is in collaboration with a brief Joseph I and J Lee and J Lee has been a class bellow for a number of years. So the food, the collaborative grant. And before we continue, I want to thank George and Linda and also the Center for Learning Analytics for this opportunity. Because I know that I personally learned so much from data. I didn't even realize they were available. And be persistent questions within our programs and departments that I didn't realize can be informed by the data. And, and I just learned an incredible amount and I'm excited about continuing increase in that direction. J has been instrumental in helping us learn to access the data and manage big data and complicated and breathe and excellent researcher. And so this collaboration has been fun and rewarding for all of us. On the next slide, we have our agenda. And there we're going to talk about our overall purpose and motivation for this study. We've done many additional exploration. As I mentioned, we're interested in continuing. Then. We'll talk about the research question that we felt that we've asked and answered with the data. Using some logistic analysis and regression type analysis. We're showing association not causality. However. And we're trying to bring to bear statistical significant and q all of our now to help us to draw inferences. Also talk about the data. First, the j will go up at that. And then we'll show you our results. And in proposed intervention that we've thought about, although this is very preliminary. All right, the next slide. So there are overall motivation. So I've then colleague or 19 years and I've noticed from structural shift and the time. And I open them differences between Kelly and other institutions I've taught, which include the University of Texas, Austin, another big state school. And my interests, it is primarily around accounting. This is where we began, although with J on board. And we can extend this to other stem type field. And accounting is a great profession. It's a great profession because It's fairly balanced in terms of its demographics. The American Institute of CPAs indicates that about 49%. So what you would expect, half of accounting graduates nationwide identify as women and about 44 percent identify as non-white and so pro-business on it. Pretty diverse. Upward mobility is provided by the profession because when you're a Certified Public Accountant, you can pretty easily go become an entrepreneur and start your own accounting practice and provide flexibility during life. I know that when I was raising my young children, being an accountant provided that flexibility if I'd still a professional and still proceeding with my career, but have some flexibility for for families. So we we feed accounting of the desirable profession that pays well. And, and it's, and it's pretty diverse. But yet what we've seen at Kelly is that although the enrollments are increasing, so over the last 18 years or so, the accounting majors as a percentage of business majors has been declining dramatically. In fact, I would say we take confound at approximately or 100 accounting majors, even though the enrollment ID which is increasing. So we're disappointed for the county area or disappointed to see that declining market share. And we have an ad hoc basis attributed this to certain structural shifts, including an increase in our direct admit students. I'm an increase in the proportion, perhaps out-of-state students. And we see on a macro basis, but I'm a pretty profound Kelly. Majors are finance majors now. I'm, where's the market share of accounting? And the other major has been declining. But beyond that, we know that even with an accounting or accounting majors that Kelly are not representative of the population of accounting majors as a whole or the larger campus. So at IU Bloomington, 49 percent of students identify as women, 29% are BIPOC. Kelly, 34 percent of students who identify as women and only 12 percent, or by art and accounting major group, an immediate SLP can also be the only 23 percent of accounting majors identify as women to 0.50% as iPod. If you compare that to the nationwide, or they're put out by the American Institute of Certified Public Accountants. You can see that that we seem to have something unique to Kelly as far as our under-represented populations. Now, you'll notice in our title fight that we decided to start at the very beginning with a class called a One 100, and a 108 required of all business majors. And Martha spoke earlier about the prospect of early failure and how early failure is known to result in decreased, persist them in colleges overall. But a 180 kind of a unique course, because if they have Term course and it can be repeated a large number of time, and then the grade can ultimately be removed from the transcript through what's called an extract that and though it shouldn't be a low stake early failure. And I think that p1 hat with a talk with study skills type course. And so it's topically related to your accounting. But I think when, when the course was originally conceived, it was meant to be meant to be a weed out course. It's an indicator to students that they needed to up their game and learn, study skill. And it has a very high DFW rate. So BRI is going to talk a little bit more about the role of a 100 in the curriculum and then j will talk about the data in them. Do I think lesly. So as you can see, the, when we wanted to look at the pathway for our business students and what's happening there, their entry points and exit points. We first started with a Degree Map. And so this is showing the 8 first year generally first year required courses for business majors. And as Lizzie was talking about this, the DFW rates, you can see that a 100 has one of the highest DFW rates of the first year courses that business majors are expected to take. Now there are absolutely some of those math courses that have much higher DFW rates. But there's a variety of different courses the students can choose to take in by Nmap, finite math and calculus. So, so we really wanted to focus on the A100 class. And particularly because it's, it's a one credit hour course with a 31 percent DFW rate. As Leslie said, it is over eight weeks. It's also unique and that I'm not many institutions have an introductory accounting course similar to ours. Many of them will have a sophomore level, a 200 level, full three credit hour sequence. So 2, 3 carrot our courses that students usually taken the sophomore year, um, but IU and Kelly is unique in offering this introductory course. So while a 100 has the opportunity to introduce students to business and in the accounting profession and accounting majors. It also has the potential to deter students. And so that's one of, one of the main reasons we are focusing on a 100s. So our research questions in many ways similar to the pathway report that Chase was just looking at. So we're looking for what predicts, first of all, non progression outcomes in a 100. So what are the factors that demographic factors that are indicative of non progression outcomes? And when I say non progression, we're actually slightly modifying the typical DFW rate because you might have noticed on that earlier slide, students have to earn at least a C or better in order to progress in the Kelly curriculum. So this isn't an added challenge for our students. So we have added onto that DFW rate and said, non progression is any grade that is lower than a C. So we're looking for non progression Atkins and a 100. And then what impact does initial A1 100 non progression have on student degree and major outcomes? So does it lead to a lower likelihood of a degree at IU, a degree at Kelly. Or a degree in accounting. And then also looking at some of those changes in majors or decisions and major. So if someone comes in initially interested in accounting, that's one of their declared majors and then they have a non progression outcome and a 100. How likely is it that they're going to switch out of accounting? And this is important for us because as Leslie noted, accounting hasn't, we haven't increased our market share her even though we have more students coming in. And so we really are trying to understand, are we losing students from the very eerie coming in at a shortfall or at some point along the way, are we losing those students because of maybe some of our courses? And then finally, as, also, as he noted that this a 100 is topically related to accounting, but more focused on study skills. Is it possible then that students who have a non progression outcome and then repeat the class to take it again. I'm, do they do better the second? So did they actually learn? So maybe it was one of those cases where failure or non progression when you try to stay away from that word, failure. But non progression can teach us here. They can actually learn a lesson from that and then they can take a 100 again and potentially do better than maybe some students you've only taken at once. I'm certainly better than they did the first time. And so potentially that is something good that could come out. I'm sorry. But look to explore what happens if they repeat the class. Now I'm going to turn it over to Jay to talk about our sample selection. Okay? I just unmuted myself. Thank you. Break. So our dataset is from our dataset is from bar, provided we have four sets of data. One way students attributes, one, student retention, which is a, which we mainly use the graduate degree data from student retention. And we have student major History, which is also called program stack. And we have student course history. And the student population goes from fall of 2006 to spring of 2021. And the level of, the level of details in these tables are slightly different. For student attributes, we have Y rule per students, but for retention for major history, we have one rule per student term. If the students existing ie your student records. And for student course history, we have one row per student per term per course. So you can see the amount of data we're trying to dealing with. So when we, we explored how we want to work with data multiple ways. And finally, we decided that we need a main table, a major table that includes all of the IOB students, not just students, because we have students trying to take a 100, even if they are not a kilometer or even if they don't have business as their intended major. So we created a wide table, include all IU students who have ever enrolled in this period of time. And we try to take the level of detail into one mole per student. And we transposed the student's history, their major history, and their core sticking history, not all the causes, only a rounded we transport transposed them from vertical to horizontal because this is the only way we can do our regression analysis. So we go, got it, George. And so we try to keep all the student majors. And the data is complicated because one term, you could have three majors. So we end up with a very white table. But with that one table, we can see the major declaration history up to 12 terms. And we also have all the students attributes merged into one white table. In our research questions, we decided to focus on A1 100. And out of all the students, we have 43,408 students who took the course. And then we have these numbers, students who are still, those who have graduated or dropped out from IU. And these are the number of students who have taken a 100 just one time. Okay? Yep. Yep. I talked about the challenges already and this is the trend that we noticed. So when we talked about the high DFW trend, and in this data we also included c minus, so non progression. And we can see that this is the enrollment of a rounded, this is the count of DFW C minus. And we can see the trend is always have always been high in these 39 represent fall, spring, and summer enrollments. Next, please. And these are the enrollment trends for a 100. This is very similar and we can see that enrollment is still increasing. City and increasing. And non progression reads are indicated by this red color. And you can see the percentage higher, lower, but pretty consistent. And repetition attempts to. So we have students we notified when in our data we only remained students who taken up to three times of 1100 attempts, but we have students who have taken this course up to seven times. And you can see the percentage 8% in fall, 22% in spring. This is really interesting. We don't know why in spring, students have more have much higher rate of re re, attempt.` Yeah, and now you can see the agreed distribution of these up to seven attempt. You can see most students passed with the first attempt, but we do have over 15% of students have to take it twice. And then still lower wise. And these students take several attempts. Did did graduate but he or she is definitely an outlier. So yeah. Do you want to shortly here? Yeah. So this, this will read some of the descriptive statistics for our main samples of all students who have ever enrolled in a 100. And so just a few things to point out. Here. You can see that 33 percent of students who have ever enrolled in A1 100 identify as female and then only about 8% identify as, as URM. If you look down at that, that SAT scores and GPA scores, it's it's fairly high. We have a mean of 1300, quite a large range there. And then I had to high-school GPA of 3, 69. And and many of them enter having at least four, are having on average about 14 credit hours. So this maybe from AP classes on or it might be if they're entering in that from the spring that they've had credit credits from the fall. Just a quick look at some correlation tables. So we wanted to make sure before we went into our regressions, We're on any variables that are highly correlated with each other. And so non progress is going to be our dependent variable in our first year regressions. And then the, you, you didn't see the rest of their independent variables. And so one of the, one of the main associations you can see is that standard emit is highly negatively correlated with SAT and GPA, which this is understandable, obviously understandable. So we've modified some of our regression analysis to focus on standard admits. In some robustness, we do take out the inter-domain, have the SAT and high-school GPA instead of that scene are going. So our first research, research question. So what predicts first-time non progression outcome in a 100s? So we did run a logistic regression. So one is if a student had a C minus or below, so c minus d plus d, f, w, and then a 0 if they did progress with 81, 100. And we ran this regression on a mole to mole to multiple different factors. We're mostly interested in female URM. And then we also want to understand how the standard admit plays into non progression. So you can see our first regression is main effects. And a female, say unfortunately female URM antinode, admit we're all positively associated with non progression. So that's bad news. That means that on average, females are 29 percent more likely to not progress in their first attempt at a 100 than other students and similar for URM. And then our standard admits actually really high. So 51 percent more or 52 percent greater likelihood of not progressing. On our second well, we also had some main effects for resident and first-generation. So what could be entrepreneurs for? Some good news here is that our first-generation students are actually less likely to not progress through A11 100. So then we wanted to look at just some interaction effects. And what we see here is that the interaction of standard admit and female students are actually less likely to not progress through their first-time attempt at a 100 wasn't any significant effect on your ends. And then we looked at the combination of female and your answer, that intersectionality, they were also more likely to not progress. Our second research question then looks at graduation outcomes. So after a 100, our students more likely to get an IUD agree I Kelly degree in an accounting degree. So we looked at three different dependent variables. And the regression that we're using here is also looking at some interaction effects of non progression times female. I'm so let's look at that 1 first. So across the 30 degrees, you see that if you have a female who did not progress in their initial attempt at a 100, the first result of it, they're more likely to get an IUD agree, might seem a bit counter-intuitive. But I think if you combine that with some of our descriptive statistics, that, that on average many of these students are highly capable and they may just have, you know, either it, they may decide that because they've had that first incident of failure that and the business degree isn't for them, but they're still motivated to get an IUD agree. The second one, Kelly degree, you can see that it's negatively associated. So I'm females that don't progress and a 100 on the first attempt are less likely to get it to tell a degree and less likely to get an accounting degree. When we look at though, the URM statistics, The Kelly degree, sorry, There's lots of red lines here. The URM who don't progress are also less likely to get a degree and then it wasn't statistically significant for an accounting degree. And then finally, our standard admits. So standard admits who didn't progress in their initial attempt were less likely to get an IUD agree. Kelly degree in an accounting degree. And I know I'm running out of time here, George. So this is one of our last regressions here. We also want to look at the, if this had an effect on student major decisions. And so we looked at if they change their degree from entry point to exit point. And so you can see here, this is just looking at accounting students. So our students overall so that they entered accounting. If they entered IU and did not declare honey as a major, that would be in the 0. And if at some point along the way they decided to add an accounting degree. So we can say that again, accounting, and then vice versa, if they entered as an accounting student and then did not graduate with an accounting degree, then we consider that Louis accounting. And so we ran a regression would deepen a variable being Louisa County or drop Accounting degree and then gain accounting degree. And, um, for females, don't progress. We did find a lower likelihood of adding accounting. I'm in similar for standard admits, but we didn't find any significance on dropping an accounting degree. So this is interpreted with some, a little bit of good news and that we're not deterring students, but we're certainly not attracting students are encouraging students that have not progressed. And 8100. All right, and then finally our third research question is looking at distance. Who don't progress the first time. Do they obtain higher grades on subsequent are elements. So that this first exploratory analysis we're just showing for students that had, that withdrew and those, those students are tricky because we don't know exactly what grade they would have had if they had completed. So you can see here that the majority of students who do withdrawal on their first attempt, when they, and repeat they come back, they persist in it. About 31% of those make it be we do have 27 percent that successfully finished with an a boat, but we do have those students that 12 percent, again withdraw, 4% have enough and 6% have a dy. So our last analysis we wanted to look at what effect repeating would have on your 8100 grade. And so in this case we're doing an OLS regression where the grade is a numeric version of the letter grade. So 0 to four. And then repeat as our variable of interest. And it's one if a student is taking a 100 for a second or third or fourth time. So you can see here when we ran our first model, when we did our main effects, there, there wasn't a statistical thing. Wasn't a statistically significant correlation with the A1000 grade. I'm going to look at are females and repeat females here. So when we looked at our main effect, females were less likely to have a higher grade, I should say Lesley was negatively associated with an A1 100 grade. But when you look at the repeat females, it's not significant. And then a similar story with are you ORMs. And then with our essays when they repeat the actually, it's negatively associated with their grade. So this, this doesn't seem like great news for those students that are repeating. It's either not associated with a grade, a higher grade, or it's negatively associated for those standard admits. So we, we definitely want to do some more research on this question and, and, and figuring out what's, what's happening. We have some students who the w's do seem to be doing better, but the students who took the class got a grade and then took it again, it doesn't seem like repeating is having an effect on that. So interventions for students success. As Leslie mentioned when this class was started, it was, we hate to use the word but a weed out course. And that's what it was considered and focus on those study skills and so on. Just this past year, one of our faculty members who has been teaching this class for the longest time and doing an excellent job, has retired. And so we pivot to hire new faculty member. And this person was new to IU. And so some of these innervate, these interventions have been implemented. And I want to, I want to say very clearly that the research we were doing and these these changes and interventions were happening simultaneously. And part of it was because we had started looking at this data about a 100 in the spring, right around the time we hired a new person. And so I was actually helping our new faculty member to understand what a 100 is. Understand what I us, and to reframe some of our learning objectives and the assessments that, that combine a 100. So I'm, so I want to be clear, this is not as a result of all of our research that we've been doing, but it's, it's happening simultaneously and it's part of the awareness of understanding that there such high DFA is 0. And I'm sorry George, I see that we need to move on. So I'll leave you with the final thing that this shift has result. We don't we don't have the full data, but for the first eight week results, we we do know that there was a 10 percent DFW rate rather than our historical 30 percent. So that's that's great. News, and thank you. All of this were you used up your pen minute? Q. And a. Oh, I'm so sorry. That's your option, I guess. So we're going to have to move on.Data-Driven curriculum development for student success - Martha Oakley
Learn more about Martha OakleyDescription of the video:
Okay, So, you know, all of you are obviously here to talk about data-driven curriculum development and other, other cluster and development for students success. I am the Associate Vice Provost for Undergraduate Education, and I've been very interested in these projects essence, I've come on board about a year and a half ago now. And I am really excited to see all of you here, both from on-campus. I'm really looking forward to hearing about your projects. I enjoyed it very much last year. And I also am delighted to see folks from around the country who can hopefully have some time to share their thoughts and questions during the Q&A sessions after each talk. And I just wanted to talk a little bit about my involvement with learning analytics and, and, and sort of communicate where I hope your projects will go in the long-term rate. We're all starting off looking at the numbers. And I, and I, I want to pose a couple of challenges to you and also offer all the support that the various portions of the vice provost office can provide in and helping you as you as you take your data and try to figure out how to make changes that will help students success. So I, I got into it into this area of learning analytics with a project that, that was done by a number of universities through a collaborative called the seismic project that focuses on stem education. And Stefano and Linda's group was really involved in. Stefano and Linder were very strongly involved in this. And I became just in awe of what the good people at bar can do. And you folks who have been involved in this learning analytics understand just as, just as well as I do what they do and just how excited they are to help folks use the data. But there is a, well, we've seen a presentation that basically said that women in science classes have what we call agreed anomaly. They do worse in their science classes. And and this is the grade they have versus their overall GPA and science courses. These are the lecture courses, but they do better in the lab courses, a little bit better in the lab courses. And and so Laura Brown and I, and this is really mostly L4 Brown's work, not mine. Applied for learning analytics grant to try to understand how this was working in in our Department of Chemistry. And one of the things that we discovered was that the issue wasn't so much with women. We found that women in chemistry just about as well as men. Now, That's only true if you don't take an intersectional lens to it. I've recently been analyzing some of our big classes. And what we find is white women do as well or better than white men in our chemistry classes, but that's not the case for women of color, for black and Latino women. And so there's clearly some work to be done. And we don't want to be ignoring the subgroups. And so I just want to talk to you about what I've been learning in this job, about who we teach well in whom we don't include as effectively. And so this, what this chart shows, and I think Stefano would probably cringe that I haven't shown you the ranges there obviously is a big range here, but I wanted to make the data relatively easy to see. It turns out, if we think about societal privilege. So if you are, if you are white or Asian American and you come from affluent family and your parents went to college. Your average GPA and all courses at, at IU is almost 3.4. But for each, for each of those, those privileges that you lack, your GPA goes down. And in fact, if you are black or Latinx, low income, and your parents didn't go to college. Your GPA for all courses on in the whole university is, um, is, is a point, is more than 0.4 points lower. So that's actually a lot when we're talking about this many students. If you're in, if you're in chemistry, that number it chemistry classes, that number can be close to a whole point. And I've seen an even higher in some classes. So what this tells me is that what we do works for some of our students pretty well. And for others of our students it works less well. And everybody's supposed to be getting the same experience here. This is a public institution that's supposed to be helping all of our students. And so I look at these data and I think she's, we have to make some changes. And so I'll pick on my own department a little bit. This is we basically have a three-course sequence that all are pre-med take. And if you get through and if you haven't had great high school chemistry, you can call it started what I'm calling course 0. So that's that, that's an optional course. And what this shows us is the DFW rates to the rate of students who do not succeed in the class who are in a, D, and F or withdraw from the class. And it shows the numbers for what the institution defines as, as under-represented minority students, in other words, black and Latinx students mostly on our campus and a non URM students. And what I hope you can see, hope you can see three things. One is that the numbers are ridiculous, especially. So for instance, if we look at course one here, 25, 25 percent of our, of our non URM students don't make it through the class, but more like 40 percent are under-represented. Minority students don't make it through the class. When you get to course two, those numbers are even bigger. In course 3, they come down a little bit, but they're still nowhere near where we need them to be. And so in Coursera, we also see similar trends. And so we know that if a student gets a single w in any class, it increases their chance of not graduating by 50 percent. If you get a D or an F in any class, it, it, it increases your chance of not graduating by, by a factor of two. Okay. And so, so we're not talking, I'm talking between six to 9%, 6 to 12 percent, but that's still a big difference. So we often think of these things is, yes, some of our students have to figure out how to study. We keep them, but they come back. But it has a real impact and can have a lasting impact on students lives. And the other thing is that we sometimes say, Hey, our job is to figure out who's good at our subject. And then we'll will nurture those students and students who aren't good at it will help them find, will help them understand that they need to go look at something else. If that were the case, what I would expect is very high DFW rates in the first-class and lower DFW rates in subsequent classes. And that's not what we see. All right. So that tells me that our curriculum is not fair and it's also not effective. We are not providing students with what they need in the first course to get to the second course and so forth and so on. So that was a pretty sobering understanding that's gotten us thinking a lot in my department about how to fix this. And I'll just point out that I taught course three for a whole year once. And I would like to tell you that it was 2013. I will tell you that was mostly my colleague, more brown. So she has some really good ideas. But, but this was, but up books I, but I taught in 2011 and I want a teaching award that year. And I got great teaching evaluations. And I thought I was doing my job. And I look at this and I wasn't. So I hope that when we look at our data, we can think about the ways in which what this tells us about what our institution is doing compared to what we would like it to be doing and what we're doing in our classrooms compared to what we would like to be doing. And I'll, I'll skip this slide in the interest of time. I'll actually, I'll just say that, you know, that that's one kind of data you look at. The other kind of data that we looked at is who persists in classes. And this helps to make arguments to deans to invest in things. If you can say Gosh, if, if underserved students were to persist in our client, our sequence at the same rate as as privileged students, we would end up with an extra 10% of students. And then you'd get that RCM money. That's helpful. And for women in chemistry, even though they do better if, if, if I do the same analysis, we would have had 800 more women in that class, which is a 25 percent increase. So these are, these are ways to talk to deans about about where to invest money to help things, and I am happy to help you have those conversations. I love trying to tell deals they need to invest in our courses. And then I want to, I want to point out that there is a department that has had a dramatic success and, and some of this may have started with some learning analytics grants. I see Michael here from Econ, what? Econ used to be one of our worst offenders. It had very high DFW rates. In fact, those rates were 60% higher than our peer institutions in the Big Ten. And so we had our eye else and he kinda what he did was they said we're trying to serve too many people, too many audiences with this one course. And they divided the courses up into a business econ macro and micro sequence and an econ macro and micro sequence. And in the first year of that first year, year and a half of that I think is the first year of that project. The DFW rates went from almost 30 percent to below 10 percent in all four classes. To me, this is, this is a stunning, stunning success and i'm, I'm convinced they've done this right and they haven't just made the classes easier. I think they figured out how to help students learn better in these classes. And, and I hope we will get a chance to hear more from these folks as time goes on. So I'll just close with that and reiterate how much I appreciate you all coming. You're all taking on those of you who are fellows, you're taking on these projects and you're sharing them with the rest of us. And I look forward to hearing more from all of you. Thank you very much, Martha. I appreciate it. I appreciate all the support and you've been giving us as well and the work you're doing by its purpose office, it's an exciting time.Description of the video:
Greetings, my name is Paul Graf and I'm a senior lecturer in the Economics department at Indiana University. Thank you very much for your time. First, I'd like to thank the Learning Analytics Fellows program funded by the Center for Learning Analytics and Student Success, Dennis Groth, the former Vice Provost of Undergraduate Education, George Rehrey, the founding director of CLASS, Harsha Manjunath and Linda Shepard at IU BAR and Nikita Lopatin, professor at Ashland University. Last year, Dr. Gerhard glom and I explored the probability of students remaining in economics after choosing economics or checking the box on their applications to Indiana University. We found gender, ethnicity, in large sections had significant effects on the likelihood of remaining in the economics major. Based on these results, Dr. glom suggested expanding the analysis to include other majors. Often, students declare a major upon arriving at IU Bloomington, which may change from when they first applied. Since students may change their major for different reasons, like their experience taking their first major course. After completing this course, students may switch for two reasons, ability or preference, or the grade they received in this course. This project focuses on the latter reason. So I believe this project may have important admission and retention policy implications that may improve these figures and therefore, the student experience at IU. This project analyzes three effects on the probability a student stays in their declared major. One: prior knowledge measured by SAT scores and high school GPA. Two: experience, represented by the first major course grade and overall student performance that semester, excluding the first major course grade designated as GPAO. 3: Other qualitative factors like gender, ethnicity, residency, and financial need. After exploring different major enrollments, over the last five years, I chose economics, business economics and public policy or BEP, accounting, finance, biology and psychology. Next, I identified the introductory courses for these majors. The introductory course chosen was E21 for economics and BEP, A100 for accounting and finance, L111 for biology and P101 for psychology. Further studying each major, I identified specific qualitative variables to account for program-specific variation. I used a binary model which I estimate using the linear probability OLS method, and for robustness, I also used probit and logistic regression. To compare all six majors using only quantitative variables, I first ran the regression of the likelihood a student remains in their declared major one year after taking the introductory course against four qualitative variables: high-school GPA, SAT score, the course grade, and that term's GPA, excluding the course grade. Next, to model major specific variation, I ran, similar regressions including identified qualitative variables specific to the major. Due to their similarity, I compared economics and BEP, identifying the qualitative variables of female Indiana Resident and first-generation based on the descriptive statistics. For economics and BEP using E201 as the first major course, I found for economics, the quantitative variables were not statistically significant. Indiana residents were, on average, more likely to remain, but first generation students were more likely to switch out of Economics one year after taking E201 For BEP, the higher the students high-school GPA and SAT scores on average, the more likely they remain in the BEP major one year after taking E201. Other qualitative variables are not statistically significant. Next, I compared accounting and finance and found similar patterns in their Descriptive Statistics. Specifically, I identified the qualitative variables of female, resident, first-generation, Asian, Hispanic, Latino, and Black African-American. Using A100 as the introductory course, I found all four quantitative variables on average, having a statistically significant effect of remaining in the declared major. The higher these values, the more likely the student remains in their declared major after one year of taking A100. For accounting, on average, Indiana residents are more likely to switch, while Black African-Americans are significantly more likely to stay one year after taking A100. For finance, on average, female and Indiana resident students are more likely to switch, while Black, African-American, and Hispanic-Latino students are more likely to stay one year after taking A100. Finally, I compared biology and psychology due to their popularity. I identified male, non-resident, first-generation, Asian, Hispanic, Latino, and Black African-American as the qualitative variables. Running the regressions for these two majors using L111 and P101 as the first course for biology and psychology respectively, I found all but a student's high-school GPA has a statistically significant effect on remaining in both majors one year after taking the first major course. On average, males, Asians and Hispanic and Latinos are more likely to remain in the biology major one year after taking L111. On average, Hispanic, Latino, and Black African American students are more likely to remain in psychology one year after taking P101. To summarize, the first major course grade has a significant effect on the most popular majors. Prior knowledge matters for students remaining in their declared majors. A students performance in other courses taken during the first major course semester, Gender and Residency has varying effects. Except for economics, first-generation has no statistically significant effects. Finally, minorities are more likely to remain in their declared major one year after taking the first major course. Further considerations, I would like to explore additional majors, generalize the results at the university, college and major levels to help students stay in the major and potentially increase the diversity and inclusion goals at IU. Furthermore, I'd like to look at the variables of grade penalty and interactive effects on changing majors. Thank you very much for your time.