Description of the video:

Michael. Thank you very much. Yeah. You guessed correctly. And somehow because I was not supposed to be the one speaking tonight. It was supposed to be early. But wherever. Last minute change in cash. The primary cancer, let's replace the secondary ones. That's a piece of bad news. Rule, rule. Last minute situation which cost us. But it will be my pleasure to welcome project. Especially because really the lion's share of the work. So it's some, my part is easier talking about this. So let me tell you what, what this is about. So this addresses what economists and modern economists call a natural experiment. A unique event happened in the spring of 2020. Well, besides the fact that endemics, yet a consequence of that was them, a lot of classes had to switch a modality, midstream. And students, obviously, outside of IU and I, you were hit with a major disruption and that caused the USU policy chef in grading. So all students, we're given an oxygen. And that was announced in April. And to choose a grade S satisfactory to place their hand on the grade that they otherwise class. And so our purpose is to study, start studying the impact of this unique natural experiment is when we call it that way. So, so one result, when one outcome of this is that everyone who acts. And an important elegant that was, that that policy was that students didn't have described right away in April, but could made their choice up until May 8th and possibly even laid some. May 8th was the mandated universal deadline. The choice, amazed, by the way, habit to be the last exam. The exams, that's an astronaut. But also the University encourage instructors to give students more leeway and give them an opportunity to learn the actual performance before, before making that decision, the choice of anoxia, great option. And so in order to figure out, so we need, we needed an additional piece of information in order to address these questions. In addition to the wonderful systematic data to now, we are getting from our concepts from Greece. So for this one source was candles. Because unlike millimeter assessment and research which tells us the outcomes, great outcomes, cameras can potentially tell us. The student progress within Canvas could tell us, in particular, and so helped could tell us how students are doing by the time they made. And I'll tell you in a second why why this turned out to be not, not so easy. And what we were able to accomplish with less a nation that we're hoping to get so bad. I would like to thank both momentum and research and learning. And the Latvian latter organization is collecting the data from Kant's. So this was a major undertaking because it entailed merging two data sets. Furthermore, merging it in such way that way. Compromise students privacy. So, so the, the two datasets have identified, the students. And the merger was accomplished in such way that we couldn't identify students who question. But we could just be assured that the bar data and the data about students, communities, and Canvas will not work properly matched. And soaps will be the questions that we wanted to address. The address understanding in the first place, understanding the factors and students choices. Well, one might just assume that it's easy that if a student is performing badly, then they're more likely to choose that stands out, is not all that simple, but there are other factors, such as students for months, students demographic characteristics, students year. You could show you one of those results. And in our longer term Go, which is true, It's too early this time around because we've labeled it took us a while to get all the approvals and go get F-measure yesterday. I don't know if you can get a little closer to your bank maybe or aiding you're fading in and out a little bit. That's all right. That's how it yeah. That's better. Thank you. Yeah. Sounds sounds in the distance. Yeah. So so a longer term go is to study the concept of choosing units. Because the choice of s effectively lowered the bar for many students to get a passing grade and to progress to higher level classes. So for instance, someone who would be otherwise getting a C minus re-explain a short time ago, then they wouldn't be able to progress to other other classes. An account. Now the he in that particular semester they could and so and they couldn't. And then the question is what happened as a result? Was that a 104 instruments to pick a pig's the semicolon on that subject matter. It wasn't that important for the subsequent progress. Maybe those students who didn't make it. Well, we did that using, using a time. Maybe they could have done just fine after that. So some, the s option is, is your experiment because it allowed basically everyone to progress. So that's our goal, but who hopefully will help to be able to pursue this next year. Once more information comes then to bar about student performance style throughout. So I very much and spoke to the radius. So we can skip the instrument, which has given us a second to take a look at this slide. And now I'm going to train him. It wasn't as much 20 nights when the policy was managed. Very very soon after the switch on the LED or your resulting from pandemic. So now we try to do, think of the situation. Imagine designation, STM design when making this decision. Should I stick with my grade that have otherwise getting or should I choose x? And sum? When economists talk about trade-offs, they meet costs and benefits. What is a cost to me are making this decision? And what is the benefit? And students performance in the course as well, is obviously relevant to the benefit, right? So, so if my performance was bad, no instance, then it would be beneficial. Um, but my pride performance awesome, That's great deal. Because let's say I used to be an, a student and now I'm getting a B minus and that's terror. Whereas if I was a student all along the spine while it actually connects cells. So these are some of the questions or would like to explore now. But there are also some, some costs may not come to mind immediately. So not all students are always. And I think all of us taught classes and, uh, you know that a lot of students don't know how they're doing. With Canvas or not, how they're doing it in a course in any given moment. So they need to undertake, to take an effort in order to explore that. They need to calculate that gray canvas may be confusing them. You need to interact with the instructor. And so sometimes decisions or to take, as I'm not a tech has maybe predicated on that. One student's willingness to invest in finding out how they're doing. And then thinking, he said, is that for me, it's not good for me hands. And the final fact is, it is what economists would call behavioral. That's, that's what we find. The most interesting, one of the most interesting ones to think. So. So if a student decided to take an S option, declared that aerator, the instructor or not, has decided no matter what. I have the Snopes, I guess. Let's see at the last minute. Well, now I'll actually permitted but I have this now. I know I hadn't stopped. And now that, that option potentially allows me to slack off. And so economists call this type of behavioral hazard, moral hazard. And so this used to be an inside term among economists. But you may recognize it because it was made, made famous during financial crisis of 2008. And let me remind you of the context. Snow. So there was a big prospect of failing companies being bailed out. And sounds for the problem and the Communists and non-economic like in the media spoke about the problem. Bailouts. Bailouts is moral, has that if you know you're going to be bailed out, why would we maintain this? So this is this is a similar situation. So if if i now that I have an option, maybe a SledDog and 10 was gives us a tool to to figure that out. But again, distance, that's something that we're soon. Very soon, but we don't have results yet. But there is this grade on canvas can tell us whether students continued to login to the class into Canvas itself, like or how much time they spent then? Yes, it is one lane. You have about five minutes left here of it. Great. Okay. So in that case, let's move on. Cameras date I already alluded to are that as snow. It details and the the instructor about all the assignments then and tells us children the same thing and grades received ants. And Portugal were found in numerous problems. With data through Canvas, calculates the braid itself, by itself, the score. And so our purpose, one main purpose in using Canvas is trying to infer students actual performed before deciding, deciding which rate data. And so, so this, this is one selection in, in thread column you see the grade that stood natural, ended up choosing move, obviously getting, so obviously these are students who chose not to DNS and then get these blacks close out. The scores that Canvas. We're telling students that we're getting corks in the app. And then you'll see that there is non-marital. So this is just a snapshot of this axle outward, but it goes all the way to two low grades. And we see that there are surprising non-white students with some, some students but a, a plus with the 75 and other students and be managed with this query. So there are some consistencies and inconsistencies in the way Canvas and science course. They, they either wait. All the assignments equally well as instructors have an option to override this. But not all instructions use instructions. So as a result, we, we could only get luminary zones. And so we decided for now not to publicize results based on that Canvas grade. And so we don't weigh album sulfate now, presenting sounds. Without tendency. So strictly using bar state, right? Yeah. And so to form a logical progression, where likelihood of choosing S as opposed to actual rate is independent parent. And so, so here are some, some results. So the higher the price cumulative GPA, the lower the likelihood that the student chose x as a grain. In the first approximation, this may seem obvious. Obviously data, This was high GPAs students. These are better students. So they're probably more likely to have performed better in the course. But remember that these choices are rather, and so this result tells us that the choices are not, not all that well, not all that for relic, that with higher GPA students still, regardless how the new grant related yet they face now Hi Barnard, match. And yet the likelihood of choosing ask is Islam. But so being male increased one month probability of choosing S, controlling for prior UP another, another characteristics. And so in the second specification, generates interacted with pride and willing to pay. And so this result sounds that the higher GPA, the greater the gap between men and women. Okay. So at high grade levels, men more likely than women to choose as risks. With me. Now, some data for raceway ASM has a benchmark and so sounds that white students controlling these are the prior GPA. It's parables are less likely to choose S, for instance, and not much. So receiving Pell Grant being first generation doesn't make much difference in terms of choosing S control for other variables. But residency. So Indiana residents are less than one, less than equals x. Now this does somewhat interesting. So the higher the year and I, you, the more likely students will choose yes. To both specifications come these results and signal. So again, controlling for performance. So the benchmarks here gets a first-year student. And so the higher the year, the more, the higher the incidence of choosing as neutrons, which has entries. And so finally, among these questions, so we broke down, we broke down students into five major academic categories. Education, of course, it's the slowest, but it couldn't be combined with anything else. Other professional schools, scam and social sciences and humanities and both business higher education is a reference point. And so students in stem controlling flow characteristics were less likely to choose S than business tunes. And the same is true for science and humanities. So business Jones, as matter of fact, most by teachers S. So maybe they wanted to progress so bad from that A1 100, two to the rest of their curricula. So it's going to other professional schools. Or the CountSketch were the least likely next step. Now, as I mentioned, and so now hopefully appropriate right now. So our plan going forward is first of all, to try to actually get hold them so better understand the Canvas grade data and then analyze the impact of that on the Canvas grade on students and essential to choose, ask, then to test for moral hazard. So was chosen gets hosing. And identifiable, has questions slacking off after after March 20 nights. And then a beggar and pursued after that is looking at consequences of choosing us. How did how did that affect the progress with students afternoon? Michael. Thank you. Yeah. I'm sorry to have to go. So I just want to open it up for a few minutes before we go over as we take one or two comments or questions, if anybody has anything they'd like to say about this.

Description of the video:

Next is going to be R. Kelly folks. So I don't know how you all plan on doing this, but I'd like to give, you know, I was giving chase the 10 minute, five-minute, two-minute kind of warning so you could keep on track. Who who should I send it to in your group sensors, three of you presenting had not defending that, had Bree. Okay. I'd just be like Okay, presented as bytes. Okay. Thanks, Leslie. Maybe introduce yourselves to again since we do have people from outside the Bloomington campus. So thank you again. All right. Thank here though, I'm lovely, hotter and I'm a Professor of Accounting at the Kelley School of Business. And this study is in collaboration with a brief Joseph I and J Lee and J Lee has been a class bellow for a number of years. So the food, the collaborative grant. And before we continue, I want to thank George and Linda and also the Center for Learning Analytics for this opportunity. Because I know that I personally learned so much from data. I didn't even realize they were available. And be persistent questions within our programs and departments that I didn't realize can be informed by the data. And, and I just learned an incredible amount and I'm excited about continuing increase in that direction. J has been instrumental in helping us learn to access the data and manage big data and complicated and breathe and excellent researcher. And so this collaboration has been fun and rewarding for all of us. On the next slide, we have our agenda. And there we're going to talk about our overall purpose and motivation for this study. We've done many additional exploration. As I mentioned, we're interested in continuing. Then. We'll talk about the research question that we felt that we've asked and answered with the data. Using some logistic analysis and regression type analysis. We're showing association not causality. However. And we're trying to bring to bear statistical significant and q all of our now to help us to draw inferences. Also talk about the data. First, the j will go up at that. And then we'll show you our results. And in proposed intervention that we've thought about, although this is very preliminary. All right, the next slide. So there are overall motivation. So I've then colleague or 19 years and I've noticed from structural shift and the time. And I open them differences between Kelly and other institutions I've taught, which include the University of Texas, Austin, another big state school. And my interests, it is primarily around accounting. This is where we began, although with J on board. And we can extend this to other stem type field. And accounting is a great profession. It's a great profession because It's fairly balanced in terms of its demographics. The American Institute of CPAs indicates that about 49%. So what you would expect, half of accounting graduates nationwide identify as women and about 44 percent identify as non-white and so pro-business on it. Pretty diverse. Upward mobility is provided by the profession because when you're a Certified Public Accountant, you can pretty easily go become an entrepreneur and start your own accounting practice and provide flexibility during life. I know that when I was raising my young children, being an accountant provided that flexibility if I'd still a professional and still proceeding with my career, but have some flexibility for for families. So we we feed accounting of the desirable profession that pays well. And, and it's, and it's pretty diverse. But yet what we've seen at Kelly is that although the enrollments are increasing, so over the last 18 years or so, the accounting majors as a percentage of business majors has been declining dramatically. In fact, I would say we take confound at approximately or 100 accounting majors, even though the enrollment ID which is increasing. So we're disappointed for the county area or disappointed to see that declining market share. And we have an ad hoc basis attributed this to certain structural shifts, including an increase in our direct admit students. I'm an increase in the proportion, perhaps out-of-state students. And we see on a macro basis, but I'm a pretty profound Kelly. Majors are finance majors now. I'm, where's the market share of accounting? And the other major has been declining. But beyond that, we know that even with an accounting or accounting majors that Kelly are not representative of the population of accounting majors as a whole or the larger campus. So at IU Bloomington, 49 percent of students identify as women, 29% are BIPOC. Kelly, 34 percent of students who identify as women and only 12 percent, or by art and accounting major group, an immediate SLP can also be the only 23 percent of accounting majors identify as women to 0.50% as iPod. If you compare that to the nationwide, or they're put out by the American Institute of Certified Public Accountants. You can see that that we seem to have something unique to Kelly as far as our under-represented populations. Now, you'll notice in our title fight that we decided to start at the very beginning with a class called a One 100, and a 108 required of all business majors. And Martha spoke earlier about the prospect of early failure and how early failure is known to result in decreased, persist them in colleges overall. But a 180 kind of a unique course, because if they have Term course and it can be repeated a large number of time, and then the grade can ultimately be removed from the transcript through what's called an extract that and though it shouldn't be a low stake early failure. And I think that p1 hat with a talk with study skills type course. And so it's topically related to your accounting. But I think when, when the course was originally conceived, it was meant to be meant to be a weed out course. It's an indicator to students that they needed to up their game and learn, study skill. And it has a very high DFW rate. So BRI is going to talk a little bit more about the role of a 100 in the curriculum and then j will talk about the data in them. Do I think lesly. So as you can see, the, when we wanted to look at the pathway for our business students and what's happening there, their entry points and exit points. We first started with a Degree Map. And so this is showing the 8 first year generally first year required courses for business majors. And as Lizzie was talking about this, the DFW rates, you can see that a 100 has one of the highest DFW rates of the first year courses that business majors are expected to take. Now there are absolutely some of those math courses that have much higher DFW rates. But there's a variety of different courses the students can choose to take in by Nmap, finite math and calculus. So, so we really wanted to focus on the A100 class. And particularly because it's, it's a one credit hour course with a 31 percent DFW rate. As Leslie said, it is over eight weeks. It's also unique and that I'm not many institutions have an introductory accounting course similar to ours. Many of them will have a sophomore level, a 200 level, full three credit hour sequence. So 2, 3 carrot our courses that students usually taken the sophomore year, um, but IU and Kelly is unique in offering this introductory course. So while a 100 has the opportunity to introduce students to business and in the accounting profession and accounting majors. It also has the potential to deter students. And so that's one of, one of the main reasons we are focusing on a 100s. So our research questions in many ways similar to the pathway report that Chase was just looking at. So we're looking for what predicts, first of all, non progression outcomes in a 100. So what are the factors that demographic factors that are indicative of non progression outcomes? And when I say non progression, we're actually slightly modifying the typical DFW rate because you might have noticed on that earlier slide, students have to earn at least a C or better in order to progress in the Kelly curriculum. So this isn't an added challenge for our students. So we have added onto that DFW rate and said, non progression is any grade that is lower than a C. So we're looking for non progression Atkins and a 100. And then what impact does initial A1 100 non progression have on student indegree and major outcomes? So does it lead to a lower likelihood of a degree at IU, a degree at Kelly. Or a degree in accounting. And then also looking at some of those changes in majors or decisions and major. So if someone comes in initially interested in accounting, that's one of their declared majors and then they have a non progression outcome and a 100. How likely is it that they're going to switch out of accounting? And this is important for us because as Leslie noted, accounting hasn't, we haven't increased our market share her even though we have more students coming in. And so we really are trying to understand, are we losing students from the very eerie coming in at a shortfall or at some point along the way, are we losing those students because of maybe some of our courses? And then finally, as, also, as he noted that this a 100 is topically related to accounting, but more focused on study skills. Is it possible then that students who have a non progression outcome and then repeat the class to take it again. I'm, do they do better the second? So did they actually learn? So maybe it was one of those cases where failure or non progression when you try to stay away from that word, failure. But non progression can teach us here. They can actually learn a lesson from that and then they can take a 100 again and potentially do better than maybe some students you've only taken at once. I'm certainly better than they did the first time. And so potentially that is something good that could come out. I'm sorry. But look to explore what happens if they repeat the class. Now I'm going to turn it over to Jay to talk about our sample selection. Okay? I just unmuted myself. Thank you. Break. So our dataset is from our dataset is from bar, provided we have four sets of data. One way students attributes, one, student retention, which is a, which we mainly use the graduate degree data from student retention. And we have student major History, which is also called program stack. And we have student course history. And the student population goes from fall of 2006 to spring of 2021. And the level of, the level of details in these tables are slightly different. For student attributes, we have Y rule per students, but for retention for major history, we have one rule per student term. If the students existing ie your student records. And for student course history, we have one row per student per term per course. So you can see the amount of data we're trying to dealing with. So when we, we explored how we want to work with data multiple ways. And finally, we decided that we need a main table, a major table that includes all of the IOB students, not just students, because we have students trying to take a 100, even if they are not a kilometer or even if they don't have business as their intended major. So we created a wide table, include all IU students who have ever enrolled in this period of time. And we try to take the level of detail into one mole per student. And we transposed the student's history, their major history, and their core sticking history, not all the causes, only a rounded we transport transposed them from vertical to horizontal because this is the only way we can do our regression analysis. So we go, got it, George. And so we try to keep all the student majors. And the data is complicated because one term, you could have three majors. So we end up with a very white table. But with that one table, we can see the major declaration history up to 12 terms. And we also have all the students attributes merged into one white table. In our research questions, we decided to focus on A1 100. And out of all the students, we have 43,408 students who took the course. And then we have these numbers, students who are still, those who have graduated or dropped out from IU. And these are the number of students who have taken a 100 just one time. Okay? Yep. Yep. I talked about the challenges already and this is the trend that we noticed. So when we talked about the high DFW trend, and in this data we also included c minus, so non progression. And we can see that this is the enrollment of a rounded, this is the count of DFW C minus. And we can see the trend is always have always been high in these 39 represent fall, spring, and summer enrollments. Next, please. And these are the enrollment trends for a 100. This is very similar and we can see that enrollment is still increasing. City and increasing. And non progression reads are indicated by this red color. And you can see the percentage higher, lower, but pretty consistent. And repetition attempts to. So we have students we notified when in our data we only remained students who taken up to three times of 1100 attempts, but we have students who have taken this course up to seven times. And you can see the percentage 8% in fall, 22% in spring. This is really interesting. We don't know why in spring, students have more have much higher rate of re re, attempt. Sku. Yeah, and now you can see the agreed distribution of these up to seven attempt. You can see most students passed with the first attempt, but we do have over 15% of students have to take it twice. And then still lower wise. And these students take several attempts. Did did graduate but he or she is definitely an outlier. So yeah. Do you want to shortly here? Yeah. So this, this will read some of the descriptive statistics for our main samples of all students who have ever enrolled in a 100. And so just a few things to point out. Here. You can see that 33 percent of students who have ever enrolled in A1 100 identify as female and then only about 8% identify as, as URM. If you look down at that, that SAT scores and GPA scores, it's it's fairly high. We have a mean of 1300, quite a large range there. And then I had to high-school GPA of 3, 69. And and many of them enter having at least four, are having on average about 14 credit hours. So this maybe from AP classes on or it might be if they're entering in that from the spring that they've had credit credits from the fall. Just a quick look at some correlation tables. So we wanted to make sure before we went into our regressions, We're on any variables that are highly correlated with each other. And so non progress is going to be our dependent variable in our first year regressions. And then the, you, you didn't see the rest of their independent variables. And so one of the, one of the main associations you can see is that standard emit is highly negatively correlated with SAT and GPA, which this is understandable, obviously understandable. So we've modified some of our regression analysis to focus on standard admits. In some robustness, we do take out the inter-domain, have the SAT and high-school GPA instead of that scene are going. So our first research, research question. So what predicts first-time non progression outcome in a 100s? So we did run a logistic regression. So one is if a student had a C minus or below, so c minus d plus d, f, w, and then a 0 if they did progress with 81, 100. And we ran this regression on a mole to mole to multiple different factors. We're mostly interested in female URM. And then we also want to understand how the standard admit plays into non progression. So you can see our first regression is main effects. And a female, say unfortunately female URM antinode, admit we're all positively associated with non progression. So that's bad news. That means that on average, females are 29 percent more likely to not progress in their first attempt at a 100 than other students and similar for URM. And then our standard admits actually really high. So 51 percent more or 52 percent greater likelihood of not progressing. On our second well, we also had some main effects for resident and first-generation. So what could be entrepreneurs for? Some good news here is that our first-generation students are actually less likely to not progress through A11 100. So then we wanted to look at just some interaction effects. And what we see here is that the interaction of standard admit and female students are actually less likely to not progress through their first-time attempt at a 100 wasn't any significant effect on your ends. And then we looked at the combination of female and your answer, that intersectionality, they were also more likely to not progress. Our second research question then looks at graduation outcomes. So after a 100, our students more likely to get an IUD agree I Kelly degree in an accounting degree. So we looked at three different dependent variables. And the regression that we're using here is also looking at some interaction effects of non progression times female. I'm so let's look at that 1 first. So across the 30 degrees, you see that if you have a female who did not progress in their initial attempt at a 100, the first result of it, they're more likely to get an IUD agree, might seem a bit counter-intuitive. But I think if you combine that with some of our descriptive statistics, that, that on average many of these students are highly capable and they may just have, you know, either it, they may decide that because they've had that first incident of failure that and the business degree isn't for them, but they're still motivated to get an IUD agree. The second one, Kelly degree, you can see that it's negatively associated. So I'm females that don't progress and a 100 on the first attempt are less likely to get it to tell a degree and less likely to get an accounting degree. When we look at though, the URM statistics, The Kelly degree, sorry, There's lots of red lines here. The URM who don't progress are also less likely to get a degree and then it wasn't statistically significant for an accounting degree. And then finally, our standard admits. So standard admits who didn't progress in their initial attempt were less likely to get an IUD agree. Kelly degree in an accounting degree. And I know I'm running out of time here, George. So this is one of our last regressions here. We also want to look at the, if this had an effect on student major decisions. And so we looked at if they change their degree from entry point to exit point. And so you can see here, this is just looking at accounting students. So our students overall so that they entered accounting. If they entered IU and did not declare honey as a major, that would be in the 0. And if at some point along the way they decided to add an accounting degree. So we can say that again, accounting, and then vice versa, if they entered as an accounting student and then did not graduate with an accounting degree, then we consider that Louis accounting. And so we ran a regression would deepen a variable being Louisa County or drop Accounting degree and then gain accounting degree. And, um, for females, don't progress. We did find a lower likelihood of adding accounting. I'm in similar for standard admits, but we didn't find any significance on dropping an accounting degree. So this is interpreted with some, a little bit of good news and that we're not deterring students, but we're certainly not attracting students are encouraging students that have not progressed. And 8100. All right, and then finally our third research question is looking at distance. Who don't progress the first time. Do they obtain higher grades on subsequent are elements. So that this first exploratory analysis we're just showing for students that had, that withdrew and those, those students are tricky because we don't know exactly what grade they would have had if they had completed. So you can see here that the majority of students who do withdrawal on their first attempt, when they, and repeat they come back, they persist in it. About 31% of those make it be we do have 27 percent that successfully finished with an a boat, but we do have those students that 12 percent, again withdraw, 4% have enough and 6% have a dy. So our last analysis we wanted to look at what effect repeating would have on your 8100 grade. And so in this case we're doing an OLS regression where the grade is a numeric version of the letter grade. So 0 to four. And then repeat as our variable of interest. And it's one if a student is taking a 100 for a second or third or fourth time. So you can see here when we ran our first model, when we did our main effects, there, there wasn't a statistical thing. Wasn't a statistically significant correlation with the A1000 grade. I'm going to look at are females and repeat females here. So when we looked at our main effect, females were less likely to have a higher grade, I should say Lesley was negatively associated with an A1 100 grade. But when you look at the repeat females, it's not significant. And then a similar story with are you ORMs. And then with our essays when they repeat the actually, it's negatively associated with their grade. So this, this doesn't seem like great news for those students that are repeating. It's either not associated with a grade, a higher grade, or it's negatively associated for those standard admits. So we, we definitely want to do some more research on this question and, and, and figuring out what's, what's happening. We have some students who the w's do seem to be doing better, but the students who took the class got a grade and then took it again, it doesn't seem like repeating is having an effect on that. So interventions for students success. As Leslie mentioned when this class was started, it was, we hate to use the word but a weed out course. And that's what it was considered and focus on those study skills and so on. Just this past year, one of our faculty members who has been teaching this class for the longest time and doing an excellent job, has retired. And so we pivot to hire new faculty member. And this person was new to IU. And so some of these innervate, these interventions have been implemented. And I want to, I want to say very clearly that the research we were doing and these these changes and interventions were happening simultaneously. And part of it was because we had started looking at this data about a 100 in the spring, right around the time we hired a new person. And so I was actually helping our new faculty member to understand what a 100 is. Understand what I us, and to reframe some of our learning objectives and the assessments that, that combine a 100. So I'm, so I want to be clear, this is not as a result of all of our research that we've been doing, but it's, it's happening simultaneously and it's part of the awareness of understanding that there such high DFA is 0. And I'm sorry George, I see that we need to move on. So I'll leave you with the final thing that this shift has result. We don't we don't have the full data, but for the first eight week results, we we do know that there was a 10 percent DFW rate rather than our historical 30 percent. So that's that's great. News, and thank you. All of this were you used up your pen minute? Q. And a. Oh, I'm so sorry. That's your option, I guess. So we're going to have to move on.

Description of the video:

Okay, So, you know, all of you are obviously here to talk about data-driven curriculum development and other, other cluster and development for students success. I am the Associate Vice Provost for Undergraduate Education, and I've been very interested in these projects essence, I've come on board about a year and a half ago now. And I am really excited to see all of you here, both from on-campus. I'm really looking forward to hearing about your projects. I enjoyed it very much last year. And I also am delighted to see folks from around the country who can hopefully have some time to share their thoughts and questions during the Q&A sessions after each talk. And I just wanted to talk a little bit about my involvement with learning analytics and, and, and sort of communicate where I hope your projects will go in the long-term rate. We're all starting off looking at the numbers. And I, and I, I want to pose a couple of challenges to you and also offer all the support that the various portions of the vice provost office can provide in and helping you as you as you take your data and try to figure out how to make changes that will help students success. So I, I got into it into this area of learning analytics with a project that, that was done by a number of universities through a collaborative called the seismic project that focuses on stem education. And Stefano and Linda's group was really involved in. Stefano and Linder were very strongly involved in this. And I became just in awe of what the good people at bar can do. And you folks who have been involved in this learning analytics understand just as, just as well as I do what they do and just how excited they are to help folks use the data. But there is a, well, we've seen a presentation that basically said that women in science classes have what we call agreed anomaly. They do worse in their science classes. And and this is the grade they have versus their overall GPA and science courses. These are the lecture courses, but they do better in the lab courses, a little bit better in the lab courses. And and so Laura Brown and I, and this is really mostly L4 Brown's work, not mine. Applied for learning analytics grant to try to understand how this was working in in our Department of Chemistry. And one of the things that we discovered was that the issue wasn't so much with women. We found that women in chemistry just about as well as men. Now, That's only true if you don't take an intersectional lens to it. I've recently been analyzing some of our big classes. And what we find is white women do as well or better than white men in our chemistry classes, but that's not the case for women of color, for black and Latino women. And so there's clearly some work to be done. And we don't want to be ignoring the subgroups. And so I just want to talk to you about what I've been learning in this job, about who we teach well in whom we don't include as effectively. And so this, what this chart shows, and I think Stefano would probably cringe that I haven't shown you the ranges there obviously is a big range here, but I wanted to make the data relatively easy to see. It turns out, if we think about societal privilege. So if you are, if you are white or Asian American and you come from affluent family and your parents went to college. Your average GPA and all courses at, at IU is almost 3.4. But for each, for each of those, those privileges that you lack, your GPA goes down. And in fact, if you are black or Latinx, low income, and your parents didn't go to college. Your GPA for all courses on in the whole university is, um, is, is a point, is more than 0.4 points lower. So that's actually a lot when we're talking about this many students. If you're in, if you're in chemistry, that number it chemistry classes, that number can be close to a whole point. And I've seen an even higher in some classes. So what this tells me is that what we do works for some of our students pretty well. And for others of our students it works less well. And everybody's supposed to be getting the same experience here. This is a public institution that's supposed to be helping all of our students. And so I look at these data and I think she's, we have to make some changes. And so I'll pick on my own department a little bit. This is we basically have a three-course sequence that all are pre-med take. And if you get through and if you haven't had great high school chemistry, you can call it started what I'm calling course 0. So that's that, that's an optional course. And what this shows us is the DFW rates to the rate of students who do not succeed in the class who are in a, D, and F or withdraw from the class. And it shows the numbers for what the institution defines as, as under-represented minority students, in other words, black and Latinx students mostly on our campus and a non URM students. And what I hope you can see, hope you can see three things. One is that the numbers are ridiculous, especially. So for instance, if we look at course one here, 25, 25 percent of our, of our non URM students don't make it through the class, but more like 40 percent are under-represented. Minority students don't make it through the class. When you get to course two, those numbers are even bigger. In course 3, they come down a little bit, but they're still nowhere near where we need them to be. And so in Coursera, we also see similar trends. And so we know that if a student gets a single w in any class, it increases their chance of not graduating by 50 percent. If you get a D or an F in any class, it, it, it increases your chance of not graduating by, by a factor of two. Okay. And so, so we're not talking, I'm talking between six to 9%, 6 to 12 percent, but that's still a big difference. So we often think of these things is, yes, some of our students have to figure out how to study. We keep them, but they come back. But it has a real impact and can have a lasting impact on students lives. And the other thing is that we sometimes say, Hey, our job is to figure out who's good at our subject. And then we'll will nurture those students and students who aren't good at it will help them find, will help them understand that they need to go look at something else. If that were the case, what I would expect is very high DFW rates in the first-class and lower DFW rates in subsequent classes. And that's not what we see. All right. So that tells me that our curriculum is not fair and it's also not effective. We are not providing students with what they need in the first course to get to the second course and so forth and so on. So that was a pretty sobering understanding that's gotten us thinking a lot in my department about how to fix this. And I'll just point out that I taught course three for a whole year once. And I would like to tell you that it was 2013. I will tell you that was mostly my colleague, more brown. So she has some really good ideas. But, but this was, but up books I, but I taught in 2011 and I want a teaching award that year. And I got great teaching evaluations. And I thought I was doing my job. And I look at this and I wasn't. So I hope that when we look at our data, we can think about the ways in which what this tells us about what our institution is doing compared to what we would like it to be doing and what we're doing in our classrooms compared to what we would like to be doing. And I'll, I'll skip this slide in the interest of time. I'll actually, I'll just say that, you know, that that's one kind of data you look at. The other kind of data that we looked at is who persists in classes. And this helps to make arguments to deans to invest in things. If you can say Gosh, if, if underserved students were to persist in our client, our sequence at the same rate as as privileged students, we would end up with an extra 10% of students. And then you'd get that RCM money. That's helpful. And for women in chemistry, even though they do better if, if, if I do the same analysis, we would have had 800 more women in that class, which is a 25 percent increase. So these are, these are ways to talk to deans about about where to invest money to help things, and I am happy to help you have those conversations. I love trying to tell deals they need to invest in our courses. And then I want to, I want to point out that there is a department that has had a dramatic success and, and some of this may have started with some learning analytics grants. I see Michael here from Econ, what? Econ used to be one of our worst offenders. It had very high DFW rates. In fact, those rates were 60% higher than our peer institutions in the Big Ten. And so we had our eye else and he kinda what he did was they said we're trying to serve too many people, too many audiences with this one course. And they divided the courses up into a business econ macro and micro sequence and an econ macro and micro sequence. And in the first year of that first year, year and a half of that I think is the first year of that project. The DFW rates went from almost 30 percent to below 10 percent in all four classes. To me, this is, this is a stunning, stunning success and i'm, I'm convinced they've done this right and they haven't just made the classes easier. I think they figured out how to help students learn better in these classes. And, and I hope we will get a chance to hear more from these folks as time goes on. So I'll just close with that and reiterate how much I appreciate you all coming. You're all taking on those of you who are fellows, you're taking on these projects and you're sharing them with the rest of us. And I look forward to hearing more from all of you. Thank you very much, Martha. I appreciate it. I appreciate all the support and you've been giving us as well and the work you're doing by its purpose office, it's an exciting time.

Description of the video:

Perfect. Alright, so I'm going to switch gears and talk about the future in skills which are increasingly becoming currency of the future. And by skills. In this project, we concentrate on heart skills, but it's relevant to hard and soft skills as well in the ideas to map learner expectation of what they expect when they sign up or enroll in our programs to pause or bring in bike worse offering. I mean, well luckily we actually, um, promise for students to want to learn and that maps to workforce need. And by workforce need, we mean any skills that in demands that when you get hired, obviously, it's part of your job description. So this is the main idea of the project. And skill shortage is, has been documented for a long time and it's still expanding. And there's a couple of stats. That one from 2018, one from 2020, one we're looking at, we're even the workers. I were a skill gap. And a lot of people get an affected by skill gap because you cannot advance into the next positions. And also, interestingly enough, in the recent years, there's a preference for a lot of people to go ahead to non-degree programs, certification programs, very short courses we gain those skills to work with versus degree program that we are offering. And there's also potential economic impact that thumb, for instance, on Deloitte predicted by 2028 in the, in the industry, given the shortage of skills necessary. So it's $2.6 trillion for example. And they also predicted 2.5 million job will be unfilled by 2028 and their industry. But it's a huge number in Definitely there's increasing skill gap between educational offering in workforce needs and that's the, the main subject of this project. So there's a challenges that been also recently pointed out too, that we as academia was speaking a different language compared to, for example, employee. So we offer a degree, but employee six, skills and proficiencies. And Mz as labor market platform, they provide a lot of studies and resources on skills and workforces. So they identify the main challenge, current main challenge that we can actually fix. And so there's a need for skilled base design approach in our course curricula that kind of map a learner to employer, right? And we kind of mediators. And just for those who never heard that word, scarification. So scalar phi means to translate curriculum content into the skill based language. So that's, I guess a word or 20, 22 is going to be scarification of your syllabus. So imagine if anyone who get enrolled into your program and looking for specific career, right, you can present only out this kind of pathway map saying that if you look in for this specific job, for example, of front web developer. So you need to gain or increase proficiency and CSS, HTML, JavaScript, but not so much in Java. And advisor may actually build specific path with a student. Where are some courses will be more relevant or not? So this is idea of presenting or mapping skills from workforce to what we offering while we're teaching and also using the labor market data point. The same way that there's some skills are more, have more weight, but also there's more jobs with those skills. For instance, in ideally, as we discuss, for instance, with DAS online program. So what they would like to have is to be able to tell students, for instance, that you own. Let's say that you take this course and gives you this course gives you the skills. And these skills are in demand for this job basically. And that's specifically to online data science program that LP from now on focusing on and of course that can be layout for other non online program. But in online program it's a little bit more specific and I'll tell you why it's more specific. So in terms of objectives, so what we'd like to do is to look at students expectation first and why they attaining or getting enrolled in the specific online data science courses. Then we'll look at or NSS online data courses in terms of syllabus. In this case, since it's pilots, I'm just look. Courses I'm teaching, so I have access to my own syllabi and then align or just first look at workforce and what skills are there so we can align and then develop a way of incorporating this type of information and scarification or information. And finally, develop some sort of tool that would map student learning with the skills offered in, obviously labor market skills in this project is a continuation from last year project where we looked specifically at DataCamp as a platform to provide students with the actual hands-on skills for data science and how to train them more souls, especially Given that this is online course, we don't have any labs that where we can meet and teach that way. So that was a way to introduce students. So partial information gained from that project goes as a foundation for this, for this kind of second stage of research. And for datum, there is some data already available, some data we're still looking at. So it's like in progress project. But the data that currently we're looking at is MZ job ads, and that says workforce information, job description that you can find online. And our typical students go online to look for jobs so they know how to obviously look and not all of them know how to match their skills in resume, but that's another interesting aspect that we can provide them so they know exactly what to put on their resume to match exactly their employer needs. Then we also look at course curricula, which will be just what skills offered and student expectation and goals will come from or coming from. Actually, discussion posts and online questionnaire that we conducted at the end of the semester or throughout the semester. And we'll also have beta student's resume, which currently we haven't looked at. However, the questions that were built in discussion post a sort of are present, we would gain from resume, but it's slightly lighter manual work because in the student resumes you can expect to have to parse and try to scrape all information. So a normal kind of focus on discussion post questionnaire, a actual data from web bone, actual data, web data from job postings, and also of course, curricula. So if you think of how can we classify our students in data science? So from last year, we found, based on their descriptions, that there's actually three kinds of students or expectations, or I would say three types of students, why they take this course. Or one type is when he liked to change career. And since data science, it's very popular topic in a job market. If you look at it's constantly increasing number from didn't need and job market. So change in career, it's one type. There are some students who come just to learn new field. Again, data science, that's relatively new field. And there's also students who are looking for just upskilling their current career. So that's all came from pilot projects from last year. If we look at our current student profile and we kind of expanded little bit, what type of students we might have. And this is based off the fall semester. We didn't actually include that question or in last year to compare. So I'll start with this Fall 2021 semester. So in our current study, we have students who are full professional. That's the majority of our students. Then we have PhD student under 10 percent in also residential pool time students. However, these courses are built as online program, online data science program. What's interesting is, is their needs are expected. Are there the reasoning why they join this program? So if you noticed, we have improving apply skills. So that's relevant to progress skilling type from last year, change in career. So it's the same, but also getting promotion, career advancement, sort of upskilling that bring some benefit to. And also there's another term we included and actually we got 30 percent who went to say that personal interests in other reason why they join this program. Quite interesting. And so it's about the same, similar to three types that we discover last year. We just don't. Gating promotion period advancement is addition to upskilling. So in terms of data scientist, for definition, I would say, it's interesting. This term is quite recent and. The data that we gain from labor market program platform does not have built-in yet the, the SOC identification number, but a description exist, right? But we actually unfortunately have to go with data scientists and mathematical science occupation together to try and to extract what skills are actually in data description. So it slightly, I would say this is my major weakness because we're including other data science occupation or a mathematical occupation tip. So if you look at national level currently, very good. Actually, it's really good to be data scientists nowadays rate because it's going up and it's about 100 thousand compensation. Of course this is average, right? In There's on-demand, so there's three job postings per each need. So it's kind of It's a need, basically, it's a good skill to a good term occupation to have. So if you look at specific data science hard skills, and that's what we actually need in order to skill a skill if you scaly phi the syllabus. So there's a three or four different numb. You notice some tendencies or trends that comes around. So we have SQL programming language and that's what we provide specific course to it. We have Python course, machine learning, R, programming language. These are majority of and there's like a top skills that everyday decides it should have in their scientific or let's say skew, skew belt, tool belt. If you actually ask current students what are their prior skills before they enroll into the program. Which is very interesting because 40 out of foam 58 and this was multiple choice, multiple answer choice. So they have Python, SQL are almost like in the same order as you would see in trending kind of on online. It's kind of interesting and a lot of students already came with this belt or with this tool set. So what else can we teach them to keep them interested and engaged in data program, online program? So we went a little bit deeper to look at more specific hard skills. And that's what MZ allow you to dig a little deeper, deeper for specific skills. Where, for instance, you start looking for scale alum, NLTK. This is just a library specific library in Python. Then it comes with What other skills listed in this job description. So it's kind of useful to pool information that you may build. Like what other libraries or packages or concept needs to be edits because they ask a lot in this type of job description. So after collecting and we still going through some, there's a lot of information actually to process and a lot of foam. Obviously, you notice number of posting or defer every time you ask for it. You can tell that deep learning is more popular. There's 5000 postings right right now that ask that skill compared to just and LP package, there's a smaller number of things, but it's all relevant and interesting information. So how did we go about trying to kill ify, the syllabus? So we started from actually literally adding the everywhere in the syllabus. What skills, what skills are offered? In, what concepts? It's kind of similar to everyone syllabus, but now everywhere there's specific place. It tells you what hard skills are, can be mapped actually to specific jobs or just in general to jobs. And also in them before class started, before Class that started this fall every longer ask what are their current skills and what they hope expectations to learn so we can kind of see what expectations actually are and compare it to the course offerings. And also every week, we're actually adding, again, hard skills that students were specifically training. In. Another hope was that students start realizing that they're not just attending claws, but they're learning skills. So kinda get translated into their future resume. So they wouldn't just say I took data science courses at IU, but they would say, I learn those hard skills in I'm ready to work at your company. So and we were also collecting some suggestions and feedback throughout the whole semester just to see how students were going through this type of scarification. So in we included data camp, of course this is continuation from past project. But what we are looking into in this is just a screenshot from data camp platform where when you assign them, when you're assigned the courses, like specific. Skew courses. Then you can see how many students are completed on time. And then later on you can see how well they did. So later on because it some can be linked to username. The hope is with the bar data, we can link those two to their overall course and overall program participation to see how well that help them or not to do the sequence through other courses. So, and we completed this semester with this discussion survey, where students are invited to first of all, complete anonymous survey, what they learn, what they expected to learn. And also, another question is what they find interesting and things that they learned. So again, we try to prompt them to share things that they learn in order to, with the hope that it stays with them and it gets translated and their future resume and then obviously job, job market. So this is example how we were collecting manually on building our data collection from student expectation because we don't have a, obviously I'm way to do it right now since we're dealing with a text to do it automatically. So we just had to go through in order respond and try to identify summary. Like what are the things that they learn? What's the most interesting things they learn, for example. And this is our result from both courses right now. And what's interesting and kind of very pleasing, at least at this point, is to see that skill learned for each course can be actually maps to what we provided in weekly overview and actually work with that throughout the weeks and the workshops and data cam, which is which is good. Because well, when we started this, it was no a clear picture where there'd be able to tell exactly what skills they learn. So this is our, we call it matching course offering with the students actual learning. And then the next couple of steps still in the process. So the next step will be to add now at the bar data and look at students progress sequences over their own program. And again, we'll be focusing more on actual online data science student, full-time professional. As you notice, that's the majority of students. We will be excluding the PhD students or any residential students because we're just interested in this bulk of fall population right now. And so the whole idea is to actually to skill if i and show that it works in, perhaps help us to bridge that gap between the course offerings and skills and needs a job markets. And also another reason besides aligning up program with market needs is to actually market courses to students. H4 will understand the value of the course of degree program versus non-degree program. Again, if you remember, the enrollment kind of goes down in degree program and goes up currently in non-degree short courses just for certification. And also, there's something's missing currently is engaging employers. But there will be another great step to bring anybody who like in Indianapolis area for instance, with data science company what exactly skills they have in their job description or what they are looking into in the future to bring perhaps soft skill development criteria. Because right now it's hard to gain from job description Other than, would say, we need communication skills or leadership skills because we're just focusing on hard skills. And obviously, that's, the aim is to equip students to market their skills. If you look actually at LinkedIn profile of a lot of students and map them to dump market skills. And that's what Amazon actually allows you to do. You notice that students do not put right skills in their resumes because they just don't know that language. And by skill, if you kind of skill, we find our syllabus, right. We're kind of adding that another plus two students resume so that they'll be able to be found. So that HR will be able to find them much easier because they speak that language of the skills basically. So that's another plus, I guess if we able to do that, would be so that's that conclude I try to be as brief as I can, but that concludes the presentation. Think we're given this great opportunity.