Turning a Flexible Assessment Approach into a Canvas Integration

Alright. So, hi everybody. My name is Gabe. Real Smith, and I'm an analyst at the University of British Columbia in Vancouver. And today, I'm gonna be talking to about an integration that we created that is based on an instructor's approach to flexible assessment. Go over what that is.

So I'll just briefly talk about the context, what flexible assessment means the process by which people are doing this, a live demonstration of our tool some of the future plans, and then we'll have time for questions. Alright. So just a little bit about the University of British Columbia. It was founded in nineteen o eight. We have campuses in Vancouver and colon We have seven over seventy thousand students, and we're ranked in the top three universities in Canada.

And I work for the faculty of land and food systems. And it was founded in nineteen fourteen as the faculty of agriculture, And we have just a little over two thousand students, and also ranked in the top three for agriculture in Canada. And, like, as you may notice, two thousand is a pretty small portion of the seventy thousand students that we have at the university as a whole. But what that allows us to do as an instructional support unit, which is what I work for. It allows us to get a lot more in-depth with what our instructors are doing.

And we can find solutions that are really specific to them. So it's kind of exciting. The funding for this project, I wanted to mention, our university has internal funding for teaching and learning that come from the students tuition. And, I don't know. Has anybody here heard of WebCT? How many people? Hands, how many people have heard of WebCT? Okay.

Quite actually not as many as I thought. But, this project was actually funded the same fund that created WebCT. So, this project really wouldn't have happened if I didn't work exactly where I work. Okay. So let's talk about what flexible assessment might be.

So this is a mean, flexible learning and flexible assessment can mean many different things to many different people. And, was kind of a good definition that I found from a professor at, West Virginia University, and it just kind of summarizes everything. Faxible assessment is about students participating in their own learning. It's about learners making informed choices on the manner and form of their performance. That demonstrates how well they have learned and how well they can apply that learning in a variety of context.

Flexible assessment is about giving students the assessment options that they believe will demonstrate their learning in the best light possible. But what are some flexible assessment strategies? So it can mean a lot of different things. So there are a lot of different strategies. Here are three common categories. So flexibility within timelines, flexibility in weighting assigned sense and flexibility in the format of assignments.

So some examples of flex timelines would be things that allow students to complete a course or to submit assignments they choose. So, for example, students, deciding due dates in advance at the beginning of a worse or late tokens, get out of jail free cards to be able to submit things late. And we've heard of these things or, oh, I just wanna wave our head. So, flexibility in waiting. So a few different examples of that.

You can drop the lowest score on a series of quizzes, or you can shift the weight of assignment onto another, or you can allow students to decide how their final, the final grade will be weighted. And that's actually what we're going to be talking about today. Examples of flexibility and format would be things like students deciding what type of activity to do or students proposing a final project. Pros and cons. So, there are some obvious benefits of these student centered approaches, like accommodating students with special needs or illnesses.

And, it can give students a more personalized learning experience, it can teach them self regulation skills like organization and planning ahead and self control. And what we've found is that even if students don't want to make the choices, or they don't want to use the options, they still like having the options. So, Nick again, none of this is probably earth shattering information for you. Some of the cons. So it's not sunshine and roses.

If you give them too much choice, it can be overwhelming. And if you are too flexible, like, allowing students to put a hundred percent of their final grade on the final exam or decide how their grades will be way at the end of the course, this can result in delayed effort. And there's some research that if you if you scan that QR code, you can read a little bit more about that research. Another interesting thing that someone found was that if you have weekly quizzes and you allow students to drop their lowest score, they actually did worse on the final exam than students who didn't have that option. So that so that was kind of when it went into, one of our instructors choice of what kind of flexible assessment she was going to use.

And for the instructor, you know, giving students all of these choices can be very time consuming, administratively. Like, if you're allowing them to do a whole bunch of different projects, and you have to make make a rubric for all those projects potentially, or it's hard to compare the students. So can kind of increase the marking time a bit. Okay. So allowing students to choose how their final grade will be weighted was somewhat of a complicated process.

So this is the original process that student that's that instructors were using to collect the desired weights from students and apply them to the final grade. So the green boxes are the student's actions, and the red boxes are the instructor's actions. And below each box, you can see there's different tools, and the people that, the people that use this approach, we're using it in different weight. So some people were collecting the students' desired weights with a Canvas survey or a Qualetric survey. Some of them had the students emailing them.

So they had over a hundred students emailing them. I want these percentages. Yeah. That happened. So like, again, you you can see that it's it's it could be very time consuming.

And then once the assessments are done, they export to excel, apply a formula to calculate the final grades and then import them into our SIS and optionally into Canvas, but most people, were not importing the grades the final grades back into Canvas. So you can see that they're switching between systems from email or online surveys to Excel to Canvas and back to Excel again. And can be time consuming depending on how good you are with those technologies. So somebody who's used to doing this and, has done it many times is actually can do it quickly, and it only takes a few hours. But some people don't have a few hours, as we all know.

So, this is the way that we so the purpose of the tool that we created was to streamline and consolidate the whole process to one system, which will be accessed through, an integration with Canvas. And we definitely didn't wanna create, like, one more system, outside of canvas because, there's already too many of to deal with, and let's face it. Most of the course activity is going on in canvas. So there is a general preference for things that are integrated with it. It just makes everything a lot easier and more convenient.

So with our app, the instructor uses the app to decide what grades are flexible and how flexible they will be. The students enter their choices or not. And then once all the grading is done, it applies the student's desired weights to their final grade, and then the instructor can import it. They can export it, rather, to a spreadsheet or press a button to board it to the Canvas Gradebook. So I'm gonna show you what it looks like.

It's like the moment of truth. This the internet doesn't work. Okay. So we have a blank course, and I have to find my cursor here. And I can click on flexible assessment here.

And so we have some instructions, on the front page. And we've actually redesigned this in the new version. So at the top here, we have some welcome instructions for the students. So the instructor can personalized that message that appears at the top. So if they wanna give them, like, specific ideas about what to do, And then we have this assessments area.

So if I press assess plus assessment, I can enter my first assessment I'll do quizzes. I'll do a default of twenty. Minimum zero and maximum thirty. And, let's do midterm. I'll just do four just to kind of Oh, did I spell that right? Okay.

It's hard to see from here. Okay. So the default percentage is is sort of like the ended. And that's what students get if they don't choose. So if they don't choose their weights, they get the defaults.

And actually, like a high proportions of them don't choose. We found it's about thirty eight percent don't choose. So they just go with what the instructor recommends. K. So midterm, I'll say twenty and forty.

And final exam. I will do forty. Where is that gonna go over? Forty. And I'm not gonna make this one flexible, actually. So I'll enter the same minimum as maximum.

And then I will have an optional assignment. Okay. My total's ninety, so I have to yeah. So it doesn't let you continue if you don't have hundred there because we really wanted to make it kind of foolproof. So you have to attempt for every assignment or assignment room? Yes.

So you have to yeah. So the question was, you have to account for every assignment and assignment group. These are these are actually the assignment groups. So that's where it pulls the grades from. And that's off that's because oftentimes people will group the quizzes together.

They might do like a drop the lowest, and they they may not make those individually flexible. Like, quiz one is flexible, quiz two is flexible. They usually wanna do that. Yeah. Well, this then populates the groups into canvas.

It is clear that we need to create the group and then Okay. So you the question was, will it create the assignment groups? No, it will not. And the reason we didn't do that was because if somebody uses it one year and then they use it the next year, it'll create duplicates. So then how does it So they do they have to go and set up their assignment groups in assignments -- Okay. -- and then match them.

Has to be established or recorded. Actually, it doesn't work that way, but I'll I'll get to that. Yeah. Okay. So let me get through this.

Okay. So now we have like comment instructions. So, like, please enter your reasons for the choice made. So there's a default there. But, I mean, you can, the instructor can can customize that.

Like, I have an instructor who has, like, weekly, she this, like, activity that she does where they opt into these discussions. So for her, she asked them about which discussions they wanna participate in, and they put it they put it in there. And then we have our dates. So these are the dates, by when the students have to submit. So you have to have a start date and an end date because we recommend having an end date.

Because you don't want them to be able to change their grades or their weights rather after they've already, like, failed every So, yeah. So there's definitely, like, a, a negative effect there if you're too flexible. So once I save it, it takes me this screen, and it tells me, like, how many students have submitted this, we've actually changed these to pie charts because the stacked bar and the colors are kinda ugly. But, it just tells you like, what your defaults are, and then what the students chose, in relation to those defaults. So now I'm gonna show it from the student view.

And, surprisingly, it does work with student view in Canvas, which I was kind of amazed by because some of our integrations don't work with it. Alright. So I'm in the student view now, and I click on flexible assessment. And there's not a lot to do, from the student perspective. They really just enter what weights are.

So let's say let's say I'm just gonna opt out of quizzes. Let's see if it'll let me do that. Midterm, I'll do forty. Final exam is not flexible, so I have to do forty. I'll okay.

So the question was what happens if you try to do others? Okay. So I'll try that. So, It's adds up to one thirty c. So, when you submit Please check this box if you want to receive. Okay.

So you have to agree that your final grade will be calculated this way. And it also there's there there's also like a red marker here. I think we did have a tooltip at one point. Maybe that's that's prioritized or something. Okay.

So it's at the top here. So total flex has to add up to a hundred. So you changed midterms of ten, I would make it still be sorry. Sorry. It's it's gonna stop me again if I do what? If I change midterm to ten now.

No. Assignments will still be fifty. Okay. So ignment. No.

You can't do fifty because the max of that one is twenty. Right. But you had fifty and there was no message that Oh, really? Okay. So it's these messages, I guess, are prioritized. So I guess that's something that we have to look at is having multiple error messages, like, you didn't check this.

You went over this. Or, So I think it's just prioritizing it right now. So that's probably just like a, oversight. But, basically, it does not let you submit unless you're within these ranges, and it adds up to a hundred. Okay.

Let's try this again. Oh, I think I have to do twenty here. Okay. And then I just submit. And then it shows me when the student goes back to this page.

It shows them their weights and, well, like, what they chose. And, if they wanna make changes, they can go back and they can go back to assessments. And they can change it again before the date. Once the date has passed, it will just show them, this screen here. And that's it for the students.

So, next, I'm gonna show you what happens after the students have submitted and all the grades are in. So I'll go to this course. Again, flexible assessment. So this one See, they've got they've got some submissions here, three out of four students. And, you can see that the student average is a little lower for this one than than the default.

So then I can go to assessments so I could see what I set up. So it's kinda similar to the one I set up before where like this one is not flexible. The rest of them are. And on this page, I can see the student choices. So I can see if they have comments here, I can see what each student chose.

So some students might wanna distribute everything equally this one. Some students might wanna make the midterm worth more. And it really depends on their strengths. Yep. Way for the faculty member to disapprove of the student's chosen way to, you know, like, for example, you have to tell me why you want this, and they don't like the reason their own what a mechanism that the faculty member can sort of send it back to this means they try to be.

Okay. So the question was, is there a mechanism by which this BN the instructor can reject what the student's choices are. So, no, because if they don't like, like, because this the instructor's defining ranges. So they're not gonna put it in a range that they wouldn't accept. But this the instructor can actually go in and click on the student's name, and they can change it if they need to later.

No. No. But yeah. Again, it's new software. So we we still have things that but thank you for that because, like, you're giving me really ideas.

Okay. So, yeah, that comment I don't think I repeated it was, Sorry. I Do the students get notified if, this the instructor changes these, and they do not? Yet. Okay. So now I'm gonna go to final grades.

And this is the page where you match your assignment group. So so there is some setup within the assignments tools. So you do have to set up your assignment groups in there. And you have to drag your assignments into those groups because it takes the grades from the assignment groups. So, this one's already matched up, but you can see like, you know, you can't select, like, the same one.

Yeah. So there's there's a little bit of, like, foolproofing here. And, yeah, I mean, it tells you if there's not enough groups here, like, you have to create one. Then I go to continue, once I've matched up my groups. And now it tells me, first of all, the override total.

So we use we another thing that this the instructor has to because the app couldn't wasn't able to do it because there's not an API for it, was the final grade override. So when you export the grades from here, it goes into final grade override column in in the grade book in Canvas. So it gives you the override total. It gives you the default total. So if the students had chosen the default, this is the this is what they would have gotten, and the difference between them.

So you can export these to a spreadsheet. So So some people might not want to submit it to Canvas, and the spreadsheet looks pretty much exactly the same as this. So they can calculate it themselves. They can check it if they need to. And then once verified, you can submit the grades to canvas.

Okay. So now when I go back to my grade book, I can see that the override column has those grades. And the student doesn't see the default. So they don't see what they would have gotten. But they would have been able they would they would be able to calculate it if they wanted to.

What's up? Do you have council students, like, filing fields saying that they could have earned a better grade? That's yes. So the question was, we had problems with students filing appeals if they could have received a better grade. So on the student page, there was a checkbox there that said, I agree that my grade will be calculated in this way. So it's sort of like a contract. So we haven't had that problem now.

We haven't had anybody challenge it, at the back there. It depends on the so the question is, if the students what kind of information they're provided with to be able to make an informed decision? So, It really depends on the instructor, like, how much direction they wanna provide to the student. Like, really think about your choices, think about how much you want this assignment to be worth. Maybe think about like your strengths. So it really depends on the instructor and like what they wanna put on their syllabus.

With the override scores. I I'm not familiar with the what if grades. Because there's no weighted percentage in the assignment for in Unless they also set up the new folks. Yeah. It would you it wouldn't interact with that feature.

I don't think. Because, I mean, I don't think think we have that. So I I I haven't heard of that honestly, the the what upgrades thing. So a student could actually Okay. What if I get a hundred percent of the essay yet? Okay.

So if the assignment hasn't been graded yet, what if I get a hundred percent. Okay. Interesting. Okay. I'll have to look at that.

Okay. Okay. Sorry. In the, right, in the, like in door syllabus, or you go to silence, if you create your So are the fact that the fact that have a clear percentage is in campus? Okay. So, you asked if they were putting percentages in the assignment groups in Canvas, and it was matched.

It was matched. No. It doesn't work that way. So, those weights get disabled when they set up the, when they set up their, flexible assessment because The problem with that is that, like, when students go into grades, they see the default percentages. So they don't want them to see so they disable that feature in, in assignments.

How does it affect the what if? Okay. I I don't know. I'll have like, I'm gonna post my, information at the end of this. So you can ask me those questions again, and I'll be able to look up the answers. Yep.

To clarify based on that on the outside of things when it's displaying the that that exportable file and where it's default and that's not the other. Is it calculating that default internally or is it pulling this total from canvas then to populate that column on it on your application side? To populate the default column on the application, it uses the defaults that the instructor put in the when they set it up. And then it's calculating based on the Those defaults. Yep. Hey, should I should I continue? It's okay.

I'm glad people have questions. Okay. Oh, I have to switch back to my Oh, okay. Okay. It's switched back from me.

Okay. Okay. So some future plans. So, We have some cosmetic upgrades, like, a bit more help text. That error thing that you guys noticed today, I'm totally gonna look at that.

We wanna pilot it with more instructors. Like, so far, just one instructor has used it. He had seventy five students and didn't have any issues, so that's good. But, we have another instructor using it now for the summer term. And then in the fall, we have, I think, about five who are interested.

So be interesting to see what happens. Another thing that I wanted to point out is, like, the need for the ongoing audit and maintenance because a lot of people get these projects funded, and they finish them, and then they just leave or the they don't have any more money for developers. So, yeah, it's really important to plan for that maintenance. And we have a pretty big team. So I just wanted to give a special thanks to my team.

And as you can see, we have three three developers on our team. So that's like definitely really useful to have when you create stuff like this. Okay. More questions. I've got time.

Do you have any discoveries that show interesting. Okay. So I've I've presented it in so the question was, how many instructors have showed interest? So I've presented this internally twice. And, one was at like a poster session, and, there was a whole bunch of concurrent going on, and it was online. And ours had twenty or thirty, and it the other ones had like two or three.

So I think there was some interest there. And, like, t tends to be when I post my contact info and I suddenly get all these pings. So I think that's probably a good sign. Yeah. So any input on student from students.

So, the the professor who came up with the approach has asked students this question on her course evaluations every year. And what she found is that they like to have the choice. Even if they're not, even if they don't necessarily use it. And another interesting thing that she found was that it it was increasing over the years. So that strongly agree that I I like this increased year by year, which was kind interesting.

So I I mean, I wonder if that means, like, students are wanting more flexibility now or expecting it. Yeah. Is that kind of where this whole idea originated from with that single professor that found Yes. So the question was, did the idea originate from that one professor? And yes, it originate from that one professor because there she she had presented it a few times. And, people really liked the approach, but they felt like it was just too much work because, like, collecting all the percentages on the students, and then you have to use a and, people are really overloaded.

I mean, professors work an average of, like, sixty hours a week or something. So so, yeah, it was that one professor. And then, some of her colleagues also, like, were using it as well. Her content, one her one of her courses is international nutrition. But we have two, two professor streams at our university.

So we have professors of research. And then we have restors of teaching. And she's a professor of teaching. Yep. Yep.

What's up? When you're in, from like, organizational perspective, was there like, oversight policy concerns about this sort of big, unique way of breaking during pushback from Provo level or at that level about doing this kind of grading or is it kinda long under the radar at least for now? Actually, it's interesting. So the question was, was there pushback, like, from higher ups? So, we like, every technology that we implement has to go through a see impact assessment. And I'm sure, like, people are familiar with that whole process of, like, vetting your, integrations. And when you create an integration, it has to go through that too. But, the interesting thing about this, system is that it actually doesn't store anything.

So it got approved automatically. Credit. Was there any concerns about recorded, you know, the instructors discretionary great, how they great? Yeah. So the question was about, like, there's pushback about grading policies. So, instructors at our university, like, they really have a lot of freedom in that area so they're they can grade the way that they wanna grade.

Like, we have people who, like, do un grading and stuff like that. What's up? Okay. Okay. So the question is, do they come out with, like, a higher or lower grade? It, like, it's benefiting them. So, what the professor who's been doing this for for quite a few years, found was that there wasn't significant difference in the final grade between the default and what they chose.

But the positive effect was more about the student satisfaction. Yeah. Now to your faculty, would choosing this to make sure that their goals and objectives aren't all in one grade category, but the student might not fit. Okay. So the question was, do I counsel faculty on this, to see if they, to make sure that they're learning objects, they're learning objectives in the course are covered, so that students can't just, like, opt out and then miss one.

So so far I've been meeting with everybody who's used this, and that does come up. So, like, you know, what if I give zero to everything? Like, as a minimum. So they can just opt out of, like, everything. Like, what if I just use minimum zero, maximum one hundred for every assignment? And, yeah, it's not recommended because, you have to obviously wanna make sure that you that they are accomplishing your learning objectives. So they're opting out of things, then they're missing, they're missing any content that they need to know, then then you need to not put zero for those ones.

The original stuff you were inquiring about how many students would they have in the veterinary class would you say? The original Okay. So the original instructor who did it, had three hundred students. And then, she has two glasses. So she had three hundred students, and then she had, like, I I think the other one might be a hundred. You said the initial staff, like, sixty percent of the students opted to Yeah.

So the the question was, like, the percentage that that opted to do it. The the ones who opted out worth was thirty eight percent. Yes. So sixty two percent wanted flexibility or wanted to choose. And, I mean, I guess there's nothing stopping them from choosing the defaults.

So if they didn't literally pay attention, they they could actually do that as well. Okay. So the question was when does this deinstructure allow this? I it's usually before the add drop date. But then, you know, they're students who come in late. So they still want them to be able to choose.

So it Yeah. I think it varies. I'm very happy to follow-up with this professor all afterward that she's made a changes to the way that she Okay. So the question was, like, if if the instructor changed her approach based on, using this strategy, I I'm not sure, actually. I don't know.

Yes. She is. Yes. So that you can email me and I'll I'll give you her information if she if she wants me to give it to you. There any insights gathered into those students who didn't make that change or who selected the default, but why didn't you take this opportunity? Do we know what's going on? Yes.

So why are students choosing this? Why are students not choosing it? Okay. So the question was, why are students choosing or not choosing? They're not choosing, and we we actually did some focus groups with students. And, they some of them, they like, it's just whelming for them. And, they feel that it's too much responsibility, and they feel like, okay, if I make the wrong choice, then it's my fault. So and then some of them say it, like, you know, I trust a professor to decide what the best weight is for my my assignments.

And, for the ones who do choose, I mean, yeah, it really depends on how flexible it is. So, like, let's say if the person's not very good at quizzes, they might, like, opt of that, like, if they have, like, test anxiety or something. So, yeah, it's I think it's good for that as well. Okay. Want to agree things that they can try over and over to get our hair and agree.

Like, they wanna wait that time to retire and wait and according to that name and only be able to open up your client's lawyer. Or do you or do do they even know your quote's weight that Okay. So the question was about, like, summative assessments. And if, students opt to make those worth more because they can keep trying them over and over to get a better grade. I don't know that question.

But I would imagine that they probably would because if they can if they can try over and over to get like a a very high grade. They're probably gonna wait that one higher. But I I I think possibly that the, instructor doesn't make that one flex cool. Yeah. Any other questions? And maybe I'll go back to that slide actually.

If you wanna read the research that, the professor did, it's it's this link right here. So this QR code will give you a link to, Candace Ride's research, and she is the instructor. Yep. Yeah. And I can show you, actually, a since we have time.

I can show you, let's see if they have this here. I can show you what her syllabus actually looks like. So, this is what her syllabus looks like. And, you can see she actually lets she she has the proposed value there. And then she has ranges.

So you can see some of them, like, she doesn't let them opt out of. So she has a final exam that's cumulative. So she does she doesn't let them opt out of that. And, they can opt out of, like, most of it, actually. But again, it still has to add up.

So you still have to do some work in the course. But, what she found was that people were not opting out of things. Like, students were actually choosing to do everything. And, she mentioned that that assignment in the middle actually Quay involved, so she was kind of surprised about that. So, yeah, that's what her syllabus looks like.

So it might give you some ideas. How long has she been doing it? It will be in that article that I, referenced. It'll it'll tell that, but it's been like several years. I don't know exactly how long, but it's been a while. Yeah.

She she might direct you to me, but that's okay. This at the beginning. Is this something that someone could grab the code on github and actually Friday to use? Okay. So so the question was, is this something that people can grab the go the code on hub and use right now. And yes, it is.

So if you wanna use this, our our GitHub will be here. So what we'll probably be doing is, like, updates, like, not during the term. So, like, like, end of December. Like, summer, that kind of thing. And, yeah, we are planning to do, like, regular updates because there's always stuff that you don't think of.

And, little things that come up, like, in the middle of a in in the middle of a conference presentation. Like, like, yeah, that having multiple errors at the same time like it should show you all of the errors. It shouldn't just like prioritize one over the other, which is what it's doing now. So that's kind of interesting. So, yeah, the little things like that come up because it doesn't do a lot, but it it it's a little bit complicated what it does.

So, There was there was a lot of little things, like, one of the thing one of the things was, like, if the student didn't that we used to have an issue was if the student didn't choose, like, they would go to that page and it would say your weights. So it, like, it didn't say, like, you did not choose. So, like, little things that you don't really think of when you're creating software. Like, I've really learned a lot about creating software with this project. Okay.

Anybody else? Like, yeah, any feel free to come up to me, like, later. I'm happy to answer questions about this. I I may not be able to answer, like, super technical questions because I'm I'm not a developer, but, thank you guys so much. For being such a, active audience. Alright.

Hey. Thanks again to Gabriel for that great presentation. Just a heads up. If you can, please use the app to rate this session and fill out the survey. It gives us really good feedback for their future infrastructure cons. And if you guys want a QR code for this session, it'll be available at the swag booth down in the arug haul tonight. Alright? Thanks a lot guys.