A Smooth Transition to New Quizzes

Welcome everyone to a smooth transition to new quizzes. AK, the, Casey and Sunshine band pre party. I was gonna play their music and figured, you know, I could just give them the royalty money when I see them this afternoon, but, hope everyone's having a great instructure con. I'm really happy to be here and, be in person again. So this is the new quizzes session in case you're looking for a different one. Let's get started here.

So this is me. I am one of the senior instructional design project leaders at the Wharton School, it's a business school at the University of Pennsylvania, joined here by some members of my team, and I've been following new quizzes of since its infancy. It doesn't feel so new anymore, but, this morning, they announced a lot of new features that are coming, and there's a lot of exciting things in the road map So I'm excited for that. I've been at Penn for eight years. I started at Penn libraries as an instructional designer and then switched over to the business school.

About five years ago. I've I'm smiling in this picture because I'm standing in front of Mititure Wonderland in Hamburg, Germany, If you ever get the chance to go, let's go back to that because it's super fun. The this is miniature wonderland. Basically, it's like a miniature version of like all the towns in the world, and it's endlessly fascinating. So put it on your bucket list as a place to go.

I love the the little things. You can also watch videos of it on on YouTube, and it's Right. So back to, our quizzes quest today. This doesn't wanna transition very well. Let's go straight to it.

Alright. Talking about upenn, there are twelve schools that comprise the University of Pennsylvania. We are finally all using Canvas, and most of us are using one instance. And, that was a long time coming. We're all at various stages in our transition to new quizzes.

But we're all using it in some way. So there's about twenty five thousand students on our, Philadelphia campus And that's comprised of about half undergrads and graduate students. Specifically in the Wharton school, we have five thousand students also about half undergrads. We also have thirteen hundred full time MBA students. We have PhD and doctoral programs.

And then, we have an executive MBA program that's split between campuses in Philadelphia and San Francisco. We also just launched our first, hybrid. It's called Wamba Global. It's a hybrid online executive MBA program. So we're really excited about that.

And, all the new opportunities for using technology there. Wharton, there's about five hundred faculty members. Half of those are standing. And, then we have a whole lot of staff to support all of that. Here's what we're gonna cover today.

I'm gonna tell you what's worked well for Warton's transition to new quizzes. And spoiler alert, we're not fully there yet. I don't know of any school that's a hundred percent new quizzes at this point. And I'll explain why that is. We'll talk about, how we're overcoming some of the hurdles, we'll find some new quiz advocates and and some strategies for getting people on board with that.

One of my favorite strategies is just, you know, start small, dream big, scale up from there And then there are lots of great resources on the Canvas community. The new quizzes hub has gotten a lot of plugs today, but it really is a great resource. And there's also lots of other things. No matter if you're just starting out in your transition or kind of far in, there's lots of really great transition resources for you there. Alright.

So here's the timeline of events according to our transition. So new quizzes became it was quizzes dot next at that point. Became available to all the paid counts in fall of twenty eighteen. We decided to start using it in a pilot stage in spring of twenty nineteen. So about couple of months after that happened.

And the impetus for our pilot was, a very large introductory operations course We're talking five hundred students. It's a required course that everyone takes, and it it was comprised of about ten, series of numeric quizzes. And, if you've tried to regrade a numeric quiz and classic quizzes, you know that's not possible. It was a really big headache with having that many students and having to manually regrade all of those quizzes. So when we found out that new quizzes could regrade classic numeric cushions, that was enough for us to be like, let's do a pilot.

So that was in spring of twenty nineteen. Spring of twenty twenty, we all know, what happened there. And we decided to use that as our opportunity to springboard into new quizzes. We're like, You know, we have this at the time pending sunsetting date for classic quizzes, and we also knew that a lot of our faculty hadn't already created a quiz themselves online. So this was a really great opportunity for us, to kinda go all in at that point.

And we enabled new quizzes for our whole, account at the business school. So three years have passed. And we've seen a lot of new developments in in the tool. We just got the new rich content editor which everyone is really excited about. We've got bulk migration coming.

The grade by question type, the analytics are getting more in-depth. So you can really see that instructure is focusing a lot of development resources on new quizzes right now. So it's a good time to be alive. In twenty twenty four, maybe maybe at the end of this year, twenty beginning of twenty twenty four, we'll see a lot of other things pick up, that are a lot of features are dependent on a public API, and so I think once that gets through, then we'll see a lot of other features that are dependent on that. And let's all just take a collective sigh of relief that there's no sunsetting date at this point.

It's it's a relief not to have that looming over us and to know that, you know, you can do your transition whenever it makes sense for your institution. So, I'm gonna be open about our successes and our struggles. But why not start with a success? I won't talk about what has worked well for Wharton. So like I said, getting faculty advocates, especially If you have any large or important high stakes kind of courses, if you can get a faculty member who will, you know, celebrate the success of this tool. That's gonna really mean a lot because they talk to other and instructors listen to instructors, not instructional designers, unfortunately, some sometimes.

But If you can get somebody who's really excited about the tool that, will really make a difference. Another thing, that we did was to focus on communications about a month, four to six weeks before, like, exam season. So if you have midterms or final exams, you wanna think about when your instructors are gonna start thinking about I should probably put an exam together. You know, not sending out messages in July when everybody's on vacation. So targeting your messaging.

With the timing is really important. Try and get the audience before they actually start building things and going down a path but also, you know, enough time that, it's on their minds. We had frequent workshops and consultations over Zoom. We still do that. And some of these were targeted to academic departments.

We noticed like a lot of finance finance exams had a very similar format with the types of questions they were asking. So we would group as many faculty as possible together with one workshop, and we continue to do lots and lots of instruction sessions for, faculty. Especially those who are new, they haven't used, quiz two before. There there's always somebody who is unfamiliar with the tool, so we we continue to do that. And we also try and do, like, on demand sessions if you can send somebody recording or send them to a guide we have this all about quizzes, all about new quizzes guide.

It's publicly available, and I'll share a link to it at the end. Very similar to what's on the instructor guides in the Canvas community, but we just felt like, you know, worthnizing this would help our faculty to accept it a little bit more. So we made lots of documentation we told in our communications to faculty that, we told them that new quizzes was going to be the default tool. And that was really important because, that saved a lot of, stress in in just having faculty you know, have to decide which tool is right for them. We, you know, said, if you are using a new new quizzes for most of your exams, it will work well.

If you need to do x, y, or z, you know, you need a file download for your course. Or you're using a specific tool that, an LTI, for example, that doesn't integrate with new quizzes then stick with the classic version, but for all other use cases, use new quizzes. Initially, we tried to have like a decision tree, but having the messaging just use new quizzes by default was really helpful for us. And another thing you can do to help promote that is talk about the strengths of new quizzes. There's new question types.

There's more granular settings, in the quiz setting. And you can control the student result view better. You can also set accommodations in advance, and that was a really big one for us. A lot of those intro large classes have a lot of students who might get extra time. And if you can just set that in advance without having to publish the quiz.

It, saves a lot of anxiety on all parts. Philly. This this image was unfortunately, timely and accurate because we we just had a city bus crash into a building, not far from this location. But, you know, despite all of our successes with that early pilot and the adoption rate that we solve really early in the spring of twenty twenty, our transition hasn't been, without its struggles. And sometimes we are riding that struggle bus as well.

So I wanna talk about overcoming hurdles and we'll see if this poll wants to work. And, I wanna ask what hurdles you've encountered in, using new quizzes. So okay. It looks like it's in work. You can scan the QR code or go to polev dot com.

Forward slash new quizzes. Alright. Here are some responses. See if we can make this bigger. Not knowing what it was.

Yeah. Migrations, kind of linked to outcomes, transitioning question banks, lack of LTI Proctoring support, managing and chairing item banks, had that headache too, Fillion accommodations after a quiz has started. Yes. Just getting people on board. You know the old quote that they say an object in classic quizzes stays in classic quizzes It's like that, you know, for faculty.

It changes hard. The survey tools, a big one on my list. Yeah, high level math questions. Accessible question types. Yeah.

All of these things. So Lots of hurdles and lots of things to overcome, with this tool. Yep. Okay. So yeah, I I feel you and feel your pain on on all those things.

We've had a lot of things to overcome as well. The biggest one for the Wharton School has been trying to support two different quiz engines for the past four years. It's very common that we get help tickets from faculty members. It'll say, I built this new quiz, and I need to do this. I need to download the student responses or I need, to set up respond to lockdown browser.

And the response is gonna differ very greatly depending on which quiz told they say. And do they ever share what quiz tool they're using, they do not. They don't often know they're, you know, clicking around and they're creating a a quiz. And so they say, yeah, I created a new quiz, but does that mean a new new quiz or or just a classic new quiz? And students, sure as heck, don't know what kind of, quiz engine they're using. They just see, you know, they open it and they don't see the name.

So we've had to do a lot of work with that. Making sure documentation is very clear about, you know, which quiz engine to use we make sure we we preface all of our consultations with faculty by saying, you know, what what features do you need and we make sure that we guide them toward using the correct tool, for their instance, but, it still comes up quite a bit that, they don't know which one they're using. They can see the the icons when they look at their quizzes index. But that that's been, a challenge having to make sure that all of our staff are well versed in both quiz tools because we never know what somebody might be using. Some other hurdles.

We talked about migration a lot in the polls. One thing that's gotten a lot better, there's trick if you're migrating question banks to new quizzes. They're I'll I'll share a video in the resources page at the end, but essentially you can put all of those questions from a question bank into a question group and build that in classic quizzes. And then when you migrate, that quiz, it will automatically become an item bank, and that will save you a lot of time. So if you have a lot of question banks, that's something you can start doing now.

If you have a lot of quizzes that you need to bulk migrate, you might just wanna wait a couple weeks when they said that the bulk migration tools will be available. So, I am seeing a lot of progress in that regard of things that will help. The other things we talked about just shifting people out of their comfort zone, they're used to using classic quizzes and Everybody just copies things from year to year, which isn't a best practice for exams anyway since, you know, once you put something on the internet, it's it's out there. So we're still seeing just a lot of, resistance, people who don't want to change, if you haven't set up a new quiz before, what we found is that a lot of instructors find new quizzes much easier and they're able to do that with very little assistance. They can follow one of our guides and they find the new UI pretty intuitive so it's definitely more of a challenge for people who are are used to classic quizzes.

And that's that's one of the biggest hurdles we're seeing there. Talking about some things you can do to overcome these challenges the first one is to really communicate and understand the differences between classic quizzes and new quizzes. Sometimes it's bad to focus on differences, but in this case, you really wanna know, you know, if you did this activity in classic quizzes, this is what it looks like in new quizzes. So this is a comparison chart that we built at U Pen. You're welcome to borrow it, make your own version.

If you make one that's better, please share it back with me and, we can do that. But essentially, you know, looking at the different tasks that you would do in classic quizzes and how do they translate to the new tool? So that that's something you can do in advance. You can also anticipate the stumbling blocks. This requires using the tool quite a bit If it stumps you as, an IT professional as an instructional designer, it's it's obviously gonna stump other people as well. One of the big ones is the build and return buttons.

I had an instructor tell me he wanted to edit a quiz he had that was in a draft. And he's like, but I can't click on the build button. I'm like, why not? And he's like, well, I already build it. I'm like, oh, well, that that helps you go back to the editing page as well. So, I often refer to that as the quiz builder page.

And then, the page that you land on when you first open to quiz is the assignment details page. So assignment details, quiz settings. Those tend to throw people off as well a lot of our instructors get caught up over, availability dates and availability windows versus time limits. They're not the same thing. Time limit is a countdown, and availability window is when students can take it.

So those are some things that we have to repeat quite a bit. The return button is really easy once you've used it once. But those are just some things that our frequent stumbling blocks for us. So we continue to offer regular trainings and make sure our documentation is clear. We're updating it all the time.

And then we really focus on promoting new quizzes strength. We're honest about its shortfalls and we wanna make sure that, you know, if somebody needs to do something that can only be done in classic quizzes that we guide them that way. But, there are a lot of selling points with the accommodations and the settings and so we wanna make sure, that they're taking advantage of that as much as possible. And that makes acceptance of a new tool, a lot easier as well. For every, pain point in building a quiz, there's usually something in new quizzes that might lessen that pain.

At least that's what I found. So, you know, the classic quiz quiz builder. If you're not familiar with it already, isn't all that intuitive. But the new quizzes has a modern cleaner UI. We've done zero training with students getting them familiar with new quizzes because they don't need it.

They just figure it out really easily. And, that's been great. If you use essay questions, you'll find that there's a word minimum, maximum, spell check, and grading notes, so a really just more robust essay type question, and that's what it looks like on the right. In classic quizzes, there were only certain actions that would trigger a regrade and, we needed a lot more than that. So just having better regrading overall is really helpful.

It's found in a different place, so that's another, you know, stumbling block potentially you now regrade through SpeedGrader, and you can regrade any question and applies to all the submissions in that quiz. So it saves a lot of time, and it works for any automatically graded questions. So pretty much everything but the essay type question. If a student submits too soon, they get a little trigger happy, and they they click the submit button in classic quizzes, you can't reopen their attempt but you can do that in new quizzes. And I'm really excited about the new build on submission feature that they're about to release.

Some eager to test that out. Remembering to set up accommodations for, you know, quiz after quiz within a course, can be a little bit tedious, but you can set extra time for quizzes in new quizzes, and it applies to all the other new quizzes. So if you're using the time limit extender, that's one of my favorite tools you can just set it for fifty percent more time. That student will just automatically get the extra time. We used to have to rely on the great James Jones and his, canvas enhancement script to be able to print classic quizzes, and you can do that really easily now in new quizzes.

You can print just a blank quiz for students to take as, you know, a backup if your internet fails, or, you can also print the answer key. We have a student who's very low vision in one of our executive MBA programs, and we're able to print extra large fought, really easily for him without having, to reformat it, it's already nicely formatted there. So, moving on to advocacy, how do you go about finding those new quiz advocates? So my suggestion to you is to find somebody who's got a problem that can be fixed with new quizzes and and make them your advocate. I talked a little bit before about the intro operations course. The faculty member who taught that.

There were actually three faculty member, but the lead one was a vice dean. He had a lot of clout and he was really dissatisfied with not being able to regrade in classic quizzes to the point that he was suggesting that we switched an to a different LMS completely. And so, you know, we saw this tool. We we did a little bit of testing in the those couple months that we had access to new quizzes and confirmed that it worked well enough. So we turned it on for his course and, he was really happy about He had a really smooth semester.

Of course, there was need to regrade again, but it it worked pretty flawlessly. And we were able to get a lot of other faculty in that department on board, because of the promotion that he was doing, and, that that calmed them down. We didn't have to switch to a different elements after all. So, the other things that worked well with that his TAs were able to build the quizzes without a lot of instruction. And, yeah, overall, it just really made him, very vocal in the positive way then.

Another thing I recommend is to start small and and scale up. So those of you from the East Coast this is the new rebuilt section of I ninety five, which, exploded a couple months ago. And, it was rebuilt in twelve days, which is pretty incredible. Unlike that that section of highway, your transition to new quizzes is unlikely to happen all in twelve days. So give yourself time.

If you're not currently exploring new quizzes, now's a great time to, check it out, or maybe maybe you've done some research before, but it's been a while And I encourage you to take a look at the community and the roadmap. There's a lot of features that have been added recently. So start building up your expertise now. Get yourself familiar with the tool. I often hear, people say that they they can't do a new quizzes transition until it has this feature or this feature and they won't even look at it until it can do this.

Which I understand, for Warton, a really big thing for us is to have that student analysis CSV download I know it's really important for a lot of schools too. But while we're waiting for that, we can be using new quizzes and taking advantage of the features that do work really well. And, you know, roll out gradually. Maybe you start with a course or a subaccount and you test it out there somewhere where you know it might be a benefit. If I had to do our transition again, in twenty twenty, I would a hundred percent do it again because, where it's worked well is it's worked really well.

I think I mentioned before we're we're only about, forty percent in new quizzes right now, and a lot of that is because We have a lot of courses that use surveys. They still need classic quizzes for that. We need that CSV download. I expect that number will increase quite a bit once we have that feature. And, yeah, there there's also just been, you know, a lot of new faculty, people reverting back to paper exams.

So we're we're going to do another big push this fall to try and get more people using new quizzes. But for the most part, a lot of our large exams, high stakes things, waiver exams for new students are all using the new quiz tool. So yeah, if you can just focus on one thing, one one course, one subaccount at a time. That's a good way to start. And, with every new change, you have to be flexible.

Things might not work exactly how you envision, but maybe they wind up being better. It just it takes some flexibility, but, for the most part, we haven't encountered any like catastrophic issues. So, it's worked well for us. Okay. I wanna talk, again, about the features that are available in the new quizzes hub.

This is a really great place to get updates on the tool. All the instructor guides are linked here so you can Go to that. You don't have to build, rebuild things from scratch. There's also student guides. Should you need those? The transition toolkit, I'll talk about in just a second.

There's also the Canvas release notes. And you'll notice that there's a lot of focus on new quizzes, when you look in the release notes. Pretty much every month, there's there's something that's related to new quizzes. So, changes are happening. I was, lucky enough to be part of a new quizzes customer advisory group for several months.

And, I know that they've divided the product team working on several different aspects of new quizzes. So that they can better focus on delivering the features that are are really key. And, you can keep voting those things up in the themes when you see things related to the quiz tool both those said that they get plenty of attention. There's also a place for discussions, and there's really active discussions in this community. You know, someone might write in that they don't know how to do a certain, procedure in classic quizzes or is this possible with this tool.

And, I've I've found a lot of great resources there just by reading you know, what are other people struggling with or who's come up with a really unique solution, to do things? One, I, I solve a while back was how to give extra time to an in progress quiz, which, isn't actually possible, but there is a workaround that I found in the Canvas hub. If you have all the students take the quiz, you can turn off the time limit and then use the reopen button and it kind of works around that too. Give somebody more time in progress. In an in podcast quiz. So, there's lots of great things.

Highly recommend, checking that out. And I think they're also gonna put the video that was shared yesterday in the new quizzes panel in there. So I'll be looking for that. I don't really wanna rewatch that. The transition tool kit is a one stop shop place for all your your large scale transition needs.

And and really this is for a transition of any size, whether, you know, your smaller school or a single department or, working on rolling it out for, you know, a large school district. Some of the things here, there's, change management plans, communication plans. If you're not sure, you know, how to go about the messaging of this new tool and the advantages of that, you can find sample documents of of all those things there. There's training plan, and there's actually some new training resources that are coming. So I'm looking forward to that.

An engagement plan. So make this toolkit your friend, and, check-in on the space pretty regularly. Okay. I also recommend looking at, Instructure roadmap. This is the current road map showing, what they're working right in right now.

So, it will break it down into different segments of what they're working on for quizzes. You can see, maybe you can't see. So build on last attempt is, something that's gonna be released very soon. They're working on the migration functionality and being able to port new quizzes outside of their Canvas account. Right now, if you try and do a a whole course export from, you know, one instance of Canvas to another, doesn't include new quizzes, but, that's something that's gonna be available pretty soon.

And then looking ahead, you can see, there's better support for outcomes. How many people are outcomes, users? A few. Yeah. We don't use outcomes or mastery paths too heavily, but I'm glad to see that there's a lot of support there. One of the more recent releases too has been with blueprint support.

How about blueprint users? Probably a lot more? Yeah. So there's now a full blueprint compatibility and you can use that with new quizzes. And if you make changes to a quiz, they will sync in the new quiz. We have found some, small inconsistencies if you're also using Respond as lockdown browser, but I think that's to be expected when you're using, you know, LTI with an LTI and blueprint. Like, it's a lot of layers of, of, complication there.

Overall we found respondus lockdown browser to work really great with new quizzes. So if that's something you're using, that can also fall under the selling points. And then in the last column, you can just see, what's coming up and, twenty twenty four and beyond. So going over our our takeaways again, definitely recommend, you know, using a comparison chart. Here's how question banks compare to item banks.

For instance. This is the one on the Canvas community in the the new quizzes hub So you can obviously use that one. That's linked there. I recommend looking for new quizzes ambassadors. Some people at this conference have told me they've surveyed faculty and just asked if anyone would like to be part of an ambassador program or you know, be an evangelist for new quizzes, maybe just share, how you use it and and what you found challenging instructors are just a lot more likely to listen to each other.

We talked about following the new quizzes hub, voting up those themes where you can. And again, This isn't gonna happen overnight. It's been four years for the wharton school, and we're still in the middle of our transition. Granted we haven't given it as much attention. We've we've kind of just, we've supported the courses that we've we've been using new quizzes for for a while, but we haven't done a really active promotion for a while.

So I'm gonna be taking advantage of, transition toolkit again to, beef up some support for that. If I want to share yet another QR code, this is just a Google doc that has a lot of resources put together, I'm gonna make this editable so that if you wanna add any of your others and other things that you're doing, if they're not already shared in the Canvas community, it would be a great place to share that. But that video where it talks about how to quickly convert a question bank into an item bank. You can find that in this link as well. And, lastly, One more QR code actually.

If you wanna get in contact with me, I'm always happy to talk about, new quizzes and to talk a little bit more in-depth about, you know, the things that worked really well for Warton and what we're currently doing to get more instructors on board, talk to a lot of different schools, and, happy to do that as well. So, thank you for being part of this, new quizzes journey. I hope everyone has, a good experience as they transition whatever stage that you're in. And I think I'm excited about, where new quizzes is going. The the rate at which the features are coming out. It's just becoming a really robust quiz engine, and I think, it's making canvas better overall. So, Thank you for listening, and I hope you have a great rest of the conference.