I'll Have To Say I Love You In A Survey

Hi everyone, my name's Ben McGrae, I'm from the University of Liverpool. I work as an educational developer within the sense for innovation and education and I'm here with my colleague Will Moindrot. - Yeah, hi, yeah, my name's Will and I also work at the University of Liverpool as an educational developer. - So today we're gonna talk about our post migration evaluation here at Liverpool into Canvas VLE. We thought it would be a useful presentation for either anyone who's in a similar position to us, who maybe has purchased Canvas and is looking to do an evaluation themselves, or maybe you are a new customer or thinking about moving to Canvas, and they're going through a change in VLE or even anyone who's experienced and using Canvas. You'll probably find this useful in terms of you ever having to evaluate your VLE or other supported tools as well.

So hopefully you'll get some ideas or inspiration. Now, we're not saying this is the correct way of doing it. This is just our way of how we evaluated it. So hopefully you'll find ideas from it, but we're not necessarily saying this is the way that you should be doing a survey. So hopefully, like I said, you'll find this useful.

So just to give you a timeline of where we're at Liverpool. So back in 2018, we were previously on the Blackboard land, virtual learning environment and the university, especially the senior leadership felt it was time to change VLE systems with the overall aim, which is presented on this slide. I won't read that one out, but as we went through the tender process, that our decision was to go with the Canvas LMS or Canvas VLE, and through between 2019 and 2021, we went through a series of stages of pilots and getting set and people in like early adopters of staff using Canvas with the goal that we would eventually move everyone over from Blackboard to Canvas in September, 2021 is which we ended up doing in the end. So as soon as we moved over VLE systems, that's when we thought we might need to go through the next stages of how we implement Canvas, which I'll move on to next slide, which we'll talk about the wifi features. - Yeah with our implementation there might be just other people, I won't read to all through them.

So we had a Canvas advisory groups. This is a group of representative from across the university, academics and support staff who were helping to guide the implementation, make decisions, plan what it would look like in the forms of times that was been very useful. We also employed a small army of Canvas coaches. So these are students who employ one-to-one support during the migration, but also lend their student perspective to some of the Canvas related projects we were working on, and finally, we've also had a Canvas Connect. We call it Canvas Connect, basically community space within Microsoft Teams where academics can get help from other lecturers, there's live events, and there's lots of discussion and that's a constantly evolving community.

It was really interesting during the lockdown to have that kind of social space and it continues to evolve. We've got about 500 members, that's been really key way to get feedback about how staff are finding things, but also to spread news about the project as well so that's been really useful too. On the right hand side, I won't go through all of these, but this is kind of the Canvas package we've got in place at University of Liverpool, and also some of the third party tools that we've got in place as well. We're both very, very happy to answer any questions on that. If anyone would like to email us and we can share what we've got.

Onto the next side. So yeah, so at the beginning of this journey, me and Ben were tasked with carrying out an evaluation and we really wanted to engage with our colleagues, other institutions, other universities in the UK, who might have carried out an evaluation on their own VLE, and the common response was, well, I thought you just migrated to Canvas, why are you carrying out an evaluation now, why now? But our kind of starting point for this really was that we technically, we've moved over to Canvas, all the courses from September, but we wanted to evaluate how is it meeting its educational goals of being an environment that supports upstate learning and teaching needs and helps to drive and support the future development of pedagogy at the university towards excellent pedagogy as our senior leaders would say. So really we wanted to find out how we were meeting that aim, to what degree we were meeting that aim, but also to point towards best practices to help shape our future work around how we learn to use this new environment, this new Canvas that we've got. Back over to you, Ben. - Yeah, so just going on over the aims of the evaluation, when we started to thinking it together, we thought, okay, the VLE isn't just a one size fits all for a generic group in the university.

We've obviously got staff, students who we need to consider, and then you can even go down further with staff to say, professional service staff, academics, early career researchers, anything like that, really, we need to consider what are the aims of this new VLE for all these different groups of people and how can we run an evaluation that considers all of these different needs, cuz they're all vary in different ways. So we both decided as we'll meet with certain people online, we had senior leadership team, which is obviously the senior members of staff in the university who were involved in implementing the VLE review in the first place. We had faculty coordinators who were members of staff, teaching staff or support staff, their responsibility was linked with Canvas. So they had a very prominent Canvas role of support and Canvas in their schools or faculties. Teaching staff, we met with as well, and from the student point of view, rather meeting with active students, we had those Canvas coaches Will alluded to earlier, who were students as well as working as Canvas coaches to find out what their aims were.

So as an exercise, what we thought we could do is meet online and use a tool called Mural, which is advertised here on this slide, and the way it works is it's like an online sticky boards or sticky post-it notes and we thought we need to consider not only just the short term aim that everyone's gone onto Canvas and it's been okay, but what are the long term aim? How do we want Canvas to be used four or five years down the line and still be a tool that meets all the needs of these different groups? So we ran it through that Mural board and tried to gather all these different sorts of comments and names. So on the next slide, from those sorts of discussions, feel free to pause this slide, but these are the sorts of statements and conversations and points of view that when we met with staff, so like professional service and teaching staff of what they wanted to get from Canvas, the evaluation itself, and we had about 311 responses to the staff survey anyway. So I'll let you pause and have a look at that. As I do so, I'll move you on to the next slide and pass you onto Will about the student survey. - Yeah, so these are the broad areas here as well, really, which was again, reviewing students general feelings about the Canvas experience at the university.

Also around their feelings around the practices that they found most useful within their course designs, how they were structured in the general environment, but also more broadly about how they wanted to be engaged via the VLE, and we got a good number of responses to the survey. Almost one and a half thousand responses so it's a nice sampling there. Onto number, next slide, and so for running the surveys, just giving you a bit of detail about how we, I suppose these are kind of tips in which other people might wanna follow, but for the staff survey, it was pretty much standard I would think for anyone looking to do something similar, which is that we advertised through the global announcement tool built into Canvas, which is very targeted and also via email as well and for staff surveys, we also looked at completion data to be able to target specific promotion to certain schools that were kind of unrepresented in the survey responses that was quite useful approach. For the student stuff, that's slightly a bit more exciting. We could de-coordinate heavily with the essential communications team for the student survey because we do suffer as most institutions probably do, something called survey fatigue, where students are sampled about their opinions too much.

So we wanted to ensure that we coordinate it against the other important surveys that the university runs. We did a prize draw for completion against, I think we had five prizes to give out, selected at random for more people completing the survey. It was Amazon voucher or a high street voucher of student's choice and we also kicked off the survey at the Guild of Students, which is our students union basically on campus and we had a link to the survey through a QR code and also we gave free cacti away to students who completed the surveys, a bit of a nice giveaway to them and that was nice. It nice green gift and also, cares about their wellbeing as well. So, onto you Ben.

- Yep, so a follow up to the student survey is that we decided to think about focus groups for students and how to design those and try and make them encouraging and sort of fun to be at. We thought, okay, we'll try and run maybe about 10 over three week periods and explore the findings of what came out of that student service. So draw out more content, more of that sort of experience of different features are in there. So one of the features, one of the ways that we thought we'd structure the session is using an activity called the diamond ranking exercise. Now on this slide, there's a link to a paper by Jill Clark about how she uses the diamond ranking for visual cues.

But also on the right hand side is one of our slides of what we did. So we knew that we'd probably have to run these mostly online. It was coming towards the end of term for most of the students, we did run some of these online, which I'll talk about later, and we met with the Canvas coaches again to construct what sort of contents would go in this diamond ranking exercise. So the way that it works is that you've got a diamond shape, we've got a question about what Canvas features are significance, and there's a series of statements that are sort of amalgamated statements that came from the student survey as well from the coaches of what they would like to evaluate. Now, I'll talk about the sorts of running off it in a second, but just to let you know, these statements, act as a sort of like baseline or starting go at conversation point.

So they're not sort of set or representative of a specific group or person in the university. So when we ran the focus groups, we did probably I'd say about eight or seven online and a few face to face. The students who came to those were encouraged through the student survey that Will talked about. So I think at the end of the student survey, there was a question at the end that said, would you like to be contacted further to be part of a focus group? So those who contact us through there, we sent an email out and invited them, only to that particular group who opted in, and again, there was an incentive to encourage them, that they would get 10 pound voucher, either Amazon or high streets again to come those. The way we ran them, especially online is either myself or Will, would host the meeting and then we would get either a Canvas coach or two to take notes and facilitate that conversation.

So when it worked, when we ran it, when we got to the diamond ranking exercise area, we would obviously have that structure and the comments and like I said, it would act as a baseline or a starting point for conversation and we try and draw out the conversation, whether that's good or positive and or negative. So for example, on the slide there, it's got something about gives a good mobile experience and we would try and say to students, okay, what reasons is it good or why is at least significant and try and draw that out. I should mention this says on the slide here, we use Canvas Studio to capture the online and face-to-face meetings which is really useful. But that conversation like I said, it don't feel like you're structured into that rigid diamond. Like you can see on the slide here, we moved certain comments alongside each other.

We had blank boxes if students had comments that didn't fit in with anything that we've had already. So the idea is that we just tried to develop something and get them to move and talk about it and then as a sort of analysis afterwards, we'd maybe see which common ones were at the top, which ones at the bottom, but mainly it was the discussion that we wanted to gather and the quotes that came from that, which we'll talk about later on too. So I'll pass you onto Will just to talk back on the survey data. - Yeah, so just sort of some headline figures really from the survey. So the top ones there from student general perceptions of Canvas and very high averages, really, from one to 10 scale, higher around effectiveness to help engage with studies, ease to which they can locate and navigate and around their learning resources and also find important information.

These are very high figures, particularly in relation to perhaps their previous experience under our older VLE. Staff perception of campus, again, very, very similar, really, very, very high levels of satisfaction, around 75% around satisfied based upon their needs for their work, very high perceptions of reliability, 92%. You know, these are all very good, very good numbers for, in terms of the projects and evaluating how well it's gone really and also around tier one utilization, again, very, very high rates of experience there. So on the next side, we were also having a look at other findings. This really was a key finding from the student survey, a real sort of take home for us really was that the percentage of students who used the mobile app.

It was much higher than, we'd never really seen it written down before, but of one and a half thousand students who responded, So very, very high indeed, and of those students, the average rating was very, very high at 7. 62 that it was a very positive experience of using the app. So it's very, very high and that's gonna have, I think, repercussions in terms of how we work with staff in the future about how they design their course, to design their course in a way that is more mobile friendly, makes more use of Canvas native tools in order to improve that mobile experience. So that was a very, very good outcome. And also we did ask them around consistency versus individuality and you can see there's a nice mix there.

Half the students want consistency. Half of them want basically an element of individuality in there as well. So let's move on to the next side. I think we'll have a look at that in a bit more detail, I think, won't we, Ben? - Yeah, so when we ran the survey, there was an option for most of the questions when they had like a like it scale or a score that just had to produce that we asked students to emphasize their answer a bit more, or give a bit more detail. Now we came back with obviously the thousands of responses to these questions.

So we thankfully managed to get funding from our project team to get students to do a thematic analysis of all the three text comments. So there's a lot in there, but we thought we're just drill it down to the more highlights here on this slide of what came outta that thematic analysis. So one of the first things that the students obviously talked about, which is really nice with Canvas is the fact that they can personalize their learning in Canvas, to an extent. The fact that with the app, especially, notifications come through, the to do list populates, does the calendar feature, and obviously, the app, obviously you can customize what comes on and what doesn't as well, and the fact that you can move things on your dashboard and hide things as well. Now, students did also say that they would like to see this even further customizable in terms of, if it's a one size fits all and they're in Canvas, why can't they add own personal stuff to the to do list or the calendar tool? You can an extent add your own personal stuff to it.

So they were wanting more, they liked that feature, that the fact that it's personal, but they like to see more personalization. Now it was overall positive, but the negative stuff as well that came through, it very much depended on the staff approach of using Canvas. So we were seeing a lot of positive comments coming from students. In fact, that the members' staff who was teaching their course was using it in a positive way. It's very clear, concise, consistent in terms of how content was laid out and easy to access, notifications from the announcements were coming through quite nicely.

The negative comments seemed to be upon staff not using Canvas very well. Maybe they didn't use the modules area appropriately. It was a free for all for the contents. There wasn't enough communication being used. So some of the features that they sort of highlighted the students, so then those four bullet points that highlighted, there was a variation and approach to assessments.

Information could either be clear or either it was unclear. How mobile friendly design was? Was a member of staff using the modules area. like I said, appropriately, were they using pages and the pages went checked using the mobile app? Were things in the right places? A lot of students complained if things were inconsistent, they were going to one area for assessments, for example, like a module that had been set up, or maybe it was in announcements in another course, it was always inconsistent or consistent and then interestingly, the students were interested in what type of training people like myself and Will were given or other members of staff were given to staff, and whether that was having an effect on staff having much more experience and being more confident in using the tool or not, and like Will highlighted on the last slide about consistency and individuality. Students were also interested in this as well, knowing how staff are using it. Now, it's an interesting approach in that when we launched Canvas, we didn't tie staff down to a particular layouts.

We gave them the basics of Canvas and showed them how to use specific tools and maybe suggested a recommended layout. But we left it up to staff and we saw some interesting things happen, whether a school, so say for example, the history department, they would design their own sort of layouts and make all their staff use it so it would create that consistency across. But others would do their own individual approach. They liked having the flexibility of pages. They liked having little icons on there and we just left them to do that.

So that's something that will maybe going forward in the long term, we're gonna have to consider in our training and how we use Canvases, whether we stick with a consistent or whether individuality, we try and promote that. So this just a slide just to go over what our next steps are from Liverpool's point of view. So part of that whole survey and gathering the data, we have to do a bit of an exercise of just doing a closure report of the whole project, the VLE review. So that has been done. Now we've produced that report.

Then now, for myself and Will, we're probably gonna have to distill all the findings from the reports and try and produce a list of recommendations and target each group in the university. So again, going back to those groups, professional service, teaching staff, students, technical support, IT services. We need to go over to them and say, these are the different recommendations that have come out from this. And then the other final thing is just that data and how that data can be pushed onto future work and groups of support. So for example, if students are requiring more accessibility needs within Canvas, we need to make sure the working group within the university looks at that.

If assessment information is inconsistent or the tools that they're using are getting inconsistent, we probably have to speak to an assessment group or an academic integrity. So that's part of our job is trying to find ownership of who takes long term responsibility for these Canvas issues that came up in the survey. I'll pass it on again to Will about the recommendations then. - Yeah, so if other people in a similar position to us wanted to carry out an evaluation of how their team or their organization's using Canvas, then these are basically some kind of points of recommendations from us overall. So I think really key to us was working with the stakeholders from really an early stage in this process, really understanding what levers they were keen to explore, how they wanted the evaluation to be useful for them in their job roles.

So that definitely also prompts by on the actual research data, research methods when it happens, be it completion rates and the survey, or volunteers for focus groups that's really key. But it also gives some really very, very valuable useful data from the outset as well that could be useful in, maybe if you're doing some sort of practice research with participants and you wanna explore how they can develop their practice. So that can give, hearing the discussion about how people visualize the placement of Canvas within their other kind of ecology of resources for example. It's a very useful discussion to have. We would definitely recommend applying for research ethics, if you have to do that within your institution.

It certainly gives, it's nice when you, at the beginning of the survey, you can give participants the confidence, their answers are gonna be dealt with and used in a kind of an ethical way. So that's very key, and if you start from the outset, if you've got the ethic outset, it means that this skills are used to help with your questions. If you're interested in student partnership that can be a really useful source of inspiration and ideas. We work very, very closely with them on the design of the focus groups, and I think that really enhanced the output on that. Getting their student perspective really helped to basically cut through to the core things that students want to respond to, that's very, very useful indeed and finally, related to that is definitely consider I think, a kind of a structured activity for your focus groups, particularly if you do them online, like we had to do.

So we use the diamond ranking approach, the paper which we highlighted before was very, very useful for that. But something like that is very nice. It gives people a task to do in a focus group and something to have a nice conversation around. So, I really recommend that, yeah. I think that's pretty much it from us, isn't it, Ben? - It is, yeah, so thanks for taking the time to listen to us and like Will said very early on, we're happy to share the slides or any of our findings.

If you do want to contact us, here's our email addresses and if you're ever in Liverpool, you're more than welcome to visit us in Liverpool as well. But just a big thank from us and taking the time and hopefully we'll get to meet you in person soon. - Great, yep, thanks a lot.