Adam Avramescu  00:00

Welcome to CELab, the customer education lab where we take customer education myths and misconceptions and banish them to the outside lions where they belong. I am Adam Avramescu, and I am so happy to welcome Mike De Gregorio to the show. Hi, Mike. Hi, Adam. How’s it going?

Mike Di Gregorio  00:18

Yeah, great. It’s pleasure to be here with you today.

Adam Avramescu  00:21

I’m so happy to have you on the show, too.

Mike Di Gregorio  00:25

And a big fan for a very long time. So, super, super excited to talk to you on the SEALAB podcast.

Adam Avramescu  00:33

Well, as I was saying earlier, I’m excited to have you on too, because I feel like having seen you in the customer education community for so long. I feel like we have this parasocial relationship already. So I’m happy to bring that into an actual relationship today.

Mike Di Gregorio  00:47

Yeah, we feel like old friends already. And so

Adam Avramescu  00:51

yeah, well, you know, it’s it’s a perfect day to start forming those relationships. Because we are actually on the National Day of the one of the smallest and probably most tight knit states. It is national, Vermont day.

Mike Di Gregorio  01:06

Love it. I have a team member from Vermont. So I that’s an amazing day to be celebrating on the podcast.

Adam Avramescu  01:12

Do you want to shout out your team member?

Mike Di Gregorio  01:14

Yeah, shout out, Haley. Yeah. Love Haley from Vermont. She’s an expert in the world of higher ed publishing. So we’re very happy to be working with her.

Adam Avramescu  01:22

Excellent, excellent. Hi, Haley from Vermont. hope hope you’re having a great national Vermont day. Hope you’re enjoying maple syrup. Is that Is that what they have in Vermont? That’s that’s what we had in Montreal. I just assumed that we used to get TV from Vermont when I was growing up in Montreal. So I knew all the like the Vermont stations.

Mike Di Gregorio  01:40

I am as Toronto Toronto person you can be so I have no idea what you’re talking about.

Adam Avramescu  01:46

Oh, so we’re fighting. I didn’t realize Oh, yeah. Montreal, Toronto, actually. We can if you renounce your, your loyalty to Toronto? Oh, how dare you? Alright, well, maybe maybe we’ll sing informer by snow later. But for now. Let’s get into it. So Mike, I think it would be helpful maybe if you could tell the audience a little bit about yourself and what you do, and how that ties into some of the instructional design topics that we’ll talk about today.

Mike Di Gregorio  02:19

Yeah, thanks. So I manage our team of instructional designers at top hat. Top Hat is a higher education company, we build courseware for higher ed. So stuff like like in class, live lecture response, online textbooks, online tests. So I lead the instructional design team there. Externally, customer education is what we do. But given that we live and breathe in the world of higher ed, we go buy instructional designers internally and with all of our customers who are primarily professors, at universities, colleges, and even what you would call centers for teaching and learning at, at universities. So that’s what we do at top hat and the team I lead.

Adam Avramescu  03:01

That’s, that’s super cool. And that gives you a really unique perspective, I think one of the reasons why I wanted to talk to you is you have such a unique perspective on instructional design and learning experience design, because you’re doing it as a customer educator, but you’re also doing it for an audience of people who also do it as their jobs.

Mike Di Gregorio  03:19

It’s very meta. My joke is that I educate the educators. And I’m like the answer to that question. But it’s like your real unique space, because a lot of times, also our customers are experts in this field. And so like, it’s a real nice challenge as a customer educator, because you really have to come prepped with your material and your suggestions and techniques, because they are really looking at it with a keen and critical eye. And you have to make sure everything you’re doing is evidence based and sound and you’re steering everybody in the right direction.

Adam Avramescu  03:54

Yeah, I think that’s so interesting. And one of the reasons why I wanted to talk to you as well, because, you know, we’ve done some instructional design episodes on the show before, I like to do a little series called instructional design. 101. But, you know, when I when I think about the opportunities out there for people getting into this field, or perhaps are more experienced, learning experience designers, instructional designers who are making the switch into customer education from either internal or higher ed, there are so many opportunities to go beyond that instructional design 101 and really start thinking about some of these evidence based techniques and practices.

Mike Di Gregorio  04:36

Yeah, 100% The interesting thing for me when I think about like the history of instructional design, is that it really started out or one of the one of the the places where I began was in the military in the US. And when you think about it, it makes a lot of sense because instructional design should be able to produce an output that is repeatable with a series of like instructional techniques. interest, it’s very, very clear what you’re going to get out of it by going through the same instruction over and over again. And that, for a variety of reasons became very popular, especially once courses, my digital and the Internet became a thing around the turn of the millennium. But over the last, I mean, let’s say 10 years or so for sure, there’s been this pivot to learning experience design. And I think our community in the customer Ed channel really breathes this as well as any community can. Because there’s so much that can go into thinking about your learning design, the learning products, you’re creating how you’re architecting learning, and that goes well beyond just making sure your lesson itself is sound. There’s there’s a lot that needs to go into it right from, like persona research, to user research to the design of your of your material. There’s a quote, which is not mine, but I’ll share in the in the phrase learning experience design, a no word should weigh more than the other. So all three are important. Learning is important so that they’re all equally important in that phrase. And I think that’s a very nice way to think about the it’s like really keep yourself grounded as you are trying to make a pivot to more evidence based practices that learning is important used to be evidence based, the user experience is very important, also needs to be evidence based. So it’s a design, your materials also need to be good. So all those things really have to work in conjunction.

Adam Avramescu  06:26

Yeah, it’s interesting to hear you say that, because he’s, as you started describing the role of learning experience design, where you started sounded to me a lot like UX design, or, or even UX research. And in some cases, unless maybe like, what we might consider traditional instructional design or some of the practices that originally came out of the the military or some of the early learning research. I would be curious to hear your perspective, Mike. And I realize that there are many different perspectives on this depending on who you ask, but what’s the what’s the difference between instructional design and learning experience design? Is there a difference? Are they just fancy names? Same thing? Like, tell me a little bit about that?

Mike Di Gregorio  07:08

No. So the the key difference is that one is focused on the learner and the learning, and the other is focused on coaching the coach, so to speak. And if you want to be learner centered, or in our case, customer centered, you like that’s, that’s the key difference, right? Instructional design is, let’s say, like a coaching behavior for yourself to make sure everything is is coherent, and sound, but Alex D, should really, really start with the learner. And they’re, like, what, what the what you can call like their moment of need, or like one moment of need they’re in, right, and how they will be learning. And like the messy business of actually just doing the learning, which is never going to be this pristinely designed kind of experience. Right. Like they’re, they’re learning it in, in like the real world in some way. Probably maybe in like in some kind of emergency work, right. And then they don’t need to learn something. And so like really need to be learner centered, is the key key difference.

Adam Avramescu  08:07

I see. So, so help me from a very practical perspective, I’d love to unpack that a little bit. So like, you might have someone who was an instructional designer, and they follow something like the ADDIE process, or maybe even Sam where they they do that savvy START to begin with. And so you might have someone who’s calling themselves an instructional designer, but saying, hey, you know what, I do an analysis phase, or I do a savvy start. And the first thing that I do, or maybe I even do like action mapping by Kathy Moore or something like that, and I’m actually going to start with the performance problem, I’m going to unpack some of those assumptions about my learner and what they need to learn. Is that is that learning experience designed by a different name? Or is that still fundamentally different from what you’re describing?

Mike Di Gregorio  08:50

No, I mean, I would say I think it’s a learning experience designed by a different name. I, and I love Kathy Moore’s action mapping. I will say that, because I think what I absolutely love it, you’re you’re bringing up an interesting topic, though, because I think a lot of a lot of constraints on the field of LSD. And learning in general is a really the result I think of expectations from like, higher up the corporate ladder and what the expectation around learning might be, and why something like Kirkpatrick model persists, because it’s very easy to, to see and to measure the big event of learning where we have the training and everybody gets a smile sheet at the end of it. Whereas like the the messy business of learning is probably happening and in many more different ways that you need to still be attuned to because that’s when somebody actually needs to apply the thing that they’re learning. But to answer your So to go back to your question Yeah, I think it’s Ed’s LSD by a different name. And I mean, my experience with it has been that the the models around like doing learning analysis and breaking down or deconstructing a learning experience These are all very good to do. And it kind of doesn’t matter which one you choose to employ. I, I like to like, whenever I’ve done this, it’s been like, you know, over a glass of whiskey in the dead of night and just like trying to like, like deconstruct something we’re doing to see how it works. There’s a great sort of five, there’s a great five elements of learning design model by a guy named Andre plough, who modeled it after sort of UX research and UX design. And it’s worked really well for me to deconstruct a lot of what we do and your thinking there. So really specifically about what the learner is experiencing, how they’re doing it, the very like minutiae of their interaction, etc, etc. Like, ultimately, I don’t know that that’s effective to like, communicate to, let’s say, an executive at your company. It’s very useful, though, for you to know so you know, how to communicate what’s relevant. So I would say so, yeah, this this is like really how I approach it like the, and from what I’ve seen useful for me has been using these models to break down something we’re doing in a way that helps me understand what to communicate and what to adjust and change.

Adam Avramescu  11:07

So it’s not like what I’m hearing, they’re not completely different fields. Obviously, we’re instructional design and learning experience design have no matrix of overlap. Some of it is like the the fundamental model that you use to solve the problem and the fundamental model you use to approach what that learning experience actually is.

Mike Di Gregorio  11:24

Yes, absolutely an easy way to think of it as a Venn diagram, two circles, instructional design, and like user research, or UX design, and the overlap of those two is traditionally thought to be Alex D, be the easiest way to think through it. So 100%, you need to incorporate a lot of the good learnings and habits from instructional design, and pull in all of these other fields that are super, super relevant to designing a good learning experience.

Adam Avramescu  11:51

That That totally makes sense. And the other thing that I’m hearing, as you were describing that is that there’s there’s maybe two, I want to say cruxes. But that’s probably not the the plural of crocs, it’s probably two cruises where they work where they differ. One is around measurement. And we should, we should definitely get into measurement and talk about the smile sheet and Kirkpatrick and being able to, you know, tie to different, but ultimately different forms of measurement and output. And the other is maybe what the intervention actually is. Because when you’re designing for, you know, human performance, you’re not necessarily making the assumption that the unit of output is always a training. And when you’re talking to the executive, you know, that this is where I kind of heard you going when you’re talking about Kathy Moore, and we were talking about reporting to the executive is that a lot of the times in more traditional instructional design, you have an executive come to you and says, like, we need a course on this. And you go and you produce the course on this and you design it, you develop it, and you implement it, and then you do Kirkpatrick evaluation on it. Whereas LSD might have a little bit more of that design thinking human performance aspect to it. Am I Am I getting those? creases?

Mike Di Gregorio  13:08

Right? Or your AP? Yes. I’ll confess, I think crux is a Latin word. I know Ancient Greek, sadly, from my academic days, but not Latin. So I have no idea what the plural is. But yeah, the two, the two pillars are 100%. Correct. And those Thank you better? And I would say the Yeah. I mean, what’s interesting is that I mean, the old, like, 7020 10 rule, right? Like, you know, 10% of training is done in a formal setting 20% And get like quasi formal settings, coaching from your, like, direct managers, etc, and the 70% on the job from your peers, right. But then, like, when you think about resourcing though, 90% of resources, go into the 10% of training, that like the formal training, not the 90%, of where you’re actually doing and learning more effectively. So I think Alex D provides a nice framework for flipping that model in a way that actually, you can then start to spend money and resources and time, like more appropriately and in a much more aligned way to where people are actually learning. But then you don’t get again, like the what’s missing is that like the big event, right, which is easy to measure, and so all of the small events become harder to measure. And on the note of measurement, often what we’re doing is, and this has come up in, in a lot of conversations in the customer Ed channel, we are often pushing on some leading indicator, the lagging indicator isn’t there for a long time. And so you really have to make good decisions for what, you know, what are the one or two levers you want to push, like in this moment, with your learning your learning program, so that way you get that output that you need, or that you want to see and live six to 12 months? It’s not always it’s not always going to be immediately relevant.

Adam Avramescu  14:50

Yeah, I think that’s a really good point because learning unlike some other some other things in the world that can be measured, doesn’t have this immediate You know, immediate output. And furthermore, it’s going to be a little bit messy in terms of some of the other signals that might interfere with it. Because in customer education, for instance, you’re going to see some of those longer tail measures, interact with other things that the customer might be doing in that time period. So there’s not necessarily always a way to correlate, and certainly, in a lot of cases, not a lot of ways to do a causal analysis of say whether a customer taking this training, then later led to a renewal. So I be really curious to hear your point of view. I know there’s been a lot of work done recently on thinking about how we assess and measure learning, I know will tell Heimer has, you know, done a lot of work to take us beyond the smile sheet, and I’m using his I’m quoting him there. How do we think about this for customer education, knowing that our metrics can be inherently a little bit messy.

Mike Di Gregorio  15:53

So what I’ve tried to do, and this isn’t maybe a great solution, I’ve just tried to measure more and more things to try and find more and more ways to correlate and so like every possible Point of measurement is like the is has been my approach to it, and tol Heimer is, and I’d recommend his book for anybody who’s listening for whom you might be a new name, although he’s he’s got I think, in addition to quickly, quickly on the way, so he said don’t like the old book. Yeah, he did say don’t buy the book didn’t. So without proviso. I’ll share what I’ve learned, which is maybe not entirely wrong. But what we did was we use tall hammers, or what I did was I use tall hammers, survey measurement, to measure our Customer Training, not just from satisfaction anymore, which was nice. Like, it’s a nice, you know, shout out in slack when somebody gets a good sort of survey after a training. But ultimately, we want to know that customers are actually applying and motivated to apply the things that they’re learning, and that they’re having a good experience as that’s happening. And so we started measuring, motivate that we started measuring all of those things with, with with different kinds of questions, right? How motivated are you to apply the learning? How much do you remember from the last time you learned it? Well, overall, what was your learning experience? Like? You know, would you recommend this to a friend, etc, etc, like, so we were trying to, like, take this the stuff that worked from the old world that you want to know, like, just in general, do people enjoy the learning? But then also, are they? Do they understand that like, how to apply what they’ve learned? Are they motivated to do so do they know where to go when they need help? Do they feel that they’re being supported in that endeavor, because the other benefit of moving away from just like the scale of one to 10 score on how well the learning session was that you then get to start to communicate what good learning looks like to like every two other decision makers in your company, and then they start to see through just the way that you’re measuring your customer training and the output from it, what good training should look like, and then you can, you know, in that way, then start to argue for things like, you know, a little bit more resourcing, or more patience for, like, just the learning learning event itself, and so on, and so on. So that’s, it’s been a, that’s been a nice change that we’ve made. And we get some really, really nice feedback from it, and what customers are expecting out of the training events that we that we put together.

Adam Avramescu  18:14

That’s really interesting. So in some ways, it’s about unpacking that generic smile sheet one to 10, or sad to happy or Yeah, isn’t it, and really asking details that actually start to if we if we put the analogy back to something like Patrick, actually using some of those post training surveys to actually ask deeper questions about what they learned what they’re going to do with it, how they’re going to apply, maybe not necessarily getting all the way to business impact, because a learner isn’t necessarily gonna be able to tell you all of that, but so that’s interesting that you can actually get some of that while you’re also collecting the leading indicators on learning satisfaction.

Mike Di Gregorio  18:54

Yeah, and tall Heimer. Maybe you’ve seen this Andrew, he’s like pretty critical of Kirkpatrick and the Kirkpatrick model, and its inability to bridge levels two and three, and that there’s nothing in the model with that can help you actually see if the learning is working. Now, having said that, it’s also important that like, as learning professionals, you do need to just develop a good learning, like a good learning experience, right. And like, that’s, it’s not all about output. Like we’re also learning professionals and tell him or like to use the phrase learning architect, which is sort of nice turn of phrase, but like, you can’t, can’t dismiss the fact that just the actual lesson you’re providing, be it like in your customer LMS. Or if it’s, you know, you got to be on site and do it, you know, in person again, like that still has to be good experience. But there we really need to focus when it comes to measurement and impact on how we are bridging, you know, just the learning and then the changing behavior and how we’re supporting that change. So surveys also like, also months later, or even weeks later, from the time you did the training are very informative, in terms of how your initial training went and can and can Direct you into how to adjust it or what to do or how to supplement it. Because it’s never going to be done. Right though the work of the training and the educating is not going to be done. So you’re really thinking about how you can bridge levels two and three. And Kirkpatrick model is very important, can can really online, that can really be the scale engine. Right? If you have to like, like, put a pin on some part of it. That’s, that’s where you can really start to scale things.

Adam Avramescu  20:24

Yeah. So tell me a little bit more about that. I’d love to hear like what kinds of decisions you might make based on the input that you get from your surveys now that you’re collecting this richer data? How you might use that to, you know, improve a course or follow up with the client? Or oh, what do you do with all that information?

Mike Di Gregorio  20:42

Yeah, what we do with all this, we look at it we we measure churn, right, like, like, like everybody else, and measure renewal and retention and make sure the customer experience is good and unified. Because the other I think the other nice thing that Alex D will do. In conjunction with it’s funny I, I started to like really take a deep dive into Alex D. At the same time as I was reading, Chief Customer Chief Customer Officer book by Jean bliss, and they really like coalesce nicely together. Because the entire customer journey is essentially a very long learning journey, writing you needed to feel coherent. So measuring the kind of the same things at different points that are happening with the customer with marketing, sales, with customer success out of post sale, and through the renewal, all you end up with a lot of nice insight. But on top of that, it has to be married also with just regular product behavior, and you know, product analytics, and are people doing the behaviors that they should be doing after having gone through your learning your learning program, and that learning regimen. Like it. So really understanding like what those are, and also understanding at the beginning, what are the things that your best customers are doing. But also like what the domain says your customers should be doing with your product, very easy for me to do this in higher ed, because there’s a whole vast body of literature on what good teaching and university looks like, which I then just need to make come to life in the product. But like me, sort of like closing the loop with the like the performance focused surveys and also app behavior, and having all this feedback into how you are designing and the sort of courses that you’re designing and delivering. And then also with a splash of just add of like actual conversations with customers. I do this routinely. Somebody goes through our micro badging, we call it top hat certification, but it’s, it’s a lowercase c certification. It’s a project based, you know, micro badge. And, you know, I was like always talking to them. We’re like talking to as many people as I can, who go through it, to make sure that we are delivering what they were expecting. They’re motivated to apply these things, you know, you adjust the content based on the evidence you collect. And I’ve learned over the years now doing this, that the gaps I thought we had in our learning experience, were not the gaps that the customers were identifying in the learning experience. So I would have been way way off had we not just done, we’re not like treated our learning as a product and done user interviews on it.

Adam Avramescu  23:07

So if you had just gone and done your own retrospectives, which I’m sure you also do, the things that you would want to fix in the next iteration are the things that you see coming, you know, from your product roadmap, actually have very little or less overlap? And you would think with what customers are telling you in those surveys. Exactly. Exactly. The interviews, not the surveys.

Mike Di Gregorio  23:27

Yeah, Precisely. Precisely. Yeah. So it’s a real holistic picture you want to get of your, of your learner, for the sake of informing your learner persona, right. And I know that all of us are going to be very familiar with customer personas. But these are very different from learner personas, your buyer persona is not who’s learning we face this very acutely. In fact, I think, I suspect much of the audience that faces acutely, I call it the customer who’s bamboozled into using the product, you have no choice and they don’t get to go through any kind of emotional buying journey that sells them on the dream. And so you have to incorporate that into the beginning of your customer education process and your the learning design of your your onboarding and, and the maturity journey that you’re mapping out for everybody. And with so anyway, so without that, yeah, it’s a it’s gonna be incomplete if you don’t have that that really holistic picture of who your learner persona is versus your buyer persona.

Adam Avramescu  24:22

And you do go and develop specific learner personas that you then map your design to,

Mike Di Gregorio  24:28

I want to but you know, I mean, so So the so it’s funny, as I was I was prepping for this conversation today where I was thinking about this old work on personas and I like had some like half baked personas for this and then I realized you know what, we’re gonna end up talking about this and then Adams gonna ask me about it. I’m gonna have to confess I haven’t actually done it properly. But in the in the spirit of, you know, design thinking and iteration like yeah, we’ll go back and adjust. And at least in the case of my product, like the top hat has grown As a product and what we do so quickly in such a short time that the persona I mapped out, like two three years ago are not going to be the same personas that we’re designing for today. So I was just like, not going to beat myself up too much about it. But I do feel a twinge of guilt.

Adam Avramescu  25:17

Now, Mike, I think it’s actually really helpful for people to hear that because, you know, you, you sometimes you go to a conference, say, and someone talks about their amazing program, and they’re puffing up, and they’re not telling you about the stuff that they haven’t done or haven’t had time to do yet or haven’t prioritizing, you know, you know, as they’re up there talking about that, that there’s something like that, because we all have something like that. In fact, most of us have many things like acid mania, that we that we would go, that sound like a really great idea, but we just haven’t really been able to prioritize them yet. And in fact, with personas specifically, I noticed that that’s work that a lot of people don’t necessarily do, you know, and create their own specific learner personas. But what some people do, and what I’ve done in the past as well is, there are other teams at your company who might be doing this work and not just mapping out buyer personas, but might actually be mapping out customer personas. And you can inherit those, at least on your learning team. So for example, at SLAC. Previously, we’ve had our UX research team and our product team come up with user personas and customer personas. And we’ve been able to use those to to inform our philosophy and our curriculum. At Optimizely. We had something similar in the past, I’m trying to remember what team initiated that but the point is, like, you don’t necessarily always need to do this work from square one within your customer education team.

Mike Di Gregorio  26:34

Yes, 100%. And I think it’s a good reminder, too, that education is another like learning. For me, it’s like so aside from instructional designs, that there’s many people who want to have a stake in customer education, when they start to see the impact it can have with all different parts of the customer journey. And so use it as an opportunity to build those bridges with and to see how customer Ed can really impact all those different parts of the customer journey and how you can play a role there. Because I think it’s a really, and Alex D can really give you some language to like think through the different parts of it, and different disciplines that make it up, right, that make up the larger super discipline can really help you just navigate those conversations internally, if you’re struggling,

Adam Avramescu  27:18

that makes a lot of sense. And it also helps probably, in some ways translate the language of instructional design to the language of the business, which can always be a hurdle. So I’d love to dig into that. Yeah, huge hurdle. I’d love to dig into that a little bit. Because when we’re talking about the design of an individual course, or a solution that you’re you’re coming up with, there are some evidence based practices that have been emerging, some of those fly in the face of, you know, like, we like to say in the intro to the show some of the myths and misconceptions about how learning actually works. So when you’re, when you’re thinking about design, I would love to hear a little bit about how you incorporate some of these evidence based practices into your customer education program. And maybe in some ways, like how, how you use them to enhance learning transfer. Because ultimately, if we, if we’ve done some of our, our research, and we know who our personas are, and we know what their job to be done is and what performance outcome we want them to have. And we’re refining based on our, our feedback, we still ultimately have to like, design the course or design the solution. So what’s actually what, what techniques are evidence based? What’s out there? What shoul d we be doing? How do you guys think about incorporating them?

Mike Di Gregorio  28:37

I love this question. I love this question. And

Adam Avramescu  28:40

I like my questions.

Mike Di Gregorio  28:42

Yeah, it’s, well, the first thing I would say we so every, every time we have a new customer, new group of customers very first thing we ask is what your goals are. And it’s funny, they don’t know how to answer that question. You know, they probably told our sales rep at some time, right, that they knew what they wanted to get out of the product. And the really, the reason that I think like you have to start there, because the emotional investment in learning will actually make the learning better, it’s the first thing. So they need to tie everything you’re doing back to that goal that your customer your learner has. And then there’ll be self motivated and driven to, to seek it, right. And ultimately, you want them to be that you want it like drive them with that internal motivation as much as possible to seek the thing. And you need to show them the baby steps along the way that they’re actually doing well with it. Right. So like, like, this is gonna sound really funny, just like just be a good coach, right? Like, you know, just like, show them how well they’re doing, share the good news with them, even if it’s small, like you know, just like fake it a little bit. And then they’ll you’ll you’ll drive them to great behavior. Second thing is we really strive as much as possible to, to build something that’s going to be immediately relevant for their life. Right. So in my case, it’s like as soon as as soon as you start your training program with top hat you’re building like a week’s worth of material to deliver to your class. And you know, at the and have your first hour with us you can, you can go do this, right it’s ready to use there, you’re, you’re ready to go, you understand the basics of like how to make your dream of like the future, you come to life in a small way, right now. So so just make sure everything is like immediately relevant can can immediately be used as no extraneous material, which then, you know, because the learning transfer really excels when the learning can replicate the real world environment in which you’re gonna have to apply the learning, right? That’s a key thing. So the more that you can build that into your customer education, process and training, the better off your customers will be the more successful and quick query, the more quickly you can measure the outputs of your of your training, because the you’re replicating the environments in which the, you’re the learning environments, replicating the environment in which the applying has to happen, the application has to happen, which is a criticism of Bloom’s, right. Yes. Like, like learners are not apt to think in a pyramid scheme of how you might be learning, right? It’s just like, No, you

Adam Avramescu  31:09

don’t learn. And then and then you start like, you know, evaluating the fact that I know I’m doing Bloom’s in the wrong order

Mike Di Gregorio  31:14

know, if I keep going backwards is actually much more successful. According to the literature. If you start with the creation, and the generation, it’s called of like, the concepts without having any prior knowledge of it, you’ll end up probably learning faster and more effectively. So it’s a it’s a turning it on its head. Yeah, more effective. And because the other thing is that it gets a relevance, right. Yeah, exactly. You get to relevance 100%. And the other thing is that good learning sucks. And by what I mean by that is a good learning if you’re a learner is really hard and painful. And so one of the, one of the great, I guess, like puzzles of just measuring us, like, measuring satisfaction or smile sheet, is that if you create an experience that feels good, you’re your customers, and learners may not have actually learned anything. And they start through a real fun presentation, the work of learning is often hard and difficult. And so you have to find a balance between like the, like real challenging training and just like fluffy, great, fun hour to spend together. Because there’s a medium between those two where you can you can really, you should really be aiming for it.

Adam Avramescu  32:23

I think good learning sucks. Now we have our episode title, but how do you? How do you? How do you how do you how do you bridge that gap, though, like, let’s, let’s say for a moment in customer education, we’re, we’re often in a position where we are. Sometimes we’re educating people who have to be there, and maybe someone bought the software, and then they have to be there because they’re implementing the software. But a lot of times they don’t, right, a lot of the times we’re working with very distributed customer base, or we have an opt in program and or customer education is playing almost a marketing function. How do we balance that, that on one hand, that desire for effectiveness and for learning transfer to make sure that what they learned in the classroom will actually be transferred to their their day to day job with with meeting that rigor, but on the other hand, knowing that this is opt in, and we have to, like actually get them to voluntarily raise their hand to have this sucky experience that we know is ultimately for their own good?

Mike Di Gregorio  33:19

Yes. So it’s, it’s a funny thing with learning to, it turns out that if you just tell learners at the beginning, why something is hard, or why it’s designed a certain way, they will buy into it more. So if you are designing something that feels really tough. So I can imagine, like, you need to learn how to do Cross Filters, and Salesforce with custom objects, or some real nasty thing, right. Like, that’s not really a fun thing to learn. But if if, like you can highlight at the beginning of it, that, you know, this is why it’s going to feel challenging and hard. This is why we’ve designed it this way, trust us. And by the way, here’s the great thing you can do at the end of the whole learning program, and you’re really going to be like, like, beyond just a badge on LinkedIn. Like, here’s the actual tangible thing and your real life, the better version of yourself is Sam Holbrook. It likes to say, right, that they that that you can get by going through this somewhat, you know, arduous training process and customer education program, you know, that’s, that’s all for the better. And so like, really just like, like, not like just being really transparent with learners is the key there telling them why you’re doing something Why might feel different, why it might not be like a different training, like another training program that you’ve seen expectations before they go through the LMS module on, on what they’ll need to produce at the end of it. So all there, they’re anchored in all these things, and you can reduce some anxiety and get some buy in and build some trust even asynchronously, and you know, through through a screen.

Adam Avramescu  34:46

Okay, so on one hand, you’ve you’ve set the stage for them in terms of what they’re actually going to be able to do and what they’re going to be able to do should be relevant enough to their job that there should be kind of a clear why. You also mentioned earlier that There should be some sort of like emotional connection to the learning or some sort of personal resonance, which makes me think a little bit about I think I’ve heard Nick Shackleton Jones talk about this how learning is ultimately effective. And ad that’s an effective with with an A not effective with an E. Yes. And so if you’re not making some sort of emotional connection with what you’re learning, you’re less likely to retain it. So in some ways, the fact that it is it might be a challenging experience, or might be unpleasant at times, helping the learner understand that connection upfront. And that this might not feel like super happy and pleasant and frictionless all the time. Gets them. It helps them go on the journey, that some

Mike Di Gregorio  35:43

of the things that I’m hearing, absolutely, absolutely. And then what helps with it also is like constant feedback, the more that you can give feedback, the better. And the more if you can figure out how to give feedback asynchronously, that still feels personalized, even better. So like all this stuff can go a long way. A simple technique I’ve used because my one of our certification courses just runs asynchronously all the time. And it’s self paced. And, you know, when I, when I look at the sort of discussions happening between different users, if something’s good, I’ll just out of nowhere email that customer. And they’ve always very appreciative, they thought, no, you know, they think they’re typing into a void. And then it turns out that you find, you find something good, and you reinforce the good behavior, and they’re very appreciative of it, right. And it’s a very small thing. But I think what’s important, there’s just the habit of trying to find connections with your learner’s at every opportunity to then allow them to keep, you know, keep finding that internal motivation to drive forward with their own learning. So that way they can be we can feel like we’re less enabling them and more, or enabling the bad behaviors, as I’ve heard you say, before and more just helping them they’re helping support them to, to get what they need out of our products.

Adam Avramescu  36:55

Yeah, yeah. So this, this kind of makes me think as well, when we go into the design, well, maybe maybe bad behaviors just made me think about it. There are some pretty persistent myths in the world about what you know, what doesn’t doesn’t work in learning. There are some ones that instructional designers love to pick on learning styles. I think there’s some really controversial stuff, you know, within like the Ebbinghaus forgetting curve, and then there’s stuff that is evidence based, and we’ve we’ve seen to work at least through, you know, reliable and repeatable invalid experiments. Can you tell me a little bit about, you know, in, in your view, what you think, works, what, what doesn’t work? And maybe how you’ve incorporated some of the pieces that really do work? And are evidence based into your own program?

Mike Di Gregorio  37:46

Absolutely. So I. So I love the point about learning styles, let me tell a quick story. Because I know, teaching, teaching university professors is interesting, but sometimes they’re cognitive psychologists, who are themselves, you know, experts in the science of learning and how the brain works in my audience. And you know, I have to teach them something. But when I want to get to a point about learning styles off and I got, I got a like, Yes, from the first year, cognitive psych Proxes. And learning styles are entirely debunked, don’t listen to them. And let me tell you why they’ve stuck around so often. It’s because when you build when you when you build for three different learning styles, you actually end up by act, it’s like two wrongs make a right, you accidentally build something that’s universally designed for learning that allows your learners multiple ways to actually do something. And that repetition, and retrieval in different environments and different ways is actually like really good. But it’s based on that because you actually created that

Adam Avramescu  38:42

you actually create a double barreled learning which is effective, but you guys exact the wrong reasons

Mike Di Gregorio  38:48

for that are totally wrong reasons and with the wrong intention, right, but it ends up working. So and the other thing is, I think that I think all of us are in customer head, because we’re probably good learners, right? And we will unconsciously just end up like reflecting the way that we learn on to the world, right thinking that Well, I learned it this way, so it must be good. And so therefore a text written slideshow is like, is great because it’s how I learned but also because we’re good learners, we probably have things that bad learners don’t have like good self control and learning and the ability to like pull things together through a messy presentation. And so you end up just like through some kind of like weird intrinsic trait that most of us have here of just being good learners and and knowing how to learn that we end up late like replicating some some bad behaviors. And also I think the I know we like the seductive details are a fun topic. The what all so I think now the the literature is also like very clear on this that there’s a lot of distraction and the cognitive load is just too much for things like talking heads or text flying in etc. But what’s, what’s interesting is that there’s, there’s a way to do it well, if you’re super, super conscious of how what you’re saying and what you’re showing can reinforce each other. So this is something that we’ve tried to do. When we run like we run in person trainings, virtual trainings, asynchronous or self paced stuff, the words and the visuals need to reinforce each other, and there shouldn’t be duplication. And there shouldn’t be anything extraneous either. So if you can, if you can really think about those two paths, because your short term memory works on these two paths, and it’s somewhat independently to figure out how to how to keep it simple, and like pull back the extraneous stuff and just work on something that’s textual, graphical, at the same time, then you can really reinforce that learning in an effective way. So that’s, that’s how we’ve tried to bring it to life. Right.

Adam Avramescu  40:49

So like so. And that’s kind of the idea of double barreled learning is having that like multimodal experience, but where you have you have two forms of media that directly reinforce each other, no distractions, no, so so there’s a difference. You’re saying, you know, between? Gosh, what would be a good example of this, like having? Having like a graphics on the screen? Oh, go ahead. Go ahead.

Mike Di Gregorio  41:14

If you I was gonna say, so when I, when I teach this concept, I try to have a very clear slide that I’m talking over, right. And like, it’s like visually reinforcing the thing I’m saying, you we do this. So when I teach my team about something called flipped learning, or team based learning, and make them do it by actually learning the thing, so they’re just immersed in the experience as they’re learning it, right. And so it’s like the same kind of idea. You’re, you’re, you’re trying to do two things at once using all the media available to but like i But again, it’s the difficult there’s moderating your your desire to go crazy with, you know, like design elements, etc, you really got to pull it back. Turns out like simple, the simplest things are the best for learning. Which, which is, again, when we think about it, we think about like the external pressures on on us as designers of learning, do you want it to look great, but sometimes just black text on a slide is enough, you know, one word to reinforce the main thing of the 62nd talk track that you have is, is enough to transfer the knowledge and to be the focus,

Adam Avramescu  42:18

instead of feeling the pressure to make everything we do like a full scale marketing production, where we have talking heads and background music and all that, because there’s a certain point where that’s not actually reinforcing the learning point that’s adding what you call it seductive details, right? Yes, yeah, the things that will actually distract us and overwhelm us and overwhelm our working memory. And that’s when we talk about cognitive load. That’s what we’re talking about.

Mike Di Gregorio  42:42

Yes, exactly. And then a good learners, though, can can parse their way through that, which is, I think, the challenge for us like the we have to really check ourselves as customer educators, because I think all of us are good learners, and we probably can make our way through some badly designed learning and still think it’s good. So we should we should be careful.

Adam Avramescu  43:00

Yeah, sort of like how oftentimes, when you’re working with a Smee, they will recommend that you teach the course the same way they learn that subject matter, not realizing that you can’t just like open up their head and create a universal experience.

Mike Di Gregorio  43:13

You know, exactly. Yeah, it’s, it’s a it’s a tough thing.

Adam Avramescu  43:17

Yeah. So maybe last one, before we, we have to call it a an episode. What about, you know, there’s been a lot of research and one one of my favorite books around this is make it stick. And I’m not gonna remember all the all the co authors on that book, but they talk a lot about some of the proven techniques around retrieval, spaced practice interleaving, basically, ways that you can knowledge check yourself to encode that information. Can you tell me a little bit about that, and how you’ve used some of those techniques?

Mike Di Gregorio  43:50

Yes. So this is it’s these techniques that I was thinking of when I said good learning sucks, because they’re really hard. And they’re very disorienting. So when you think about how you learned something, or the last time you learned a new skill, you probably went about it via massed practice, right? You just like tried to memorize a thing, and really go through it just like like spend some time concentrating on something for an hour or two. And that kind of practice is really good for like short term retrieval. If you need to, like I need to know something by the end of the day, I just need to spend an hour studying it right now. So that works. Long term, though, horrible, does not work. And so that’s where you need to do things like retrieval practice spaced retrieval, because it turns out that the way your brain works the best, the the best thing for it is just practicing retrieving the information. And the more that you can practice retrieving it in an environment that represents where you actually need to apply it, the better for learning. So that’s that’s all retrieval practice is just the testing effect trusting yourself over and over again, spaced retrieval and spacing this out in ways that make sense and interleaving is but it’s my favorite one is very interesting. And it’s Admittedly, this is a really hard one for me to incorporate. Because it’s just, it’s kind of so bananas when you describe it. So what you want to do is, before you finish teaching a topic, you need to throw in parts of the next thing that you want to cover. And then before you even done that, when you throw in the third topic, and then by the time it comes to do a formal assessment, you’re measuring all these things together. But that’s actually how you learn, right. And this is certainly how you live day to day, when you have to retrieve these things, you don’t just get to, you know, you have learning objective one, and then you just have to think and work within learning objective one, you’re pulling in things that you know, from all over the place. And so interleaving is meant to replicate that, that real world learning experience in a structured way. And but at least from from my experience has been hard to build into our, our own learning, or our own customer education program. But we talk about it a lot with our professors who then can go do in their class.

Adam Avramescu  45:57

Yeah, it might be easier when you have that that sort of structure around your learning program. But hey, let’s let’s issue this challenge to our listeners, if you have a program where you have a learning path with a sequence of courses that are meant to go linear linearly, or if you’re running, say, multi, our multi day learning workshops, where you have the opportunity to do progressive knowledge checks, look up interleaving. See if you can experiment with this, there might be some really good creative opportunities to actually work it into your program and see how it resonates.

Mike Di Gregorio  46:27

Yeah. And I think the the main thing is like the disorient, the disorientation for learners, which you can get around again, just by being transparent with, with how you are designing your learning at the beginning. And luckily, this is something I learned from a very good educator that I get to work with, who’s a longtime customer at tapa. So I get to, I get to cheat a little bit, because the experts in the field are customers of ours, right? A nice thing. Yeah. But it’s a it’s a interleaving is very, very effective if it can be done well. And you know, all of this is reminding me I don’t like actually, the way I introduced these concepts, when I’m talking about them, and trying to teach them is reminding people that they don’t have to be a cognitive psychologist to be a good teacher, but also that there’s a world that isn’t just anecdotes of things that kind of worked when I tried them. And so there’s like a, like, usually, like learn, there’s like learning tips that your friend did. And then there’s the stuff at the brain science. And so there’s a sweet spot in the middle, which I think all of us are trying to occupy. And so if we can just keep that firmly in view with, you know, trying to test the things that we think work, incorporating the signs, and the good evidence based practices where we can, we’re going to be in really, really great shape as a as a bunch of experts in customer education.

Adam Avramescu  47:40

I agree and drawing upon a thread that you said earlier, sharing that, that transparency with your learners, if you’ve helped them understand from the very beginning of the learning experience, how this is going to be relevant to them, but also why you’re going to do some of the things that you’re going to do in the course and ultimately how that’s going to benefit them, then then you’re helping bring them on the journey to so I really I love that as a light bulb moment as well. Absolutely. So, Mike, as we wrap up, I’m sure there are many, many books, courses, blogs resources out there. For people who want to get deeper into this field. Do you have any recommendations on you know, if if anyone’s listening who maybe doesn’t have a formal background in ID or LSD? What would you recommend to them?

Mike Di Gregorio  48:26

So just let me let me preface this by saying I did not have any formal background before I actually had to start doing the job as an accidental instructional designer, like so many others. Like the one benefit is that so coming out of academia is that I had read all of the books that were informing academic practice for, you know, 2500 years, and I’d bet my tongue when you’re talking about effective because I wrote my dissertation on that, so did

Adam Avramescu  48:51

amazing. Yeah, I shouldn’t I should have quoted you not nickeled and Jones,

Mike Di Gregorio  48:55

no, abs Absolutely not. Nothing worth quoting in that dissertation. But nonetheless, so there’s there’s a course so there’s there’s Alex D courses kind of everywhere. Now, a few years ago, I took an intro to Learning Experience Design at the University of Toronto, which I know a few of us in the customer channel now I have to so I would recommend that one only from personal experience. Great Books on this oh, you know, that okay, I’m sorry, I’d have to think about that. The trend so I love make it stick. I love all of the all of the resources from retrieval. practice.org very, very good. The and everything in the user research and user onboarding realm. Like I really like Sam hooks book that like I thought that was really good. And I mean, I’m pulling from a lot of things here. But I would say ultimately, like just just read, just start and like start seeing how this stuff can be incorporated into your own learning, design and learning journey and customer education journey that I have debt of gratitude to your book, which really helped me think strategically about customer education and what it means. And there’s the theory and practice of like pristine customer read. And then there’s the messy business of what it means, at know your particular company in your particular moment with your particular customers. And that really helped me like you’re really helped clarify that for me, and I suspect many others too. So thank you. Yeah, so we’ll start there. Yeah. There’s a very nice book called intentional tech, which is geared towards the higher ed audience, but by Derek Bruff, which I which I got a lot out of some some neat stuff you can try there. By the Legion. So much. So man, I

Adam Avramescu  50:44

know there’s so much my favorite book.

Mike Di Gregorio  50:46

I know my favorite customer read is, bizarrely, the challenger sale. I think it’s all customer education. And I’m like, nobody believes me on this. But I mean, this is entirely a book about customer read. It’s not a book about sales.

Adam Avramescu  50:59

No, I agree. It is a book about customer education. And and there’s the Challenger customer as well, which is yes is relevant, if not more, so

Mike Di Gregorio  51:06

100% relevant how to train a group and how to win a group over and no change management with all this stuff. So like there’s so you can find customer education everywhere, when you start looking, which is sort of why my my, my recommendations here are all scattered and all over the place. But we’ll

Adam Avramescu  51:22

see if we can curate these in the episode description or on our website at customer dot education, and maybe a few others that we think of afterwards because you’re right. There is

Mike Di Gregorio  51:31

so much on a Weber’s book was amazing, right? Yeah. Just Oh, yeah.

Adam Avramescu  51:35

Onboarding matters. Yeah. We had her on the show not too long ago. Yeah. Well, Mike, thank you so much for joining us today. It was really great talking to you about the nitty gritty of learning experience design.

Mike Di Gregorio  51:47

Oh, Adam was a it was an honor and a pleasure. And I’ll cheer for Montreal once in your honor. The next time like alright, but only

Adam Avramescu  51:58

one right? Well, yeah, you know what? I’ll take it. I’ll take it. Maybe Maybe I’ll give you one back in the in the one Toronto. If people want to find you, Mike, where can they find you?

Mike Di Gregorio  52:08

So they can find me Mike underscore digs on Twitter. The handle is because I was very early adopter of Twitter, and Mike Di Gregorio on LinkedIn. The only Mike D currently a top hat, though we have many mics. The only Mike D. So should be easy to find. And you want to email me Mike D atop.com. Happy to chat. happy to chat of course, the customer ad network. Shoot me. That’s right. That’s right.

Adam Avramescu  52:34

Well, good. Well, we’ll see you around the way over there and for our listeners, until next time, keep on educating keep on experimenting and finding the others. And thank you very much.