[00:00:00] Dave Derington: Hi, everyone. Dave, here from CELab podcast. If you’re finding value in this podcast, we’d really like to get to know who you are. So we’re going to ask you to click on our site and have yourself to our mailing list. We’re going to agree not to spam you, but at some point in the future, we may have special events, certain podcasts we want you to join on, or we might even throw in a class or two.

So sign up today. It’s not going to cost you anything. And again, we promise not to bug you. Thanks everybody.

Adam Avramescu: Welcome to CELab the customer education lab, where we take customer education, myths and misconceptions, and give them the traditional Viking funeral at sea I’m. Adam Avramescu and I am here alone today without my co-host Dave Derington. And that’s because we’re doing a little bit of a mini episode now.

We’ve actually done an episode like this before. these are instructional design one Oh [00:01:00] one episode. So consider this a very quick and dirty instructional design school. And one of the reasons that I’m doing this episode is because. We’ve actually had a lot of folks that we’ve heard from recently who, you know, either because of COVID-19 or for whatever other reasons, they’re looking to transition into more formal instructional design types of roles.

And so if we can do a little bit to help here, we are happy to. And Last time we did an episode like this, we did it on Kirkpatrick’s four levels of evaluation. And I actually have a bit of an error to acknowledge from that episode, in the interest of accuracy when I was talking about Donald Kirkpatrick, whose framework, that is, I was actually speaking about him in the present tense and what I didn’t.

Acknowledge or maybe I didn’t realize it at the time was that unfortunately he had already passed away. I still had relatively recent memories of [00:02:00] going to a conference where he had spoken. I think his memory was alive and well, but anyway, I wanted to acknowledge that correction today, we will be talking about a different theorist Benjamin bloom, who is a different person than Benjamin button in case you were wondering, he did not age backwards.

So Benjamin bloom was born in 1913. Died in 1999, but he was an educational psychologist. And specifically what he is known for is developing a model called Bloom’s taxonomy. That thinks about the different levels of sophistication that you can use when developing an educational or instructional activity.

His taxonomy is all about measuring the different levels of what we would call the cognitive domain. So different levels at which you can know or think or cogitate about something. And this is different from the effective domain, which is all about, [00:03:00] emotions and social aspects and how you feel about things.

and the psychomotor domain, which is all about physical and kinesthetic. So cognitive domains are all about thinking. And in a lot of cases, when we are developing instruction, what we’re really trying to focus on is. Thinking about the ways that people think if that’s meta enough. we’re certainly we have a role to play in terms of changing how they feel, and developing social opportunities, giving them opportunities to practice things with physical and kinesthetic approaches, but really it’s that cognitive domain that we tend to focus on the most.

And that’s why I Bloom’s taxonomy becomes a really helpful, guiding device. Really quickly, let’s talk about the taxonomy that he developed. And we’ll talk about the different levels. We’ll talk about how it’s visualized and we’ll talk about how it has been adapted since, because it got a pretty big overhaul in 2001, actually by one [00:04:00] of Bloom’s original partners.

So here are the different levels at which you can think about the sophistication of what you’re teaching someone. So the first level of Bloom’s taxonomy is the lowest order is knowledge. So do you know something? Can you remember it? Can you retrieve it from your memory? next up is comprehension, okay.

something, you can remember it, you can recite it, but do you actually understand it? Can you restate it in your own words? Can you identify something about it? Can you describe it? The third level is application. okay. Maybe, you know what it is, maybe you understand it, but can you actually turn that into some sort of a situational aspect, right?

can you use that material and incorporate it in a new situation? Level four is analysis. So that takes it a step further. Now we’re not just applying it [00:05:00] to a situation, but we’re. Making some decisions about what the appropriate course of action is, or we can break things down into its parts, and understand the structure.

Number five is synthesis. okay. We can analyze something, but now it’s a kind of a higher order of thinking perhaps to actually create a new. Whole, a completely new project based on those principles that you learned. And then the final, tier for him in this taxonomy is evaluation. So he defines that, as the ability to, maybe critique the material, to evaluate its worthiness, to, to judge it or argue for or against it.

As you can imagine as you go up the taxonomy, you’re thinking about things that are inherently harder to test on, harder to teach, because they’re simply more complex. I could teach you in, a few [00:06:00] minutes, to recite something from memory. Or to remember a list of something, but it’s probably going to take a lot longer for you to be able to create some of these projects, net new top to tail, or for you to really develop a sophisticated analysis and evaluation of something.

Now throughout you probably heard me using some verbs. And when we think about Bloom’s taxonomy, thinking about the verbs, can be a really useful way of thinking about how to apply this on the job, because really. Bloom’s taxonomy in a lot of cases is used to evaluate and structure learning objectives and learning objectives.

As we, often say are all about the verbs that drive them. we’ll come back to the verbs in a moment because I actually want to talk a little bit about how Bloom’s taxonomy got revised in 2001. And that revision was actually done by Lauren Anderson, who was one of Bloom’s students. And, David , who was actually Bloom’s [00:07:00] original partner.

So even though bloom gets a credit for the taxonomy, it’s his name? That’s attached. David Krathwohl was also doing a lot of this research with him and is also credited in a lot of cases for a lot of research around the effective domain. Let’s talk a little bit about this restructuring. one of the restructures here was instead of using the old nouns knowledge, comprehension, application analysis, synthesis, evaluation.

now it got restructured actually as verbs are or as Jarons. So instead of knowledge, we now have, remember we’re remembering, so here, this is about again, it’s still the same thing. It’s, we’re calling knowledge from memory. it’s all about just recall, Yeah. Basic recall recalling definitions, facts, lists, in a lot of cases, it could even just be a trivia comprehension turned into understand or understanding.

So again, we’re going one step beyond just knowing the thing or being able to recall the thing. Now we’re proving that we can interpret [00:08:00] it or classify it or explain it or compare it to something else. Application became apply or applying. So this is actually now carrying out some sort of action, that builds upon the knowledge that you learned.

So here, this might be applying the material in a simulation or a model or a role-play or something like that. Actually applying the skills that you should have learned analysis becomes analyze or analyzing. same concept again, it’s still about breaking materials into parts. it’s maybe about creating mental models for things, being able to map out, the parts, versus the whole, those types of pieces.

maybe again, still some sort of like discovery or inspection, or building experiments around the knowledge that you should have learned next synthesis. [00:09:00] Actually becomes something a little bit different. So one thing that happened is that synthesis became create or creating. So instead of the idea of just taking the parts and pulling them into the hole, this one got rearchitected a little bit about creating something completely, net new, creating a new product.

Based on that learning. So it’s similar obviously to what synthesis was originally defined as, but it’s interesting to see that’s one where we didn’t just take the original verb. It didn’t become synthesized. It became create, and evaluation became evaluate or evaluating. But one interesting thing here, like I mentioned is.

These ones changed a little bit more meaningfully because they got flipped. So while in the original Bloom’s taxonomy, the order of the taxonomy was from least to most complex knowledge, comprehension, application analysis, synthesis evaluation [00:10:00] in the 2000, 2001, you go from remember, understand, apply analyze.

So all four of those were the same. but now the last two flip, so after analyze is evaluate, and then after I evaluate. Is create. So they’re making a statement here that in some ways, being able to create something is a more complex task than just evaluating it and being able to, make judgment calls about it.

And in a way I think that actually does match the change in terminology. You can pull the parts into a whole in terms of synthesis, but maybe, evaluation seems a little bit more metacognitive than just synthesis and building a project. But when you frame it as evaluate versus create, it’s like the difference between being, a movie critic and being a director.

Not that there’s anything wrong with movie critics. I love criticism. So when we think about the depiction of Bloom’s taxonomy, a lot of times it’s depicted as a pyramid. So [00:11:00] you’ve got remember at the bottom and you’ve got create at the top. And a lot of times, again, I’m just looking at one particular visualization.

This one’s on vanderbilt.edu. so Vanderbilt’s website, the verbs that they have attached to these for remember, they’ve got the definition is recall basic facts and basic concepts, okay. What might have learning objective for remember, look like the verb might be defined list. Memorize, repeat state.

For understand, this is explain ideas or concepts. What verbs might you use here? It might be classify. Describe, discuss, explain, identify, recognize select four apply. Now it’s about using information in new situations. Your verbs might be execute, implement, solve, demonstrate, use. And this is interesting, right?

Because as we think about this, we can think about that. The learning objectives that we’ve created [00:12:00] over time and granted, not all courses have learning objectives. Some people do away with learning objectives, entirely, at least as they’re presented to the end user or the learner. But in general, even if we’re not necessarily presenting the learning objectives to them, we’re at least doing some work on the back end to really know.

Like, how will we know that this learning was, this was successful sometimes, when people think about learning and when they think about what a learning objective or instructional objectives should be, they think about it in terms of, Oh, for learning to have occurred, the person just needs to understand these concepts and that’s how you end up getting all these learning objectives that are written.

Like by the end of this course, the learner will understand how, feature X in this software works. And that’s usually, but not always, but usually it’s not a great learning objective because it’s, first of all, it’s a little bit too high level, but second of all, you’re not necessarily thinking of a way that you would [00:13:00] actually monitor and observe someone understanding that at the end of the day.

so according to Bloom’s taxonomy, someone will understand that feature. Maybe if they can describe it or if they can identify it or if they can locate it. But, when a lot of people talk about learning, that’s not actually the objective that they’re driving for, Someone might tell you, Hey, we really need to create a course on feature X because we want our users to understand it.

And then you can ask them, okay, if this course is successful, what do you want learners to be able to do? Do you want them to be able to describe the feature? Do you want them to be able to identify it in the product or do you want them to maybe, use that feature or demonstrate that they can use that feature?

And in a lot of cases, the answer would actually be, yes, we want them to apply. Their skills. And that requires us to create a little bit of a different course, maybe different measurement activities. maybe it’s more interactive if we actually want them to apply these skills. And if you heard in a previous episode, when I was talking with Daniel quick, we were briefly railing on the idea of.

[00:14:00] jeopardy, sorry. I was re I was railing on it. Daniel likes jeopardy activities, and I don’t mind jeopardy activities either, but we were talking about how jeopardy as, an instructional device gets misused, because it’s a game it’s fun. It’s inherently competitive, but often it’s used to measure the wrong thing.

So if I create a course where at the end of the course, I want someone to use information in a realistic scenario instead of just remembering it by memory. And I use a jeopardy game as the assessment at the end. And I am probably not actually testing the right level of Bloom’s taxonomy compared to the result that I want to see.

So my instructional objective was centered around the idea of them applying something. But what I tested in that jeopardy game was really just whether they remember the concept. Or maybe at best sometimes, I don’t know, maybe jeopardy games can go into the understand tier and I’m focusing a lot on these bottom three parts of Bloom’s [00:15:00] taxonomy, because these are the ones that I see most often.

in learning and especially in customer education, there are certainly customer education programs that go up into the higher tiers, analyze, evaluate, create. If you think about that might be in more advanced certification programs. For example, that require a work sample from someone to be able to say that they’re certified, you might see this in, lab based.

Interactivities where people, are given a prompt and kind of are given a sandbox to complete some sample activities that require them to not only apply their skills, but maybe, respond to certain problem statements in certain ways. So they’re out there. They’re just not the most common things that I see.

And part of this is, it’s what, when we were talking about, Kirkpatrick’s four levels. The levels at the bottom are the most widespread partially because they’re the easiest to create and the easiest to measure. It’s really, pretty easy for me [00:16:00] to remember, or sorry for me to measure whether someone remembers something because I can have a quick, multiple choice quiz that measures that it’s a lot harder for me to really measure, whether they can produce new or original work or, have a great way of measuring that more objectively.

So I think that’s part of why you see it less often. One thing I will say is that a lot of, the debate in discussion around in product education and around using in product activity, as a way to measure learning is a way to try to shortcut up to the higher parts of Bloom’s taxonomy. So if we could say, someone takes a course, on feature acts, and then we’re going to measure whether they’re actually using.

That feature afterwards. That’s us trying to get rid of some of the proxy of measuring whether they remember or understand something and actually skip straight to a, whether they applied those skills. So I think it’s an interesting and worthwhile idea, but you don’t necessarily see everyone have the tools to implement those successfully.

When we think about using [00:17:00] Bloom’s taxonomy in the real world, again, I would say usually it’s about. Selecting learning objectives for the course. And furthermore, thinking about how those learning objectives will inform, not just what’s taught, but what’s measured, how we assess it and ultimately what actions that we want to see at the end.

And again, there’s great resources out there that talk about the actual verbs that you might use to measure at each level. I’m looking at one right now. tips.url.edu has. I guess that’s the university of Arkansas has a great resource on the different verbs associated with each level of Bloom’s taxonomy.

And they’re not, is that different from the ones that I read in the Vanderbilt article to you a little while ago? So I won’t belabor the point, but, as much as to say they’re out there, these resources are out there. And even if they use different verbs to describe it’s not that there’s just one set of verbs that’s appropriate for each level.

It’s more about thinking about what you’re actually trying to measure. And the main [00:18:00] way that I use it on the job is to really detect when there are mismatches in terms of maybe what someone told me they wanted the course to be about or what information they wanted to include or how they wanted to test it.

And then what result we actually expect to see at the end. So I’ll actually try to make this point a little bit more clearly Bloom’s taxonomy does not necessarily inform the order in which you teach something. Again, it’s really about what you’re trying to measure, what you’re trying to achieve with the course, how sophisticated the output should be.

So to do this, I’ll use an example from a book called design for how people learn it’s by Julie Dirksen. It’s one of my favorite, very practical instructional design primers. so it’s great for someone who’s just getting into the field or someone who wants to refresh their skills and the example that she’s giving in this book, it’s actually around someone who is taking a course on the principles of effective design.

So someone who wants to learn about, contrast repetition, alignment, and proximity, those would be the core [00:19:00] concepts of what makes something effective or ineffective from a design standpoint. Now you could teach a course that really just, defines those four different, principles of visual design.

And you can be done with it, but that person probably wouldn’t be very good at actually going and designing something themselves. So you could teach them the traditional kind of bottom-up techsonomy way. So you could start by explaining what those four principles are and what they mean, and you could have them, then start to apply their skills.

Maybe like the example she gives is organizing the elements of a webpage using the principles, and then all the way to finally. Creating a lab from scratch. This is typically how we scaffold a course like this. We start with the most basic, we start with the conceptual, but you could use Bloom’s taxonomy in reverse sequentially.

And of course, just as easily. And in fact it might be a more interesting, learning experience. So let’s say you start with [00:20:00] create, the example she gives here is start the course with some actual, elements that you could use to build a webpage product photo copy logo. and then have them create a mock-up.

So they’re actually going to try their skills and they’re probably going to fail. So then the next part of the course could be having learners compare their mock-ups to a few professional examples and have them discuss what they did right. And wrong. Then you could break down that discussion into some of the design elements that work versus don’t work.

And you could group the elements that work. Into the four design principles. So now the learning has become very experiential. You go on and on. So now, you get to apply. So you could have them redesign their original designs, using the principles that they’ve just grouped, those successful elements into.

And finally you can cap off then not only by summarizing the design principles and flushing out the definition. So that gets us to understand. But you might be thinking of this point. how do we put, remember at the end? [00:21:00] Like why do we even need to remember the design principles at this point?

it’s actually, because we’re going to go use them on the job. So you could as a final step, and again, this is still her example. You would have students create a cheat sheet. Or a job aid of those four principals so that they can then go and remember it on the job. So you don’t have to have them, remember it by giving them a multiple choice quiz or a jeopardy game.

You have them remember it by giving them a job aid or a cheat sheet that they actually use to remember it. And in some ways that’s more realistic to how we actually work, right? The things that you learn in a classroom or that you want to learn in a classroom, or sometimes the higher order skills, the things that are more cognitively complex.

And then for the things that you just need to remember, or maybe principles that you just need to defined, that’s probably not why you’re usually taking a class you’re usually going to a help center for that, or job AIDS or resources. So a good customer education program often includes all of those components for that reason.

Okay. So hopefully that [00:22:00] was a really helpful, or at least a somewhat helpful intro to an instructional design concept. and for those of you who want to learn more, there are. So many resources out there about Bloom’s taxonomy, many universities like the ones I mentioned Vanderbilt and university of Arkansas have very freely available tutorials online, and then design for how people learn by Julie Dirksen is the book that I recommend.

doesn’t just cover Bloom’s taxonomy, but it covers a lot of instructional design one-on-one based on our analytics. I think the last time we did this episode on Kirkpatrick’s four levels of evaluation, that was a very. listened to and well downloaded episodes. So let us know if these are valuable to you, cause we can keep doing them.

There’s more instructional design concepts, obviously that we can talk about. Other than that, though, if you want to learn more, we have a podcast website at HTT, T P S whatever letters you put before, a URL, customer.education. Where you can find show notes and other material [00:23:00] I am at FMSU on Twitter.

Dave is @davederington special, thanks to Alan Koda for our theme music. And if this helped you out, you can help us out by subscribing and Apple podcasts, overcast, Stitcher, Spotify, or wherever you get quality podcasts. Leave us a positive review on Apple podcasts because that really helps and ultimately will help expose the podcast to other people and help us keep this thing going.

So to our audience. Thanks for joining. Go out and educate experiment and find your people. Thanks for listening. [00:24:00]