Dave Derington  0:00 
Welcome to CELab the customer education lab where we explore how to build customer education programs, experiment with new approaches, and exterminate the myths and bad advice that stop growth dead in its tracks. I am Dave Derington. And today I’m being joined by Debbie Smith.

Debbie Smith  0:18 
Dave, it’s great to be here with you today. I’m Debbie Smith. I just recently joined Smartsheet as the senior manager of Smartsheet university where I’m bringing together the entire University City concept, including new certifications, which we will be launching in the next coming in the coming year, we’ll be doing legally defensible certifications.

Dave Derington  0:39 
Woohoo, okay, and that’s my friends, this is exactly what we want to start today. So let me set the stage today. This episode is part of what I’d call our practitioner series. This is where leaders from the customer education space, step up and show their voice about some of the major issues that we have to encounter as customer education leaders today. In a sense, we’re calling this customer education 200 level material, we’re going deeper. And this is important because, you know, we’re seeing really heavy interest in companies on this subject of certification. You might have had your leadership come to you and say, hey, guess what, I want you to roll out three, four or five certifications this year, or something like that. Maybe you’re doing research, maybe you’re trying to figure out okay, well, that’s, that’s interesting. Let’s go and try to do this. Let’s see going to take, maybe you’ve even tried to develop your own certification once, maybe you’ve done it twice, maybe you’ve had mixed results. So I’m going to speak personally, right? As, as the biggest help that I’ve had to understand this space is from Debbie here. So we’re gonna get into this Debbie was senior certification Manager for quickbase, then you were recently a braised now you’re at Smartsheet. So we’re really excited to see your growth and to see where you’re going to go next. So let’s get into this. Here’s the hypothesis. certification programs for SaaS for software as a service specifically, can be delivered in much shorter timelines. But with caveats, true or false? What do you think?

Debbie Smith  2:21  
Absolutely. Definitely can be done faster. Okay, I think that we have to think about is it’s a risk factor analysis? Hmm. So the most important thing is to understand what the steps are and why they’re there. And then how we can tweak them to make them happen faster.

Dave Derington  2:39 
And that’s, it’s kind of hard, right? So I think it’s really, really important to understand one thing, certification program isn’t something it is not something insignificant. What’s the gravity? What’s the weight behind that phrase? Like, what is it really like, you know, I take a test, and I take a certification, what is the real difference there?

Debbie Smith  3:03 
So certify certification was actually defined what we have to do in labor laws in ’99. Right? So that’s, that’s why it’s a big thing. Certification. Why do companies want it, they want it because they want market recognition. But they also want to be able to hire and promote people who are certified because they want to know they have those skills, right? Oh, if you want to know we have the skills, then you have to have a valid and reliable exam. If you don’t follow the full process development methodology. That’s not you can still shortcut things. But you have to follow the full methodology to actually get a valid exam. If your exam is not proctored, it’s actually not going to be valid, because the questions are going to be out there. So some people have the answers. If some people have the answers to your questions, no, your test is no longer valid. That’s a piece that people miss a lot of times,

Dave Derington  4:00 
a lot of times as SaaS companies, we really get excited about that. But this is something that is going to likely take a lot of money and a lot of time. And what we’re talking about today is how to shortcuts a lot much of this, but it still does not obviate the, or is still extremely significant for us to think about how much goes into this right? Like, you might need to hire a psychometrician or consultant, you may you will definitely need to have staff. Right. Okay. Okay. Yes. Again, we’re just kind of framing up again, we’re not getting haven’t gotten into all the steps that we can do to talk that I really want you to open up and talk about ways we can accelerate.

Debbie Smith  4:42 
But think about the time input for it’s just, you know, just like in numbers, and this is a rough estimate. My rough estimate is that it’s about 1000 hours of work to go into a certification. Wow. And that’s a lot of work. Right, for one person, I mean, there are there’s 2000 hours in, in work year, right? That’s half your year. So but what we’re really talking about is there’s approximately 500 hours for the test developer or certification manager, whoever’s managing it, doing the reviews, doing the psychometrics is about 500 hours in that, then there’s about 500 hours of time, you have to remember that because that Smee time, if you can’t get that Smee time to develop the exam, you’re gonna have a really rough go of it. Goodness. Well,

Dave Derington  5:34 
he also talked to me a little bit about the money side of things is, from an average point of view, what what kind of number would you ascribe to, you know, a basic, no frills, certification first run kind of program.

Debbie Smith  5:50 
So the industry data that’s out there from cert mag, estimated at about $300,000. To create one, the first exam, whoa, like one? Yeah, and now my numbers are lower than that. But that is what, what it’s estimated out in the industry yet.

Dave Derington  6:07 
That’s tremendous. So that that incorporates not just the product and platform, but also like fees. And people

Debbie Smith  6:16 
write people in implementation to deliver it for the year.

Dave Derington  6:20 
Wow. Okay, so that’s a lot.

Debbie Smith  6:22 
That’s a lot for the first year that of course, your costs go down after that. But there are also it depends on how much consulting fee you’re paying, or whether you’re using internal folks, consulting fees can get really expensive. For an exam, it can run you just to develop it per exam, it can run you 50 to $150,000, in consulting fees if you pay an outside consultant,

Dave Derington  6:47 
right. But sometimes that’s that’s warranted, particularly like, if I were to frame this up from my own personal experience. For me, I my leadership goes, Oh, okay. that much. Wow, okay. How many? Wow, okay, I’m listing all these things out, but then the stark reality hidden somewhere in the middle of the project. And that’s what I want to get into today. The look, let’s think of the real gravity of this, you have tremendous experience, I have a little bit. But I have this pain that that’s lurking that, like I want to I want to relieve this pain, I want to understand going into my next one, how much faster I can get. And this all started, because practically, we’re trying to do this ourselves, we want to do some more certifications, they’re going to be different spectrum, the first one we started was probably the hardest one we could have done. And that was not bad. But now we’re gonna go the next step. So

Debbie Smith  7:42 
the first one is always the hardest one day was the hardest one for a number of reasons, they’ll be on the development, too. It’s the socialization in the company and the company not understanding the value. So trying to get the SME. So you have a lot of SMEs, who are volunteer, but their time isn’t actually carved out on the calendar because even though we’ve explained what the time is going to be that they there’s not a full understanding. And to your point about bringing in services, if you don’t have someone who already understand certification, bringing in a consultant at first, for your first exam to walk you through all of the processes, and do it in a very, very formalized way, like you did gives you that understanding of what the full process is, and then we can talk about how you can shortcut it. But if you don’t understand the whole process, it’s a little bit harder to shortcut it because you don’t understand the whys you do it?

Dave Derington  8:39 
Yes. And that’s why we did that. It was extremely helpful for us because I’ll give a shout out to Patricia Young from Kryterion who is our, our, our savior, a lot of ways, because she knows the space, she could keep us calm down in moments of stress and anxiety. And I’m going to tell you the first run through if you don’t have someone like Debbie, you’re starting out at Smartsheet now and you have the knowledge so that’s amazing. You won’t need to get outside resources likely, except to help you out. Well, I

Debbie Smith  9:12 
actually am I’m actually using to oh my gosh, yeah. So yeah, she’s gonna because I don’t. I’m in a position where I’m kicking off a certification program and I don’t have a full time dedicated staff. I don’t even have one staff member yet. for that role.

Dave Derington  9:29 
There you go. So Trish, hello. We’re gonna hear, ears are ringing. That’s That’s amazing. So, okay, let’s start from there. Let’s start backing into this. And let me just kind of do a preview of the subject areas that that we’re going to cover today. So kind of like an index. First, I’d like to start this out. Again, we’ve already talked resources a little bit. So let’s dive into project management because I understand from from your perspective when we’ve talked before, that’s one of the main pillars of certification then Then we’re gonna roll through some of the key areas that you might get stuck on in certification, I know that we struggled through some of these, the job task analysis JTA, then item writing, and oh my goodness, there’s so much interesting stuff in involved in that. And there’s a lot of logistical issues that you need to overcome. Data beta testing, got got some lessons learned to share in that one. And then the back end or the actual underpinnings the insert interstices is what I like to use the guts of what’s on making this go. And then, for example, when you get launched, you got to maintain this, you got to keep it up and running, you got systems to work with, you need processes behind that. So let’s start right back at the top, Debbie, the obvious thing is, you have said to me, and I believe you and I just believe you, I have experienced this project map is all really comes down to project management. So what do you mean by that?

Debbie Smith  11:01 
Well, even if you’re going to work with someone as great as Trisha, you still need a project manager on your team, you need someone to manage that project, because she’s going to do the consulting for you, but she’s not going to wrangle all of your sneeze. In exam can’t, it can’t be written by one or two people, you really need an army to write it. When I first started, I always said I had 25 people to write it, because that was the best practice of 25 people right? Again, getting 25 SMEs for that long is really hard, you really need to have at least five people writing your questions so that you have different, different questions coming in. Right, right. So who’s going to wrangle those who’s going to set up all the meetings, there’s a lot of meetings around this, you also have to schedule time on people’s calendars for them to actually do the work and make sure that it gets done. And you have to hold people accountable for delivering when they said they were and on time, if you don’t hold your speakers accountable to delivering on time, your exam development time can double, triple, quadruple, right. And unfortunately, I’ve had that experience. Or if you have the wrong SMEs, you’ll have the same problem. I also had that experience where I’ve made a couple of mistakes in my sneeze. The first mistake that I made on one exam that I was doing is I thought I knew who the right subject matter experts were, I brought them all into write questions. And when they wrote the questions, I found out they were the wrong space.

Dave Derington  12:34 
Oh goodness,

Debbie Smith  12:36 
the same exam, if that wasn’t bad enough, I went external to have partners come in. But I forgot to put in the prereq that they were all native English speakers, or fluent English speakers, and we were only delivering the test in English. So the questions that we got, were not things that we could actually use because they weren’t understandable. Wow. So the project management of all of this is really a lot of work, especially if you’re pulling in partners, or customers to write you kind of wrangle cats.

Dave Derington  13:09 
Yeah, you know, this is a, this is a moment for me where I’m gonna be a little vulnerable and transparent. Because as I’ve done this twice, and the second time, a real first time I did it, I did a certification program or as a pilot in and it was using all the best practices, but we didn’t really have a big massive test, right, like we would using criteria. And we’ve done that since it gainsight. And now an outrage. And the moment of, I guess, anxiety that really was palpable to me, was okay, we’ve gotten things set up, we’ve already announced to we’ve selected a team, we’ve we’ve got this list, we’re reaching out to them, we’re reaching out to managers, and we get into the thick of things. And my leader comes up to me and says, Hey, got some complaints. This taking the folks that are involved in this and realize how much time this was and, and there’s and in part of it actually was on me too, because part of them project management is not just getting this Misa line, but it’s your engagement with them. That if you don’t have your you know what to gather, and to facilitate and structure, that experience that they’re going to have, whether it’s the job task analysis, you know, objectives or item writing that then it’s you might have people sitting in a room not doing anything, or wasting an ungodly amount of time. And that’s not fair. But it’s also something from an accountability perspective you have to have. So though there are some areas in there that I really like want to dive in on as we’re walking through this,

Debbie Smith  14:51 
definitely because there are some of those project management things that I’ve definitely learned and I’ve gotten better at over over the years of doing this because I’ve now produced Seven legally defensible exams. That’s massive. And my team this year, in the next few years will be doing another five exams. So this, this is definitely something that I do a lot of.

Dave Derington  15:17 
I just can’t, it just can’t, you know, five exams in a year. And that’s where I really want to dig out Debbie is that it was a lot to do one. And I felt like it was a marathon and to get to the end, and I’m like, I don’t know if I ever want to do this again. But I kind of want to do it again. It’s that masochist trait I have, you want to learn more? Right? You think this is an end the the thing that I walked away from this first experience, Debbie was this like, we’ve been talking a lot we talk with other folks. There is a there’s an event happening. And you know, for time travelers, this is happening and may 14, we have a kind of like a meetup called C beers because medication beers if you haven’t heard of that, so this community Welcome to that. We can kind of commiserated with all of the everything that’s going on, you need to have a strong group, a peer group to help you out, you need to have a strong team. So let’s stop there on the product management side, it’s going to pervade this discussion. And let’s start talking about like, the actual steps as we progress through it, like the like you want to talk with your peers and get some get some help. Because nobody can believe, five and a year. How do we do that? We all want to know. Let’s start from the top. Let’s talk about the job task analysis.

Debbie Smith  16:35 
And you know what I just realized, Dave, we actually skipped the step.

Dave Derington  16:39 
We did.

Debbie Smith  16:40 
Okay. And that’s, that’s because I think on the fly, and I like, I stopped doing it. This is where like my project management comes in. It’s like, Okay, I got to remember to write this piece down. Because I’m missing this. What we’re really missing. First is the test definition.

Dave Derington  16:53 
Oh my gosh, yeah, we totally forgot about that, didn’t we?

Debbie Smith  16:55 
Yep, you can’t do a job task analysis, if you don’t know what you’re defining. So you have to create, you have to do a test definition, it’s a really a test definition. What that does, is tells you who the exam is for? Is the exam for an associate level or at advanced level. You know, it’s six months experience using your software and no experience using your software. Are you doing a product based? Is that a core product certification? Or is it a role based certifications, those are all going to change the way you think about your job task analysis. But they’re also going to be a bigger part of your, your entire certification picture. Because if you develop a core product certification, I will put it in writing that someone’s going to ask you for an expert, they always want that next level, right? So whatever you do, first, you’re always going to be asked for something to build on it. So that test definition when you’re doing that, you may actually want to start and think as a holistic view where you want to be in three years, you know, what all of your certifications would be in three years?

Dave Derington  18:00 
Oh, goodness. So this really goes back to our roadmap, you know, planning ahead. I know I will, named Rob bill, a shard because it Phil always talks about roadmaps, you know, two years, five years, something, you know, crazy. But I believe that, and I know that I work with that if I’ve gotten stuck in my role a couple of times, when you’re like going and going and going and grinding and grinding and grinding, and then all of a sudden, you’ve got a trance. I’m a chemist by education. So this thing called a phase transition. It’s that when a substance change, you know, it’s like ice melting, you’re going from a solid to a liquid or a vapor or whatever, beyond that. But it’s very subtle and different than all of a sudden, you’re like, Okay, I hit a wall. I literally hit a wall. And now we’re doing something completely different. And if I’m not, if I don’t have like a map to going forward, in, and I’m going to challenge the audience here, too, you should be thinking about certification when you’re starting out. So if you’re brand new, maybe a year in, it’s going to hit you somebody is going to ask you this, and you’re probably going to need it. But yeah, that’s a that’s a fantastic point, not only so then you’re saying not only think about the definition of the exam that you’re trying to build, but back up a step, get a whiteboard, sketch out all the other things that you know are going to pop out of this like for us right now we’re already looking at we released the admin one and function like, Oh, well, what about the admin 200 level that certifies partners on this advanced? Okay, what about our sales development reps at outreach? Now, that’s one of our main, you know, we call them a prospect or somebody that’s trying to reach out to you What about the managers of those? What about account executives? What? Oh my gosh, so now that’s where I am in this moment. And we’re thinking about what’s next

Debbie Smith  19:49 
in I know, we’re gonna stay on legally defensible certification, but that’s, that’s where you really need to think about your whole credentialing system and think about it as a credentialing system. Instead of thinking about just certification, and that’s also going to make your life a lot easier, because if you think about a credentialing system, and you put in some type of badging system, where you bring in badges for your certification, then add skills badges, or maybe advanced level badges that you can only participate in after you get your certification. That’s another faster way to get to a certified solution, without building as many certified certification exams. So there’s a few different ways to start to think about that in a whole ecosystem. And that’s also part of what I’m working on right now. At Smartsheet. I’m working, I’ve got a great team of people. And they’ve already done all the background work for this. So I’m just coming in to help pull everything together for them, because they’ve done great work thinking about the credentials. And so we will be putting in a credentialing system where we’re doing the legally defensible certifications, we’ll have certificate courses where you take a course and earn a badge. We’ll also do skills badges, all of these together are going to play into our credential system.

Dave Derington  21:07 
That’s fabulous. And I know we’ve done we did it backwards, Debbie, we actually Well, not so much we built our certification at the very end of that certification is when we implemented the credential the badge. And now we’re actually extending that back to the beginning of the story, such that we want you so again, I I could probably put a graphic up on the screen. At the moment, we have three learning pathways at the company, I work for outreach. So I’ve got gives you our branded today. And we have the admin. So that’s a really and most software companies in our space have this you have somebody that’s configuring the platform. To me, that’s almost a table stakes To start with, even though it’s harder. But the problem that I usually have had to solve both at gainsight, both had outreach is that coming in, the admin has had the hardest time, they don’t have the frame up, they don’t have all the pieces, they don’t understand anything. So one of the things we put a lot of work on in my current position that outreach, is we built an entire support system for the admins, all on demand. And then there’s a lot more office hours and things like that, so that they can go to earn their cert, now we’re coming back and re surfacing that, and thinking of all the different roles that we can do. But there, I don’t think I’m going to do it as high stakes, but I want to do it faster. So that’s what we need. We need your your wisdom.

Debbie Smith  22:37 
That’s we’re doing a product, a core product certification and a system admin, like you said, system admin is definitely a need, and then the core product, and then we’re doing the role based on top of those.

Dave Derington  22:49 
That’s cool. That’s fun. And as a user of Smartsheet. Frankly, I’m, this is so meta To me, this whole talk is, okay, we know each other, we’re friends, we’re peers, we’ve worked together in different projects, like, generally, and then I use Smartsheet. And I use outreach. Peanut butter, jelly tongue. No, it’s really cool, because I would love to go and start to earn those credentials. Because here, here’s where I’m at going in, let’s think about Smartsheet. You could use any of a constellation of software. And, you know, I’ve used Asana, I’ve used all kinds of different tools. But for for me, I’m finding more value in settling on one standardizing on that for the team that we’re on. But there’s things I need to learn. And the the thing about SAS and again, we probably get back on track here. But I think this is really important to say that I like the concept of credentialing leading up to a cert, because it’s showing that I’ve got like, milestones in my journey of learning. And then for me, I might come in and say, Okay, I don’t have a lot of time. I’m just like you. I’m a senior manager, customer education program. I’m doing a lot. I’m learning a lot. And then I have to stop scratch needle go learn that I want to do that in smaller doses. That’s why I think what you’re talking about a credentialing is super important because it allows me to step into it.

Debbie Smith  24:14 
Right. And it also for those of you who get a mandate from the CEO that they want a certification in three months. Like I’ve had that I know, Dave, you’ve had a similar request. If you bring the credentialing system in first, you’re often hitting what they’re actually asking for. Because when they when sea level, ask for certification, they’re usually referring to the badges that they’re seeing in there, that’s really what they’re talking about. They do want the certification to but they don’t understand the magnitude. So if you bring in the credentialing, it’s that quick win that you can get in the first place. Yeah, that’s why I like to launch that first. Now. I didn’t do that originally. That’s my new method of doing it.

Dave Derington  24:59 
Well, what Is that even a thing? I mean, I’ve seen the rise of cradley and incredible and other badging, tooling. But has that even been a thing as much as it is today? two years, three years ago?

Debbie Smith  25:14 
It started. I’m trying to think PMI pushed my badge out, like, I got my PMI certification in 2013. But then I think it was 2015 that they started with the badges. And I think that’s when I started to see more of the SAS companies do more of the badging was in 2015. But it

Dave Derington  25:32 
was in there, too, right? They’ve started doing the kind of badging platform of their own. Yep. That’s Salesforce.

Debbie Smith  25:40 
Salesforce, right. Yep. And, and so that 2015, it started to gain recognition. And I think you’re right, I think it’s just in the last two to three years that it’s become really big for sass companies. Yeah. And it could be very impactful,

Dave Derington  25:57 
hugely impactful, I know, I really want to see when the experiment kicks off, we’re kicking off a credentialing system for our main learning pathways very soon. I don’t have any eta on that. But when we relaunched, we launched that the admin cert program, we saw the certs hit LinkedIn, and Twitter, the podcast prior to this one, Debbie, just as we’re speaking, it’s launching, it was with Sam Cummings of LinkedIn. And we talked about this social selling, and I’m gonna have to embed a clip here of that, of him talking about my program, and like, parroting back to me like how exciting This is to see all this stuff happen in the badges appear on social, and that the storytelling that happens around that, for example, I will point out one person, friend of mine, part of our project team here at outreach, john phone, he can’t he I asked him like how I saw his badge online, on LinkedIn. And I saw the numbers were just ridiculous all the views like 1000s of views on this one badge. And he was what’s happening, how what this is amazing, this is amazing. This is significant all of our team, when they did that all of our customers, when they did that, you would see this resounding spread of the badge of the world and others are looking at like, I could get that too.

Debbie Smith  27:20 
I saw the exact same thing at braze. In three months, at braze, we certified to have over 250 people around the world. Within 90 days, we had a global certification program that there were people on all content, all populated confidence, taking exams, and sharing them on LinkedIn. And the shares and the reshares. Were just amazing. And it was just great to see all this in the feed that it was it was just intrinsic. It was great marketing that was happening organically. It was fantastic.

Dave Derington  27:55 
Yeah, marketing teams should be on alert for I work more and more with marketing on this kind of stuff. And it’s almost a gift to them. Yes, what’s happened, oh, this is amazing. This free marketing of our platform. And it’s easy to not think about that. Because if you’re so down in the weeds building a program, and then it’s actually really exciting. At the end of it, it’s so rewarding to see people and I, we definitely need to get on to the next topic. But I’ll tell you this story that I see, the customers and even our own team members just light up with joy when they when they see that badge and it goes out on social and then their peers are proud of them. Right, exactly. It’s just that part where it’s the heartwarming piece to certification,

Debbie Smith  28:42 
that and when you get someone who tells you that they got a job promotion, or they got a new job, or a better job because they got the certification. Those are the those are the reasons that I keep doing it, even though it can sometimes be painful to do. Yeah. So now that we’ve talked about the test definition, and the credentialing and fitting all that and the next thing that we need to do is the job task analysis. So what is the job task analysis? And what’s the purpose of the job task analysis? So let’s let’s look at that first. The purpose of the job task analysis is to identify everything that your test taker needs to know or do to pass that exam. So you want to start by breaking down what it is that they have to do in your product, what is it that they have to know to use your product? Hmm. So once we come up with this full list of everything they need to know, where do how do we figure out what they need to actually be tested on? That’s the purpose of the job task analysis. Now, there’s two ways to do this. The traditional way, which will take probably six to eight weeks to do is to do a job task analysis survey which we tend to just call the job task analysis. Technically, that’s the survey, the job task analysis is really listing out the tasks to do that survey, that survey can give you amazing data. When you take that survey, and you start out with applicability. And you list different things who needs to do it, you might be able to get multiple test structures from one job task analysis in multiple roles. Wait, wait, okay,

Dave Derington  30:30 
scratch record, hold on a second. So you can that was painful. And we’re not that painful, but it was painful. You had a lot of time and critical discussions with people. But you’re saying that instead of like me, I got one out of this. We could possibly get multiple. And that’s, I want to learn, I really want to learn more. And I think the audience does, too.

Debbie Smith  30:50 
So let’s look at something like for example, with spreadsheets right now, what are some of those roles that I’m considering? I’m not sure I’m not committing to sending these out their viewers nicely, either. We’re still in the process of honing these in. But we’re we’re talking about a project management, role certification, a program management, role certification, and a process management, role certification, right. They are three distinct functions that people use Smartsheet for all the time. There’s a lot of overlap, though. There’s a lot of things that, you know, a program manager needs that a project manager needs and a process manager needs. So when I do my job task analysis, if I take the tasks from all three of them, put them into one list, divide them by domains. And then instead of asking the three questions, how important is it? How hard is it to do? And how often do you do it, I add one more dimension. And that is, and that’s who needs to do it. And so does the Process Manager need to do it program project, if you do it as multi select, you can let them choose one or all or any combination of them. So now, instead of having to do three separate job task analysis, I have just cut this time down. I’ve saved if I’ve got done three in one. I’ve actually saved 12 weeks worth of time. That’s substantial.

Dave Derington  32:24 
Three months. Yeah. That’s an understatement. But okay, so let me process that. And we’re going into our next phase where earlier on I mentioned, we have three, you know, and I call them end users, you know, their core users, we have a prospector, somebody who’s a BDR, SDR. You know, they’re, they’re working on the product more than anybody else. That’s our 80%. Well, very high. And then we have a manager of those people, which there’s nuance differences, you have to understand all of that. But you need to go more now and think about, well, how can I coach? How can I mentor? So you need to know all the same stuff plus a little bit. And then we have, like an account executive that’s very similar. But the differentiation between an account executive who is a closer, right, what normally in the sales motion, a prospect or sales or business development rep is bringing in people you know, having a having a good interaction with somebody and getting them excited, gauging their interest. And then you say, I want to buy your product. Cool. Let’s go over to our closer clinics. Now, they’re in different kinds of motions, but they’re using our platform similarly. And then. So you’re basically saying it’s kind of like a view and I would think of in a database, so I could collect all this stuff. Then filter out on SDR, boom, here’s this job task analysis for that filter out on account exact. Here’s that and we’ve done it all in one. Yep. It might take a little bit more time. But we’re not.

Debbie Smith  33:57 
It really doesn’t take as much time because those those discussions that you’re having around that job task analysis, those questions are already coming up. Because I’m sure as you’re talking to people there, they’ll say, well, the SDR needs to know it. But the account executive does it. Yes. Because they’re already saying that to you. Yeah, that’s also your internal point of view. And here’s what’s fascinating from the way I do my job task analysis survey to is I break my survey down after it’s done for the, for the certification, I break it down for internally for us to look at, because I separated out by segment. So I separate out what are my customers saying? What is this? Especially What are they saying is hard to do and how often they do things. When we ask internal folks, are they saying the same thing so hard, and that our customers do it as often usually there’s some difference in what we think is hard and what they think is different. And then also our partners. So if I can break it down and show you who thinks what It’s difficult the data that I can give back to the UX team, and to product marketing and product management, all of a sudden, they are now bought into certification because we’re giving them data and we’re helping them it creates a really good cross functional team.

Dave Derington  35:17 
Oh, my gosh. And that I don’t think that’s usually brought out in this discussion, the discussion about certification, that so you’re we are serving other audiences, internally, as well as externally. That, you know, I’ve actually had my product teams start to ask us questions about, hey, we’ve got this great big new functionality, actually a separate product in the works. How will our certification for admins evolve to incorporate that? And I love this, you’re thinking ahead for some time, too. But it shows that that organizations bought in. Right, that that’s trust, that’s cross departmental trust, now you’re seeing the real value and a new value of customer education. Exactly. Oh, my gosh, okay. All right. So is anything else on on the the job has, I think you’d said something about us in content? Yeah. So

Debbie Smith  36:13 
there’s one, there’s a couple places that I start, when I start my job task analysis. One of the first places I go to get all of my lists of what needs to be done is I start with every company I’ve gone into has already had elearning created and I lt created. So I start with their lists of objectives for their training as a faster way to start to assemble lists instead of starting with a blank page. So that’s one place that I look. The other place that I look to start with the job task analysis, is if your company has something like a continuum at braze, we had a continuum where customers would move through different levels, and we could see the different usage, right. And as we were doing that there were different tasks that we knew that they completed for each of those. So we incorporated those into our job task analysis. What that did is it made me speed up the actual collection of the tasks, because I could go to a Smee with a list already created and have them add to it, or tell me something was worded wrong. And really refine it and then build on it, it’s much easier for me to start with a list of something, then to start from blank. So that’s, that’s one thing in the analysis phase. The other thing is that you don’t have to do a survey, it is a best practice to do it. But if you have to get get it out in two months or three months, you might not have time to do the job task analysis survey. So you can take your job task analysis that you have all your tasks, and sit down with three to five speeds in a meeting and take a couple of hours and go through it with them. And have them tell you which ones are the most important and how to think about waiting the exam. I’ve done that to get an exam out first. And then as part of recertification run the job task analysis to verify what I’ve done with my species in the next level. So there’s some a little bit of risk with it, but there’s not much of a risk. So it’s something to consider,

Dave Derington  38:18 
especially for my more things that you already have a good context on. Exactly like the roles that I just talked about. We have tremendous amount of information. We have tons of thought leadership, we have huge number of best practices. Frankly, I think we got all the dirt, we need to tell us a really good exam,

Debbie Smith  38:35 
right? And have them tell you, you know, break it down into your domains, which, which is really, you know, is it worth spending 25% of an exam here? Should we be spending 50%? Should we be spending 10%? Some of that’s going to depend on how many tasks are under it, and how important those tasks are. But you just needs to tell you that at first, if you want to get it through first faster.

Dave Derington  38:59 
Yeah. And there’s also other things too, I think you mentioned like using definitely content we already have. But you know, I have surveys on the content that we’ve been using for a year plus. And that tells me other information too, because some of the features and workflows and ideas are received the less well, the lower survey ratings, right then other ones. I know like, Oh, well feature x or product x is a little bit harder, and I’m gonna have to do some more, but I have information on what they’re getting stuck on. Oh, so basically what you’re saying is get the lay of the land. See what you’ve got. Bring that all in first, and you can shortcut. Maybe months of time. Exactly. Good lord. Oh, you got saved 12 weeks, we saved a couple months. And this is like So on average, I think this is an important moment to step back and say, on an on average, a certification program. Let’s say vanilla, you’re not shortcutting you’re just doing it straight best practices, Debbie. What might that look like from a time perspective?

Debbie Smith  40:01 
So looking at it from a time perspective, if you use a psychometrician, to do all the typical, you’re usually talking seven to nine months.

Dave Derington  40:10 
Yeah, that’s about right for our experience.

Debbie Smith  40:13 
So, it especially in this remote world, right, if you’re talking about pre COVID days, when we all got in a room to do it together, you, if you have your job task analysis done, from that point to, to get into beta, you can actually get to that in six weeks. If you have all the people in can pull them all on site, and do two week work, do a two week workshop with them, you can really shortcut it. And that the problem with that is I’ve never worked at a company that was willing to give me 10 or 15, or 20 people that I needed for two solid weeks, like no one will do that. So that’s why the process takes longer, the process takes longer, because we can’t get all of those people in the room at the same time. You look at some of the bigger companies. Joe konata, from connexus. His normal development time is six weeks, because that’s the he pulls everyone into the run, and does it that way. So he can get them out fast because he’s able to get the resources he needs.

Dave Derington  41:18 
That’s amazing. And he’s lucky, because a lot of his lucky, particularly with COVID. Now it’s all this. We went into this. As we were starting to spin up, we had the doubts like Oh, is this going to end before? We have to start all of our meetings? And cut that out? Is this going to start before we had all of our meetings? And No, it didn’t. And it was tough, like really tough. I know where How are you on time? I don’t have anything after this. So we can run over? Okay, good. Because I didn’t want to I tend to get tired to speed things up. And then it’s not as fun. Okay. Yeah. Okay, let’s shift into another gear. So we’ve already talked about the job task analysis, we’ve talked about other key steps in doing this, let’s, let’s get to the meat of it. So we’re doing item writing.

Debbie Smith  42:11 
So from the job task analysis, that’s going to tell you how many questions you need for each objective. Right? So that’s going to give you your exam blueprint. Right? Right. So once you have your exam blueprint, then we go into the item writing. Item, right, writing is hard.

Dave Derington  42:28 
Tell me about? Oh, my gosh,

Debbie Smith  42:30 
people don’t understand how hard it is. Why is it hard? It’s hard, because if you’re writing multiple choice questions, you have to have, okay, research analysis, you can do three questions. I’m old school, I still do. I’m sorry. research says you have to have three answers. I’m old school, I still do four answers. It just feels right to me. And I never want to alternate between three and four answers for questions. If you alternate the number of questions you have. And some questions have three and some questions have four, I will guarantee you that you’re going to have test takers, who will email you or will put comments in the questions saying that the question did not load completely. Right. So for consistency, and for end user experience, I think it’s just easier to always do four. So that’s, that’s why I do four. But when you’re writing four, you need one absolutely correct answer. And you need three plausible distractors. What does that mean? They have to be real. They can’t sound fake, they can’t just be all of the above and none of the above which we don’t ever use. And they can’t be Mickey Mouse answers. They can’t be answers that people just know are wrong, right? Or it reduces the validity of your exam. So your questions have to be written really well. Your distractors have to be the same length, they have to be the same format. So if they, if one starts with a noun, they all have to start with a noun. They can’t give away the answer. And your distractors. And your questions can’t be teaching. So you don’t need to give the information to teach about something else as you’re doing it. And a lot of times when we write questions, we don’t even realize we’re doing that and we try to incorporate that type of thing. So there’s a lot to write in questions.

Dave Derington  44:30 
It’s, I want to pause on this a little bit intentionally because Okay, some of my experiences have been I’ve been a university professor, the in that I had to create tests, I had to create exams, and then in that, in that experience, this this is something that I don’t know if a lot of people like bring thought to. A lot of universities now have a lot of adjuncts right, I was an adjunct. Those universities will train you how to write exams. They don’t train you how to be a professor. Those are the things like I always say, You’re either a teacher or not. It’s just kind of all. And most of us have gone through the educational system. So we’ve just, like, absorbed the ways of doing things. canonically. Yeah. And that means when we go to write questions to take tests, you have teachers that made really good tests that you go, Oh, this test was hard. That person likely knew how to write an exam, they probably could have done a good certification exam. But most of us don’t like we Is it right for eLearning? the kinds of questions that I would write intentionally for eLearning would be, I want to check, I just kind of want to move you through to think a little bit I want I don’t want to make it hard. I want you to process and retain and challenge. But the the certification, again, we’re coming back to the basics, the goal is that I want you to be if I’m teaching you something, to be able to bring that back and say, you can prove that you understand this. Without Google, you know, everybody’s Google genius these days. Without Google without anything, you’re completely stripped of all of your notes. It’s just you and the computer. And that’s why I can’t express enough to anybody listening here to really think about this step, learn, practice, get your team up to speed, but set the stage that this is something that I actually did this Debbie on my team, I said, hey, guess what, like I invited all our enablement team, who we know they’re in different teams at our company, said, You know what, I’m really gonna invite you to join into this, I already know you have subject matter matter expert, but this is going to teach you how to write really good questions. And that’s going to go back and seed into the work you do. And they were like, Oh, my God, you’re right. I’m writing better questions. There’s a huge value in ROI in that day, you don’t even think about?

Debbie Smith  46:54 
there absolutely is. And if you did launch a credentialing program, one of the things that you may want to consider doing is having subject matter expert badges where you actually award badges to people who learn how to write questions, too, so that they have that skill set documented. And you know, who you can go back to, so that’s easier. And in elearning, we often write true false questions, right? Because we just want people to stop and think in certification, we can not use true false questions. true false questions, don’t tell us anything, because it’s a 5050. Guess. So 50% of people are going to get the question, right, just by the fact that it’s on the paper, right? And if the answer is false, you have no idea what you’re testing is true. You’re testing the objective. But if you’re giving an answer that the answer is going to be false, you don’t know what you’re actually testing. And you don’t want all your multiple, all your true false to be true, because that would get caught on very quickly. So we really need to stick with multiple choice, or multiple selects, I will tell you, our customers all hates multiple select. Hands down, I’m gonna tell you right now, they all hate it. That being said, we tend to use it and it’s easier to write, it’s easier for us. I had the unfortunate experience of having to go back through and rewrite every one of my multiple select questions into multiple choice, because we had so many complaints when I was at quickbase from our customers. So I took all of the multiple select out of it, it made the customers happy. If you do multiple select, and you have three right answers, then you have to add it to distractors, you have to add two wrong answers. So then you have to have five questions, five answers, you always have to have at least two to have a statistically significant question.

Dave Derington  48:49 
Okay, so let me repeat it back like, Well, technically, we’ve gone over the key tips, best practices like multiple select, I know people complain about that. So you would advocate for kind of a balance, or use them sparingly. Or a step into it if your audience like everybody’s audience that you’re trying to serve is different. Yep. So some may not be opposed, like, but it would depend on the kind of exam you’re doing. For example, if I was doing a sales development rep exam, I want those individuals to feel supported around only certain kinds of things like I’m really working to drill in best practices around our product. It would probably be lighter weight, but an admin, when they’re on the line in the heat of the moment, and they have a data issue or something fundamental. The job you’re doing the job we’re doing in certification is work. Who was that I was talking to? I may have been trash. You could think about your exam a serving tea. Are you serving a doctor? I mean, is this person gonna be operating on you? Is it are you serving someone who’s flying a plane, you know, and you want her to be a spot on and in In the heat of the moment, be able to make the right decision that’s informed. So that’s the call to action for us. We’re wanting to challenge people, we can do it in different degrees. Let me ask you another thing, though. Yeah, I think we’ve got some other questions around, like, tips on ratings and comments and things that How long did the review cycle goes? For these questions?

Debbie Smith  50:28 
Great question. So part of the item writing isn’t just writing the the items, there is a few things that have to happen, you have to you can call it either a technical review, some people call it a technical review, some people call it an alpha review. And really, what that is, is it’s asking you to rate each question on a scale of one to five, how clear and concise it is. Does it measure the test objective? Is there only one correct answer? And how clear how technically accurate? It is? Is it actually correct? Believe it or not, you are going to have subject matter experts write questions that are wrong. answers will just be wrong. And

Dave Derington  51:18 
that I’ve experienced it.

Debbie Smith  51:20 
Exactly. You don’t think it’s gonna happen, but it does. So with that technical review, you have to make sure that it’s technically accurate, clear, and concise, plausible, distractors, and tests the objective. For about 150 questions, it takes three hours for each person to review that.

Dave Derington  51:38 
That’s all the commitment.

Debbie Smith  51:40 
It’s a commitment, and you really want to have at least five people review the questions.

Dave Derington  51:45 
So they’re ours forever. This is why you were talking about before, you know, 500 hours for the person running the show. 500 hours spread, it

Debbie Smith  51:53 
cost the sneeze. Ouch. And for context, my, my five where that comes from, is that comes right from, from Jacob Nielsen, the usability expert, who did research in the late 90s, and showed that in order when you’re reviewing anything, it doesn’t matter what it is, it takes five experts to review it, everything to find 95% of the mistakes.

Okay, that’s, that is my rule of thumb. And that’s exactly where that comes from.

Dave Derington  52:25 
Five experts 90 95%. That’s, that’s excellent. Um, well, let me let me flip back a little bit, because you had just said something about pushback. So what I’m thinking now is, as we’re talking about resource time, we can go back and have that palpable sense of Oh, my God, you’ve got sneeze that you’re asking your leadership for time from, or customers might be involved partners might be involved. point is that there’s dollars. And there’s real impact to the bottom line that’s caused by our engagement with these teams. Right? So part of that, like, I’m not really going to talk about the managerial pushback, because you shouldn’t be getting any from your leadership. But you do have, you do need to think about efficiency. And that’s something that I was called out on. And I’m transparent about it. It was a hard conversation to have. And I go, there’s no other option. Right? We don’t have because we’re moving really fast. We didn’t have the resources we need. We had to do things the way we did the first time. I’m, I’m wiser. Now. I’m been beat up a little bit by the experience that we’re trying to help all of you get through here. But let’s talk about then. How do we reward and incense people through this, you just said badges. I love that. Like, you could have a technical review badge and item writing review badge. That’s really super cool. Because when it gives you that badge that goes on LinkedIn, that’s part of your portfolio. These are things that we’re learning as you go, maybe even your enablement team might even get a wise to this too. And you know, I’ll call it my friend Sean, who’s our enablement team manager for professional services. Hey, Shawn, we can do a badge for our team. Dude enablement flow course one, that will be really cool. Exactly. But what about there’s two other things? You mentioned stuff around partners, what we could do to incent them. And also, like those who are doing commission commission based resources, like I’m going to be engaging with sales development Rep. They’re going to be called, they’re going to be losing some of their their time to close deals. Right. So what are some other things that you would use to bait and send?

Debbie Smith  54:30 
So one, one thing that I did that went over really well? One of the first exams I did, I got everyone in a room. We could be in a room back then. But you could you could do this virtually still. But I did a pizza beer lunch. So I bought a bunch of pizza had the beer in there. And then I had $50 gift cards to give to the person who wrote the most usable questions for each. They had And I actually stood at the front of the room with a wheel, and I had all my domains on it. And then I had the list of the objectives. So it’s been it is the domain, here’s the objectives. And okay, we’re gonna write questions on that for 15 minutes. And then we just spun through the wheel to get through questions. And we did that a couple of days in a row. And then for people who weren’t able to attend the pizza and bear lunch, then I did a week long, open writing that you could write on into the platform, write the questions, and whoever wrote the most that way also got it. And, you know, people who are in the roles of like solutions, architects and sales, they tend to get intense support, they get into the gaming piece of it. So I had two great subject matter experts, who were keeping track of how many questions the other one wrote, and one upping each other that they were writing questions.

Dave Derington  55:55 
A certification.

Debbie Smith  55:59 
It was funny, it was great, but I got that’s how I got the right. So that was great. And I think I’ve done it as partners. So when we bring partners in to write questions, one, there’s this thing called an administrative grants. What’s an administrative grant? That is when you have subject matter experts writing the questions, if they’ve written the questions, they obviously know the material. So when the exam comes out, we automatically award them that that certification because they know it, so why will they do it? One, they’re the first ones to get it, they get to post it before the end, others partners love that customers love that and employees like that, too. That’s one way to motivate them. It depends on your partner’s structure. But at braze, we had offered the partners who chose to help us a chance to be preferred partners so that they would get resource before some other partners. There’s all different types of incentives that you can use, you need to work with your partner team to see what incentives will work with your partners. Just make sure that whoever’s writing your questions, if you’re writing your questions in English, are fluent in the language. Right? Right. If they’re fluent in another language, and you’re going to translate to that language, then that’s okay, too. Because if you’re going to write the Spanish, then you may want Spanish people, you know, Spanish speakers writing your questions, you’ll definitely want them to translate. But you need to really be careful of your language.

Dave Derington  57:29 
These are this is gold. And these are things that I, we’ve did a little bit of this, and we did have some like pizza beer kind of, well, I kind of pushed back on the beer because it was so technical. We did later. We did that after. But I love this kind of stuff. Because the creative thinking involved in you. This is way beyond project management. Now this is motivating people and getting them to share your vision. And actually, it’s kind of exciting. Because as a team as a company, I like to say the certification program is a product itself. Right? It is in product teams, if you’re listening. This is for you, in a way, like we are, this is one of the unsung hero moments of customer education. I think, you know, it goes back to some experiences I’ve had where we did some analysis, we did some big project, we share this information that we’ve learned about our customers with executive suite, and they go product, are you are you getting this? Like? Are you seeing this because you don’t think of education as being that although we’re we’re involved in a different kind of emotion, we’re involved in helping people retain what they’ve learned about how the product works. And we’re also in customer education challenged by a shifting sands of time that the product changes so quickly.

Debbie Smith  58:51 
Exactly. And I had a customer. When I do question what I do by exam, when I launch it, I leave it so that customers, anyone taking the exam can give me feedback on the question in the middle of the exam, right? So they could just post it right there. That way I get that. I had a couple of questions that customers had commented on that, while this is the coolest use case, I’d never thought of doing that. Now I’m gonna go do that. And it said they were gonna, they were going up a level in their product in order to do it. Right. So product adoption is definitely affected by certification. And, you know, scenario based questions can really help drive that. Because the other piece of review that we didn’t talk about yet is the modified angoff review. The modified angoff review is where we cut set our cut score before we go into beta testing. And so that consists of one question, and it’s out of 100 minimally qualified candidates, how many people would get this question correct? The answer has to be above 25 because statistically speaking, 25 would get it right, just a random, and it should be below 90. So, really between 20 and 90 is what we’re looking for, for scores. And that’s going to help us get our beta test. But that’s one more review. If you’re in a real time crunch, one thing you could consider doing is merging your technical review and your angoff review. Oh, asking five questions at the technical review phase. The drawback to that is you might have to do more reviews than you planned, because if they didn’t make it out of the technical review, they’re gonna have to be re redone through the app off again anyways. But it’s still there depends on how how close your questions are to being good. And as you’re doing an edit of your questions before you send them to technical review. Most of us, if you, if you’re a user of your software, you can start to see if they’re going to be if they’ll work or not. And you could start to see where you think there’s gonna be red flags.

Dave Derington  1:01:07 
Yeah, that that’s going to be a user of your own software, and you’re building programs for sure. It makes it a lot easier. So let’s talk more about where we’re up there. technical review, we’re now in the beta testing mode. So we’ll talk about a couple different things to get beta testing. And then we want to talk a little bit about some back end stuff. Better testing, I think is where things should have been straightforward for us, but ended up kind of getting difficult. And one of the things I know we work really hard to do was to stay with the best practice the industry metric of like getting 100 people involved. And we got high 60s, we actually then we did some more stuff. So we got there. But it took a long time. And I wanted to challenge that a little bit. So how can we bring the time involved in beta testing down?

Debbie Smith  1:02:00 
Okay, well, I don’t go for like if I got 100 it site quit. Yeah, I’d like but I don’t plan on getting on it, I’m looking for 30. There’s a statistical number around 30, that if you put it out on the, on the normal curve, that 30 is really what you need. And so I, I use the N of 30 all the time, I go back to my old statistics books, when I had to have an N of 30. And, and that’s what I use. So as long as I get 30, great. I like to keep my beta open for a month. I don’t want it to go longer than that. And so I should be able to get 30. If I get 100 in the month to do it. Great. I always have 100 sign up for it. But only half of those who sign up for it actually do it is what I fight.

Dave Derington  1:02:53 
Yeah, that’s usually that. But that’s the thing that I wanted to bring out in this discussion. And this point is that in customer education, again, we’re framing this up as a practice, this is a discipline that’s not been around, in name and in, you know, how we do it for very long, and we’re still trying to evolve into that. But this is one of those moments, like all of the things that you’re talking about Debbie with certification are kind of adaptive, right? Because we work in an environment and this is why it’s so Okay, let’s let’s compare contrast and talk instructional designer. You know, formal instructional designer coming in from the industry with a background, maybe even degree in that, who comes into customer education. Those individuals typically have some adaptations to that they have some they have to change a little bit, because now we’re moving faster, we’re not going to do things the same way. We need to think about the the North Star, you know, my goal is, is go to market, how can it get to market as soon as possible with the best possible product with a little rough edges? And that’s okay. And that’s where I think this is the everything you’re talking about is how we do cursors certification and how SAS does certification. And it’s different from formal stuff. So it’s okay to change in Frankly, I think the results that you’re getting and others are getting are pretty amazing. It’s it’s we’re having good results for certification programs.

Debbie Smith  1:04:22 
Yeah, wait. I want to collect the numbers somewhere that we can actually all report on them. But right now from talking to everyone I know from talking to all my friends and certification, every one of them is telling me that they see a decrease in churn that they see increase in product adoption, they see increase in active users. That’s what they measure. They see increase in annual contract value arr MRR, whichever term you want to use, there’s we’re definitely seeing all of those impacts. So we want to get it out there quickly and iterate. So there’s things that we need to constantly be doing At least on a quarterly basis, we want to be reviewing the questions. I mean, people will tell you, you should do it every month. That’s just not real. For most companies, it’s just not enough time of the day. So it’s going to happen, at least quarterly that you’re reviewing your questions for a couple of things, one, that your product hasn’t changed, our products change so quickly, that you have to make sure that your questions are still right that, that the button is still where you said it wasn’t still called what you said it was called, because it had happens. You also want to be looking at your P values and your correlation coefficients. And many of the platforms will give you those numbers, either right beside the question, or in a separate report that you could just pull down the report. And you can see all of those p value correlation coefficients. And there’s some standards around where everything should fall on the P values in total correlation coefficients, so that we can look at those ourselves and say, Okay, yeah, this question is not performing very well. The P value is a one, a p value of a one means everyone’s getting it right. If everyone is getting a question, right, the question is not doing you any good. It shouldn’t be on your test. Yeah. If it’s zero, the questions really bad, and no one’s getting it right. correlation coefficient, if your correlation coefficient is really low, that means that the people who are getting the question right, but not passing the exam. Now, you know, you’ve got a really interesting question. That’s definitely not not doing you any good. So you really need to look at that. That question. So those numbers that you need to look at in the systems all give us that today? Yeah.

Dave Derington  1:06:43 
This is great. Okay, let’s get to the last subject matter. And then we can get out the door. But I hate to rush this. I think this is this is such a, in, in the course of you talking, one of the thing that I thought about is there have been a number of really good talks about customer about certification, we’ve done, I think, the least two, we were on a panel together, I think, a year or two ago at skill jar. And that was great. There’s a lot of references out there. And I think you’re even developing some more resources too, aren’t you? I am

Debbie Smith  1:07:18  
Jamie, Jamie Cantor and I are working on a book together on how to do SAS certifications. I just published a book chapter on learner credential outcomes, which is what’s the benefit of certification and credentials for learners? So there, there’s more and more that we’re putting out all the time on it, because it’s just it’s such a need. There’s people asking questions about this every day.

Dave Derington  1:07:45 
I agree. Okay, with that in mind, let’s go into the final, you know, I don’t know, we’ll call it a lightning round. But let’s talk about the back end of this. Let’s talk about some other things that may help us foster a good relationship with our, like the people that are earning this certificate in our group. What about, here’s the subject that I want to talk about at the end? PDS? You brought that up? I didn’t even think about this. What What is the PDU? Why is this something that you can use to really help make for an amazing certification program?

Debbie Smith  1:08:24 
So there’s, there’s two ways of looking at your certification, you can either put a date on your certification. And so it has a year date on it, Uh, huh. can not put a date on it and just say, to have it automatically expire after so many years. You can do it by your product date, like Microsoft does. So Microsoft, for example, I am Microsoft Office 10 certified. So I was never going back to take an exam again, because that’s how I have to redo it at Microsoft does, I have to go back and study for it and take that exam again. And that’s not happening. So what I’ve started doing is following the project management Institute’s method, which is PD use, and what a PDU is, is it’s simply one hour of training. And with Project Management Institute, over three years, they have you get 60 hours of PD use. Normally what I’ve been doing with us is I’ve been doing 10 hours of PD is to renew your certification to automatically get the next year certification. The reason I think about doing it that way, is because I really want to make sure people stay engaged with the platform that they’re still using it and that if they’re still doing the training that I know they’re still using it, they’re staying current with what’s happening. If they’re not staying current with it. Then their certification can expire. But as we use PD use It does a couple things, it boosts our training in boosting engagement. It’s the method I actually surveyed my customers to find out what which method they wanted to renew, I gave them the choice of annual exams, quarterly exams, or PD is hands down. Every single customer said PD is few of the managers said they wanted their people to take the exams again, but they didn’t want to take the exams again. So that was an interesting little tidbit. Beyond the training, I also give PD use for engaging at our conferences, because if you go to our conference at conferences, a three day conference, in training all day, most user conferences are really training conferences, even though we don’t call them

Dave Derington  1:10:45 
thank you for saying that. I believe that as well. When you’re asked to do training, I’m like, why are we going to be more true, I could do that online, go and learn other stuff.

Debbie Smith  1:10:53 
Yep. User conferences are training, they’re all about training. And the same thing, we do regional meetups, you know, we do a social hour, but there’s always some component of training. So we get PD use for that. Marketing loves that it helps drive attendance. But customers love it, because it’s an easy way for them to stay engaged and renew their certification without having to retest. So that’s really good. The other thing is, if you use PD use is a standard measure, the PD use could be translated to see us which is continuing ed credits, which is 10 hours for everyone. Okay. Si p, which is what the accountants use is one hour of training for each CP. So where we have already aligned it with the PD use, it’s really easy for them to print out their training record and give it to their record body and get credit for. So at smart sheets, we’ve actually taken the step to become an authorized provider at PMI, because we do so much with Project PMI. So we’re authorized through them, but even our courses that aren’t authorized for them. If you are a PMP, you can print out your your record and enter it in when you go to renew your certification. So what are we doing for our customers, we’re getting them to benefits for their PD use, they’re renewing with us and they should renew with their other system too. So customers like it for that it’s really helpful for them.

Dave Derington  1:12:22 
Yeah, and you know what, this, this really frames of certification, your certification program as community tool, it’s something that’s kind of at the core of helping, I guess this is what everybody wants to be that SaaS companies are again, they’ve we, how do I even I was thinking about this the other day, and why I like software, and why I like working in SAS so much, particularly for startups that are fast growing, it’s that somebody comes up with a really great idea. And it’s disruptive and disruptive in a good way. I’ve, I’ve got this idea to make this platform like an outlet. Let’s use outreach, because I’m working there now. It’s pretty fabulous to see this concept of a new term of sales engagement. And what does that even mean? So we tend to go out and go, Oh, here’s this new thing. And here’s the software I built to support that new thing. Nobody knows anything about that. Right? And that makes certification even more important for our field? Because it’s like, Okay, well, nobody knows anything. The first people it’s like, mining for gold, the Gold Rush that’s happened, you know, in the West a couple of times. And it was like, like, I see this all the time partners, our partners are really some of those people that are crafty, and smart and talented. And they’re like, I want to be there. First, I have the subject matter expertise. And they are like, really on these certifications, because they want to show it, they want to prove it. Right. And a lot of our customers are like that too. Because imagine you’re you’re a salesperson, you’re early in your career, how do you differentiate yourself from anybody else? How well you could say you did stuff. And you could say you used outrage. But if I come to you and say, here’s the badge, proof, you can be very bit, just take the test. It’s not that expensive, but it’s can take us on board. So it’s it’s a new time. It’s a new era. This is really exciting. But it’s different, palpably different from other kinds of disciplines. And I really love seeing what you’re doing with that.

Debbie Smith  1:14:23 
With all the changes that have happened to society over the last year. I mean, we can’t ignore the last year right what has happened the last year stuck in a house for a year. People aren’t finishing college. Right? And so not only are people not finishing college, but many SaaS companies have re removed the requirement for a bachelor’s degree. So how do we know people know how to do things certification is going to fill in that gap. And those certifications are really going to help people be able to get jobs especially if they haven’t gone to college. So this is whole new area that’s about to explode. And, like, we really need to start thinking about this because as we remove education barriers, we need still need something to help upskill people so that they have the skills to come in to do the jobs that we need.

Dave Derington  1:15:18 
Yeah, and it’s not just like going on any online learning platform. And just taking some courses. This is real deep learning. This is real, meaningful learning, I kind of hate the MOOC world where you can go out, go and learn a little bit of this, a little bit of that, it’s like going to a smorgasbord and just, yeah, I had all the things and I got a stomachache. Instead, I’m gonna go have that nice, I’m going to go to the restaurant with my partner, we’re going to have an amazing meal, we’re going to pay a lot of money for it. And we’re going to come out of there pretty darn happy about it. And I’ve had the experience and then have something to say about it. And it’s it’s meaningful, it’s significant. It’s not fast food. It’s really intentionality. I love that analogy. That’s a great analogy. It’s cool. Sometimes I get lucky. All right, Debbie, I think we’ve gone really run a little long, but I’m glad you were, you’re, you’re able to spend this time with me. And again, I’m going to thank you for being so available. That and this is number one, thank you for all the work you’re doing in the community of people that we’ve talked about on this, this podcast, if if you’re out there, you want to talk about subjects like this as well, let us know because we’re we’re here to elevate you and get the word out particularly like I look at you as my go to for a certification. And I just thank my lucky stars that I can reach out to you and say like I’ve had hard moments, I’ve had really hard moments. And we’ve had this conversation a lighter context before. But this is sharing with the world. So thanks for that. Okay, let’s wrap up. And, again, this is kind of a 200 level certification, we’ve talked about this before you can look at the other episodes we had, we’ll give links. And Debbie will call to you to like put some more links to your works as well and really want to know about your upcoming book, when it comes out, we will be we’ve got your back to help promote. So in retrospect, I think we’ve proven our hypothesis pretty clearly. And the call to action that I can give to you is really sit down, okay, your manager, your leader, maybe you’re going to look at interviewing at a job. What’s going to help you stand out, is thinking about what Debbie has said here in our responses and our strategies and our best practices. For SAS, this is hard, this is new, this is different, take some time to really sit back, think about what you’re going what you’re doing. If you’re going to create a program, right? I tend to think many leaders that, you know, I was talking with some of my leadership and they think about this is a market differentiator. It is absolutely that this is something that is a must have for a lot of companies. But if you do this without thinking about the potential downsides, you’re really setting yourself up for potential problems down the road. And also, let’s go back to what we were talking about the term certification, I have to say this again, as a PSA, if you’re going to use that term, have you had anybody else using that term, and they’re not using it in context of what Debbie you have set out? Like your your our canonical reference now for certification in sets, that this is, it’s something significant, you need to go through a process, you need to understand that it’s going to cost money, it’s going to cost you resource time. And if any of your leadership is resistant to any of that. Sit down, take a moment, share this podcast, get them to Debbie, if you have time, get them to me, I’m happy to talk I’m happy to talk to you. But if you’re just making tests, no property, no controls, it’s not certification. Anything else to add to that? Before we close out,

Debbie Smith  1:19:01 
totally agree spot on. And don’t hesitate to connect with me on LinkedIn and reach out to me. I’m happy to talk to people about this. My goal in talking about this is to help to raise all of us in customer education. And so we all have a standard and can explain this to people because it’s it’s hard. I’ve struggled through it and I want to make it easier for others.

Dave Derington  1:19:24 
Yeah, it is really hard. It’s really hard with people that have not had any background to it or exposure. Exactly. Thank you so much for having me today. Dave. You’re quite welcome. And now 10 this out if you want to learn more. We have of course our podcast website at customer dot education. You can find their show notes other material and including transcripts of the entire talk that we had. I like to take some extra time to bring out bullets and things that have been said and and bring Particular attention that so go check that out. On Twitter, I’m at Dave Derington. At Adam Sq is Adams Are you on the Twitter’s twitterverse News. I don’t, I don’t do Twitter. I do like, rarely, but I’m out there.

Debbie Smith  1:20:06 
Yeah, I actually I do have a Twitter, my Twitter is days thoughts. Oh, I like the creative ones. That’s my Twitter that I created. Like the year Twitter came out, like I said, but don’t actually use it a whole lot. But I do get the notifications if someone does this to me there. But I’m on LinkedIn all the time. And it’s Debbie Carrie Smith, on LinkedIn. And then I’m also very active in sigma and in the customer education Slack channel.

Dave Derington  1:20:38 
Fabulous. And we are where there’s a lot of us on there, if you need to hear more about that just reach out. So a special thanks to Alan Kota Allen again, your music for our podcast is amazing. We love it. And if this helped you out, you can help us out by subscribing and Apple podcasts, overcast, whatever your pod catcher of choice, one of the things I really encourage you to do is get out there. If this helps you if this touched you if this made your job easier. leave us a review. We’re now streaming these on YouTube as well. Subscribe, and like we’re gonna have a lot more content coming in again, this conversation we’ve had with Debbie here, debit and thank you so much. This is the core of what we want to do at sea lab. We’re a laboratory we’re thinking, we’re creating, we’re experimenting, we’re learning together. I don’t have all the answers all the time. That’s why I’m interviewing Debbie and others. And so if you want to reach out and and talk about and go deep on a subject area in this space, reach out to us. We want to hear from you. So to our audience. Thanks for joining us. Go out, educate, experiment and find your people. Thanks, everybody.