Dave Derington 00:01
Welcome to CELab the customer education laboratory where we explore how to build customer education programs, experiment with new approaches, and exterminate the myths and bad advice that stopped growth deadness track. I am Dave Derington. And today we are joined by Christy Hollingshead, you’ll go ahead and introduce yourself, I see that your role may be actually changing a little bit.
Christy Hollingshead 01:06
Yeah, I am the senior director of customer education at Heap. As of today, that is our online programs, our instructor led learning. But recently, just as of yesterday, I’m excited to also be taking on our scaled programs, really, how do we market to our customers? How do we use other tactics besides courses and live training to educate our users. So really excited about that?
Dave Derington 01:33
I’m really excited about that. And that’s, I think that’s a really good edge to pursue here. Now, now you’ve been working on building a program. Now you’re kind of bubbling up a little bit more to look at the bigger picture and see how this all integrates together. So we’ve got a lot of good stuff to talk about. Let me frame this up Christy and to our audience. If, if you are out there thinking about oh my gosh, how do I go back? How do I report? How do I build dashboards? How do I see the North Star and and be able to guide myself to that, you know, it’s one thing for us to say that we lead with data, you know, I’ll put those in air quotes. It’s another thing entirely to get to that where you’re constructing something that you have in front of you, and you can share and, and really helps you out. So that’s what we’re going to talk about today. And as always, before we get into that I’ll always like to have a little bit of fun. This is our national day of and today is celebrate your unique talent day. I thought this was totally appropriate because we have Christy and her unique talent is leading with data. Cool. Anything else youwant to say before we get rolling Christy?
Christy Hollingshead 02:42
I’m really excited to chat about data and metrics. I know this is something that I have struggled with in most of my customers career up until now. And I think I’m starting to figure it out. I don’t have all the answers. But I found some success and excited to share what I found.
Dave Derington 03:00
That is fabulous. And and that’s important because in customer education where this is a new thing. It’s new because it’s emerged from customer success. And it’s also taken those of us who have gravitated because we love education from other fields, l&d you know, enablement training, and we’re all here now. And now we’re going well, we’re in a different universe, we’re in the SaaS world. And in SaaS, the most important thing is to be able to see the impact. We’re kind of like, scientist, right, we want to see where we’re going. And I could say, oh, I have more people coming into my program this month. And what happened? What did I do to make that change? What can I do to make that better? So how do we do that is the big question on my mind. And I know and you know, after we’ve built several programs, this is super hard. This is this is one of the hardest things ever. It’s not just, I’m not just building content, now I’m going how does that content actually work? So we also know that this isn’t your first priority, right? Yep. For a company, most companies are like, Okay, we’ll get to that I’ve been in that boat. So let’s go ahead and start unpacking this. And, you know, I’m gonna try for a hypothesis. This is a god awful hypothesis. But, you know, I want to say this, anybody, any customer education professional, can easily, maybe not so easily work with the data that they have about their program, and they can get to building reports and information that will help. So how’s that for a start?
Christy Hollingshead 04:26
I think that’s a great one. And I buy in!
Dave Derington 04:29
I buy in offices, too. All right. So I want to believe let’s go for it. So let’s look, you’ve talked to us a little bit about your role. Christy, what I like to do is give you the the forum here and tell us a little bit more about how you got to heap How did you step into this role? And what really gets you excited about doing what you do?
Christy Hollingshead 04:54
Yeah, I guess I’ll go back to the beginning. Where are they? I was in. I was in grad school, and I thought I wanted to be a professor, actually, I was getting my master’s in communications. And I think what I quickly realized in grad school was, you know, I realized that world was the publisher perish mentality. And I actually hated writing papers, it didn’t really like the research, the part of that job that I really liked was teaching and being in that classroom, and so with that realization, you know, decided to step out of the academic world, and head over to tech, where I landed at Microsoft, doing education, and actually a variety of different roles. They’re in sort of a rotation program that they had. And I was, you know, lucky enough that again, it became really quick, really obvious through those rotations that, like, education was the thing that I liked, so started doing training for them. And you know, after a couple years of that, transitioned into a startup, a company called Walk me on their education team there to lead their partner education programs, which was really exciting. And that was a huge transition, right, coming from a company like Microsoft that has every resource in the world, to joining a pretty small startup at this time, I think it was 75th. employee. Wow. I mean, they’re so it was really, hey, build this program from scratch, right, we don’t have a ton of resources, there isn’t everything that you need, you got to sort of build the plane while flying the plane. Yep, spent some time there, I think doing some pretty good work. And that actually landed me at my current company heap, where I was tapped to lead the entire customer education program.
Dave Derington 06:52
That’s exciting. And so I like I really, really liked your story. Because I feel many people I talked to myself included, have had that kind of journey where we discover there’s a part of us that loves to teach. And we follow this journey. I mean, I’ve been in big companies and small companies too. But I think that’s the fabric of what makes you, you know, unique, and makes us unique in this field. Because we, we synthesize these things, you’ve been at Microsoft, and it’s big. And I like how you said, all the resources in the world. So that means you’ve got support. And you came from education. And you said, you want to be a professor, which, you know, I know, I found my inner teacher, but it’s, it’s the joy of talking to somebody and helping guide them to learning and you see it, right, and it’s like, you see the light bulb go on. And then walk me is a really good story, too. Because that’s kind of an educated product. You know, it’s really meant to help pull people through and understand the platform. And now heap. So that’s where I think you really can unpack and talk to us about, like, what got you what got you really excited about starting to build this program out? And what’s unique about it? What, what’s unique about working with data?
Christy Hollingshead 08:07
Yeah, I think, you know, I alluded to this a little bit earlier been my entire career and education, I think I’ve had the questions like, basically every peer that I talked with, in customer education, everyone wants to understand the impact that you’re making your bosses are constantly pressuring you to prove ROI. But I think the challenge is that you don’t have the resources. And like you mentioned earlier, even if you have the data, and you have a data team, customer education is not on the top of their priority list. So there’s just this frustration of like, I want the data, I want to be able to answer these questions. But you know, feeling stuck, like you don’t have the resources to do so. Yeah. And, you know, I think of myself, I really try to have this growth mindset, right, not get stuck in the limitations, that sort of I feel like there are so thinking, how can I find a way around this? How can I get the data that I need? And how can I get insights from what I do have maybe I don’t have every piece of data that I want. And I don’t know how to run the SQL query. The more complex questions but what what can I do with what I have? And I think it really just started there. Yeah, walk me doing some of this work.
Dave Derington 09:27
Yeah. So like, what? That’s a great place to start. So what are the kinds of things that you found that you did have to begin with? Like, maybe you had an LMS? Maybe you didn’t? Maybe you had Wistia? Maybe you had Google survey? What are the things that so so here you are at walk me at heap and you go, alright, what have I got? Whatever? What am I gonna do with it?
Christy Hollingshead 09:50
Yeah, all of the above at different times, on all of those platforms, but yeah, I mean, I think of metrics or data. They’re sort of four different types of data. And a lot of the ones that you should start with are the ones you’re mostly going to have access to are what I call like engagement data. So this is exactly just how many people are taking your courses, how many people are showing up to your webinars, how many people are watching your Wistia, video? Whatever it is, right, you should have some type of information just on usage. And I think that is a really great place to start once you have an LMS most LMS is or at least going to provide you that basic information. Mm hmm. Oh, go ahead.
Dave Derington 10:36
I’m sorry. So but I wanted to wanted to key in on something. So the things that you were talking about first were like, how many? How many? You know, like, it seems like an aggregate at that first level where I don’t know exactly. Who know, here’s Jane. Here’s Judy. Here’s Joe. But I know, oh, I’ve got volume. Right. And then I think what you’re doing is saying now we’re getting into an LMS where it’s more specific, right?
Christy Hollingshead 11:00
Yeah, exactly. And you also don’t want to discount the qualitative data either. Oh, boy,
Dave Derington 11:07
what like why? What do you mean quality? So?
Christy Hollingshead 11:11
Yeah, so it is that satisfaction data, right. So it’s the Google Form, or the survey, or the thumbs up your webinar poll, whatever it is. So as much as you want people to take your courses in your classes, it’s also important that they are enjoying that, that experience for the learner. And I think, you know, a lot of people now I think, kind of discount that data. Well, this doesn’t actually tell me a whole lot of information, right. More recently, it’s being referred to as vanity metrics. Right. And I do agree with that, to a certain extent, you know, when it comes to the ROI questions, there isn’t a lot that you can prove by just having people take your content, right, whether you know who they are or not. But I still think there are some really good insights you can get like with that engagement data. So first and foremost, you know, you do need people to take your content, if they aren’t, you need to really understand well, what is it right? Am I not building for the right audience? Is this a marketing problem? What channels are people coming in from? Can I partner with my customer success team to be talking about these resources more? Right? And so of course, you want to start there? What can you do to drive this? And that data is going to tell you that first thing, right, so that is the first hurdle, I think when you’re starting any program is like getting people, you know, build it, and they don’t always come. So
Dave Derington 12:41
what is it? I know I’ve heard Donna Weber say that before, in some of you know, she talks a lot about onboarding. It’s if you build it, they won’t come out. And that’s, that’s kind of like a customer education mantra that we have to we have to actually think about that first. Because I want to tell just a really brief story about this. From my own perspective, a couple of times, I get so focused Christi, when I’m building on building in my entire world is, hey, what are we doing next? How are we doing this got problems? Like how do we launch and then you go, Oh, my God, I forgot about marketing, or in that seems, it seems like something that you shouldn’t forget. But it is fundamentally important to lead with because if you’re not talking with your marketing team, or you’re not thinking about social, you’ve missed one of the big hooks, and that’s a natural. It’s not even a mistake. It’s just a natural thing that happens. We have to think about it going it right.
Christy Hollingshead 13:36
Yep. Yeah. And I think one of the things, particularly having worked with product managers as my primary persona, the last eight years that I’ve really come to adopt and realize and you have to is that having a hypothesis when you get data, when you get an insight is really important. Right? And once you have that hypothesis, then it’s super important to start testing on that hypothesis. So if those you know that volume is low, well, why What do you think it is? What change can you make, to try to drive those numbers? Is your hypothesis correct? That it is a marketing problem? Is your hypothesis correct? That people aren’t taking your secondary courses? Because it’s too low, you know, on the page in your LMS? Is it not discoverable? There are so many things that you can just do with that basic, again, engagement level data, just how many people are engaging with content
Dave Derington 14:39
that I can’t tell you how much I appreciate the hypothesis approach because, you know, I was a trained I was trained as a scientist, I went to school for too long. But the one thing that I got out of that was that like thinking about where like we in SAS were natural scientist. I think this is a really weird kind of a bold, contentious statement I make, I think we’re scientists, because number one, we’re moving really fast. And number two, we just can’t let things go. We’ve got to say, Okay, I only have this data, I’ve got to make an experiment. The difference between science, as you know, SAS, science and science sciences, it’s super fast. So you make a hypothesis and test it really quick. And if it works, great, if it didn’t, no, you’re not going to do peer testing and stuff like that. But you are using the same fundamental thinking. And that’s why, you know, you’ve you’ve gone to school for like you were talking about becoming a professor, you know, you obviously have that academic flair to you. And I think we all need to adopt this. So maybe maybe we should dive into unpack this a little bit more Christie, that, okay, you’ve got data. We’ve got data, we’ve got some data. Now I’m sitting there, I’m scratching my head. I’ve got Excel up. I’ve got a data, but I don’t know, whatever tools. What are those first steps that I should be making? Like, we look at registrations, completions, attendance, you’ve already mentioned some of those. But what would be like a quick punch list of things that I should look at first?
Christy Hollingshead 16:06
Yeah, um, I think, you know, you nailed it right on the head registrations and completions are definitely things that you’re going to want to look at. I think, also, if you have the user level data, understanding sort of volume of courses, are people coming in and taking a single course? Are they taking multiple courses? But yeah, but you know, and you don’t want to over index on one or the other completions, I think is the big one that gets people is, you know, most of us, and I don’t have all the data. But I think most of us find, we get a lot more registrations than completions. And that tends to panic, sort of a lot of people, it’s like, why aren’t they completing the content? Over the years, I’ve started to care less and less about completions. You know, if they’re dropping off right away, I think that’s a very different problem. But if they’re getting 80%, through the content, and they’re not finishing sort of the last few things, oftentimes, what I find is like, you know, they want to start applying, and it’s, you know, it’s your quiz or your survey or your wrap up, right, and people have so, so little time, and I’ll get into why I care less about this if you have data further down the pipe. But I always just like to mention that because I just hear a lot of people panic on their, their low completion rates. And at least my opinion, is that they’re not as important as we’ve made them out to be.
Dave Derington 17:35
Thank you. Thank you for saying that. Because here’s a here’s a data point that I like to share. And I’ll be a little, you know, transparent and say, I’ve had leaders come to me and go, what’s our completion rate? Oh, that’s kind of low. And that’s, and then the conversation stops. And then I feel backed in a corner, like, Oh, my God, you know, my failing. But what you said there, Christie was so important, it’s that let’s, I can’t put a picture on because this is an audio. But let’s say you have a graph and you’re looking at the X, Y, and at the x axis over here, at the top, you have all the registrations. So let’s say 100 registrations, and then if you would look and I know, I’ll just use skill jar as an example, because skill jar had, if you go into the administrative back end, and you start looking through the data, there are these, I call them drop charts, Adam actually had some of these that he did in one of his platforms, I think you have something like this to where you can see engagement go over time within a series of modules. And then I would always see a progression downward. And what you’re saying to me is like, the way that we had them constructed at one point was there was a survey and a quiz, and people would just not do it. And I always would try to bait them by using badges. So you do your quizzes, you get a badge that help help the bring the completion rate up. But what you’re saying is important. If I got 80%, through the fact that I have a volume of people getting that far as the most important, and it’s not a binary, it’s not one zero, there’s Bactrim of engagement. That’s important.
Christy Hollingshead 19:05
Yeah. And that data, again, will tell you so much. So I you touched on this earlier, but so often, I think we get caught up in building the next new thing. And like what’s next, what’s next, especially when we’re building a program that we aren’t doing this evaluation in this measurement. But again, coming from working with product managers, I have really adopted the strategy of iterating constantly. And again, like most of us, you have your core curriculum, your core content, and that is where the volume and the bulk of your your learners are engaging with your platform. And if you aren’t spending time, I think constantly trying to improve that experience. You’re missing out on some really positive impacts that you can have by focusing and continuing to iterate on the content that is successful and is going well. Right
Dave Derington 19:58
so so Let’s, let’s maybe paint an example of this. I know one thing that I did is I had all of my content might come next to see my, uh, we had all of our content as a team in our LMS. And we had videos that were also hosted in the video platform. So you have two different data sources. And then what we do is every month or so is come back and look at because we had attached to Google Analytics, I know, I knew exactly how many people in my system in the LMS went and registered and completed. But I also know from Google, how many people came in, and how many unique pageviews there are. And then I can actually make a graph. This is, again, this is an idea that that came from Adam, that I adopted in us, so I could see from month to month, the like where a course would be, I could see if it was dropping, I could see actually where it was on a chart. And I actually made a quadrant, you know, like a, you know, a square. And if things were up into the right, that means they’re both a high number of views. And they and the word was discoverability. They were discoverable, and they were used. So if anything hit that high, I there was a lot of pageviews. And there are a lot of people actually registering and completing it. But I knew that course I don’t touch. It’s fine. I just do updates. But of course over here on the bottom left, meaning low pageviews. And nobody could find it. I knew Okay, so I got ever iterate. So now I could go back monthly and iterate, iterate, iterate, because that’s kind of a technique that you would you would adopt
Christy Hollingshead 21:29
100%? Yeah, we have a very similar visualization to that. We call it our engagement matrix. And again, this is, you know, I just keep coming back to all of the things that align with product, right? This is the same thing product managers do to understand what features in their product are getting utilized by users. And it’s very similar. And if you think of your program, like a product, what courses what content is successful, what’s not. And I actually think like, you know, the upper left quadrant is a really interesting one. So this tends to be, hey, a small number of users are doing this, but they’re getting really far. And oftentimes, I think those are the ones that are pretty easy to move to that upper right, if you can understand, you know, what, well, why is it the low number of users? Is it the discoverability? Here? Is it that it’s for a small subset of my users, and this is expected? Have I not made it clear, like when and how, you know, you should engage with this content. But that, you know, again, just that, like can tell you so much about your your content? And and what is sort of appealing to your learners and not?
Dave Derington 22:44
Yeah, I’m actually thinking of a really clear example that that I had experienced where I had seen. So in that upper left quadrant, where you know, your low number, I guess, a low number of visitors, it would be actually for me, I think it was the bottom away, I had it set up. So up in the upper right, you have really high page counts, but the lower page counts would be over here on the right in the bottom in my chat chart, and that correlated to admin users. So there were a smaller subset. So like, we had 1000s, and 1000s, or 10s, of 1000s of end users, and a small subset of admins. So that would make sense. And if I if I would paint that I would just do different colors, and show that or maybe separate that data entirely, because then you can move those pictures up when I’m talking to my executives. And they don’t question Which gets me to my next question, I think is, I’d like to really keep an eye before we get on to ROI getting on get on, like, how do I start working through all this stuff? The big question on my mind from what you were saying prior to our discussion, and just now is this relationship of with product management, and thinking about education as a product, which has taken me a long time to come to terms with because in fact, I think education is part of your product. It’s part of it. So what what can you tell me about that?
Christy Hollingshead 24:06
Yeah, I mean, I do think education is part of your your product experience. But when I’m talking about this tie in, I think it’s really about how you treat your own program within the org and how you run your function. You know, like I said, if you’re thinking through typical product process, right, you’re uncovering, Hey, what is the pain that my user who’s feeling? What am I going to build to help solve that pain? You’re gonna launch it, you’re gonna measure, you know, how successful are these metrics, but then a key part that product does that we’ve touched on a little bit already is iteration is part of that product development process, on a good product team, just like it should be on a good education. Right. I think having this mindset again, it’s just so critical that like, it’s not about just building and being done, people I think even when you have the data, they’ll they’ll get an insight or see something interesting. But even then they rarely sort of act on it to go back and like, Okay, well, what do I do with this now? But yeah, I mean, again, I think it’s about trying to find, well, where should I focus my efforts, right? Pareto Principle? Yeah. Which is, is really just where can I spend my time, that’s going to have the most amount of impact. And I think this is where Yeah, data becomes your best friend. So chances are, if you want to use that quadrant metric, again, your core curriculum, for us, we have heap 101, right, that is skyrocketed up until the right kind of people, everyone completes it, because we’ve set sort of that primary assumption that, hey, everyone should take this, right. This is your starting place. This is your foundation. And so we’re driving a ton of people there, we’ve iterated on it enough. It’s pretty successful. But now, I think it’s, you know, going through a rest of our content. So even if I’m just looking at, again, the highest level metrics of what, what’s my course that like, isn’t getting any registrations, like what isn’t getting used. And if you can start to make these hypotheses about why this is going to help you understand your learner’s so much more, if you can start to answer those questions. What is it about this course? That is unappealing? Is it the wrong audience? Is the content bad, right? This is where you would maybe start to dig in to sort of one of those drop off reports, are they dropping off right away? Because they’re like, Oh, this is not, you know, what I’m trying to do? Or if you see something like, Hey, they’re making it like most of the way through the course that people that find this, then I think it becomes about is it not discoverable? Can my audience like not find this, so just there, right, you don’t always need to be building new things, utilize the content that you have, and use that early data to inform you about what’s working, what’s not. And this is then going to inform what you do build in the future. Yeah, have a course that’s performing poorly, you’re not going to build a you know, the 102 version of that, or the follow up?
Dave Derington 27:17
Okay, let me unpack a couple things. Because I think what you’re saying is gospel for us, like, this is the founder. This is like the foundations of a chapter I would write if I wrote a book on data, in customer education of where to start or a book, you know, it’s that we have stuff. Like, I think one of the things I was talking with somebody one day, who had a kind of a sense of panic around them, like, well, I don’t know where I’m going, I don’t know what to do, I don’t know where to start in. It’s that it’s kinda like that first blush you have after you start to build a program. And then you go, Oh, well, how’s it doing? And in going back to what you’d said before, often you’re, you’re blinded by building, you focus on building and you get into that cadence. But here’s the thing that came to me over a period of many years and, and I tap back into thinking about agile and thinking about it at one point in my life, I was fortunate to be to do some development work. So we did an agile we did sprints, in this term of a retrospective came up. And which is, it’s that moment where, you know, if I, if I would act, act out of like, I take a breath, the team sits around the table in those days that we could do that. Or now we’re in virtual reality somewhere in a fake, you know, meta table. But I don’t think we do that enough. I know I did, where I sat down, and I had the team go through what worked. Let’s look through all of our stuff, what we did start doing was a monthly maintenance meeting. And a maintenance meeting was where we kind of did that retrospective. And to sit down and go, okay, here, let’s throw all that data out. Let’s look at our you know, the engagement map, the engagement model that you were talking about, and I’ve called the discoverability matrix, and go okay, hey, I’ve got this one course it’s standing out. It’s, it’s, it’s weird. Or the one course that we built that should be really performing has a low number of registrations, just like you’re saying. So I think what you’re what you’re really getting us to is, you will have a, let’s say, let’s make a list of all the core data of all the core metrics who want to look at registrations, completions. Let’s have on deck more information, like I can go into my LMS and see the drop offs. Let’s have on like, I’m wanting to put in my fingertips, just the basics, and then dive in. And then every month or every couple of weeks, I do a retro, and then I go back and say okay, now I know I’m building new stuff. But I should go back and look at this one course. So you’re letting you’re navigating. You’re using your Northstar be some of that data. And what you’re saying is, even if we’ve got haven’t gotten to the next step in our conversation that I want to take you which is what are the dashboards and stuff I could automate, right that’s my dream. I’m always thinking about can all this stuff just Be there. Yeah, but you’re saying I’ve got the stuff, I could pull it up. It’s not that hard. But I’m looking at no registrations completion rates, I can even do stuff like attach rate play simple calculation of all the contracts I have coming in, how many of those accounts have actually touched my education? So you think on that?
Christy Hollingshead 30:20
Yeah, yeah, absolutely. Um, I think, you know, I can say this, because I’ve made this mistake too many times of just get getting caught up in the build, I’ve built my one year roadmap, right of all the courses I’m going to build for my learners. Yeah, and not slowed down enough in that build process, to evaluate if my plan is the right one. And if I’m driving the results that I wanted to drive with, you know, this course roadmap that I that I outlined in the first place, and it doesn’t do you a whole, it doesn’t do you a lot. It’s not super helpful to have Wani courses, if your learners aren’t engaging with them, if they aren’t finding value in them. And ultimately, if you aren’t driving value for the business with that content, but it’s, it’s just so easy, I think, to get caught in that trap. And again, I’ve done it so many times. And then, you know, after that year, I’ve built those 20 courses. And then I go and look at the data. And I’m like, Oh, wow, like, six of these courses have never been used. I spent a lot of time building those. You know, and then how to make sort of changes then, and really just scrapping that work. So I’ve learned to sort of slow down to speed up.
Dave Derington 31:37
Oh, you’re saying such good things slowing down to speeding up? We’re doing retrospectives. And you know, what, what struck my attention? Let’s, let’s get into talking about ROI. Because I think, where for me? You know, Adam has used this phrase in some of our lead ends, which is a joke, but it’s actually deadpan serious to me. It’s the stone cold anxiety I have when I look into the eyes of my leader, and they’re asking, What am I going to do next? And that usually is this incentive or this push to build, build, build? Because I think that build trap, like, let’s let’s give you attribution for that. I love that the build trap. The build trap is my leadership has an expectation that I’m going to be building content over and over and over. But in that is what you’re saying to me. And what I feel palpably is that it’s not about how much we have. It’s more, what is that doing for us? And how do we iterate? So we have, like, I’m thinking about this, we can make all kinds of education, stuff, webinars, all that jazz, whatever. But the core of our program, that tight ball of this is we’ve got everybody’s mind, everybody’s understanding about what our product does push down to a diamond. And that it’s, it’s, it’s something that to get to a diamond we have to apply pressure. So it means we have to continue to iterate, iterate, iterate, iterate, iterate, but that’s I know, I’ve failed at it. We’ve all failed at it. And it’s the slowing down to speeding up. That’s it. These are little catchphrases, but they actually are meaningful in this context. That what what I want to if a leader is listening to this, I know we have a lot of leaders do that the part of the challenge you should make to your team going in, is that have you put in time to maintain and improve and to cut? Because we don’t want garbage? Right? So now it’s all it’s gonna unpack Roi. Roi is so okay, I’m gonna slow down. I get really excited about this person, if you can tell it’s this. This is at the spirit. This is the core of our all of our program, the heart. And the heart is my leader comes up, she came up to me one day and say, Dave, how do we get information about the ROI of this program, to our leadership to our board. And I have to tell you, this is the holy grail, what I dream of is that I’ll give you I’ll give you an example of the best thing I’ve ever seen. And this is a shout out to my friends at Gainsight. Because Gainsight kind of like Heap has a really great platform of data and analysis. And it made it really easy for me to actually tie in my education program, my LMS straight into the platform is to produce charts and graphs. I think that’s very similar to what you could do. And very similar to other companies out there. Like imagine working for Tableau Wow. It’s that. Now I want to take what I have an LMS. And I have a data lake, and I have a CRM, and I like I have all this data. And now I want to munch it together and say, Hey, senior leadership, here’s your dashboard. Here’s my dashboard. This is all the other stuff. Like I’m talking reports and stuff that happen automatically so I can go it’s hot, I’m not going to have to assemble it and work for two weeks to put together my data. So that’s how I want to start for the rest of our conversation and really kind of talk to this leadership level.
Christy Hollingshead 34:59
Yeah, I mean, the simplest place I can start is, you need to be talking with those leaders and asking them, what are the outcomes we want to drive? Right, first and foremost for the business. And oftentimes, the role of education is to assist with adoption of your product, right? It’s to help learners that get the most value from your tool. So, really sort of hammering home that point. I mean, that often is, the first barrier is leaders aren’t thinking about that. You know, they’re not really thinking about education, and how it’s helping support and drive those outcomes. But start there, sort of ask them, okay, well, what are the outcomes that we want to drive? And then the second part of that is, and how are we going to measure those outcomes? Now, this is where really close partnership with your product team can be really helpful. So whoever your product leader is, they should be having these same conversations with your executives, right? Well, what are we trying to do with the product? How do they define an active user in the product? What are some of the key features that are correlated with user retention? Right, start to sort of have these conversations with them, ask them these questions. And then you want to align your programs to those behaviors, as much as possible. So you want your learning objectives and outcomes within a course to match these behaviors, you know, that other parts of your organization are trying to drive?
Dave Derington 36:40
So, okay, could you do Sorry?
Christy Hollingshead 36:43
No, and then I was gonna say, yeah, and then once you’ve done that, and you’ve built for that, then this is where it starts to get really exciting if you have access to behavioral data or product data. And you can start to bring this engagement data in with the behavioral data and start to merge those and really paint a picture of how are my education programs actually driving this data, this behavior that we’re seeing in product and usually like once you can show that like, that is where the the light bulb goes off for these executives, and your boss and their boss.
Dave Derington 37:21
So I wanted to key in a couple things you said there in something that came to mind immediately, as you’re painting this picture of, I’m going to talk with my executives, I think that would translate to you, my senior leadership and customer success, support, professional services, sales, sales, engineering, sales, etc. There’s a lot of people. And one of the things that can happen is you get a blank look like, especially when you say how do we measure that? And especially early phase now, this is not obvious, always true. But what I’m trying to get to here is that I want for leaders to take a B, and go wait a minute now. So you’re saying to me, Christy, that education affects adoption, retention, revenue, overall product usage? And what does that look like? To me, what I’ve seen is, again, Gainsight, was a great example of this, where we had a lot of product usage data built into the platform. So I knew that, you know, user one, two, and three went to the website after I did a training. And I could see that spike. And I could also see after a training, like I love this curve, so you have a have a line that’s going across, and that so I went to Acme and I did a training engagement. In on that day, I saw 100% usage, because everybody was there. And then it goes down a little bit. But then what happens, so that was a live training. And then I get into my university, and you see people kind of level off. And eventually, I have a net line going across a stable line. And then my team comes on does an expansion, and then that curve goes up again. And then they do another one and expands to a different team and it goes up again, and it never really comes down. But then then you’re looking at over time, okay, I’m watching that company, are they doing education? Are they using the product? So education is another indicator. And for me, it’s always like, if I see a decrease in, if I can get to that data, this kind of stuff, like really ends up helping support Customer Success Initiatives by a lot. Because where I think a lot of folks want like, this is the other trap. I think everybody’s everybody’s an educator in a very early phase company. And we get caught in this trap of, you know what, I can’t trust anybody else. I’m on the line here. I’m trying to close a deal. I’m trying to retain a customer. I’m going to jump on a call with a customer and do that. And that’s where we come into play. Because the pitch the executive of my hand is number one, we’ve got to centralize but number two, we can look at the data objectively now. Yeah,
Christy Hollingshead 40:02
yeah. And I think, you know, you talked a lot about aligning with the leaders in a customer success function, and then kind of giving you those blank looks. But this is where again, just want to emphasize like, I’ve had a lot of success in partnering with my product leaders, leading those products. If your company has this data, it is the product team who was going to have this data already in understanding this. So they often are going to be able to answer some of those questions for you on wait, well, how do I measure this? What are we trying to drive, and they’ll often give you a really good starting place for the things that are working. And I found, you know, when you use the data to tell you this, it is often very different from what the customer success managers are telling you the key, you know, behavior to drive. So another sort of benefit of data there, right data doesn’t lie.
Dave Derington 40:56
It doesn’t lie. I think we’ve got our catchphrase for the show data doesn’t lie. Sounds like a bond visualizations can lie.
Christy Hollingshead 41:03
But that’s it. So like, yeah, make friends with your your product team. And too often, again, I just don’t see this connection happening enough. Too often, customer education, is spending a ton of time with just those and customer success. And their input is certainly valuable. And you’re gonna work really closely with them. But expand that circle, bring in the product team, because they’re also on the hook, you know, for a lot of these same things around adoption and retention, getting people to find value in the tool. And you want to make sure that they don’t feel like you’re competing with them also, right? That that you’re not training people because the product is bad. You want to have shared goals, you want to help them see also, Hey, how can education help you hit your adoption metrics, right, we’re really confident that this feature is going to drive value for our users, they don’t seem to be discovering it. So can we enhance discoverability through education. And once you help them see you, as a peer, and as someone who’s going to help them hit their metrics, they’re going to be coming to you in the same way that your customer successes, and talking about getting the data and having access to the data. This is also going to build some of that goodwill so that they can help you do some of your analysis if you don’t have a tool where you can do it yourself.
Dave Derington 42:32
Yeah, I love working with product. Hey, sometimes I wonder if customer education should like we always ask, where does customer education live? I don’t necessarily believe it’s in product. But I believe what you’re saying in I live it. If you’re not talking with your product team every day or every week, a couple times a week? Well, how are you going to know what’s changing for one, but you’re really kindred with them. The difference is we find ourselves often in I think we find ourselves in customer success, because we’re really truly aligned with that, that that outcome, like we’re focused on getting the desired result to the customer, you know, thinking in terms of like what Lincoln Murphy might say about customer success, you know, a customer has in mind, I want to use this product to actually achieve my goal. But we have this this discussion between product and everybody in the company like I found those two that our product team thing is thinking one thing and our success team is saying another thing, and sometimes we have to resonate between the two. So that’s Oh my gosh, I I’m really thankful that you said this and hope and I know our audiences listening, but if you haven’t talked to your product team, you definitely should. We’re coming up on time, we’ve got about nine minutes. But before, before we go, I definitely want to talk about anything else you’d like to talk about. I’d like to think about some, some good table stakes type reports and dashboards and things that you might recommend. Once you get your substrate layer of data all together, what are the kinds of things Christy you love to build, so that you can help just automate that? Like, do you have a report delivered to your VP or SVP or somebody or your board about what your education program is doing? And how do you get there?
Christy Hollingshead 44:14
Yeah, I think it’s very different what we deliver to our board. First, what I look at my top reports, and I’ll talk about both. But I think once you have access to more of this data, you can tie in your engagement data with your behavioral data. My favorite report is simply taking a course you know, engagement, and then looking at the learning objectives for that, and then tying that to product behavior. So very specifically, right, what behaviors Am I tried to drive with a particular course? And did I influence that in the product and just like you said, I mean, it’s magic. When you can see that somebody hasn’t engaged with a feature or certain part of your product they take Make the course right and then are now engaging, you know with that. And that is, I think, at the end of the day really going to tell you is your content successful at driving value. It doesn’t matter if you have, you know, 1005 star reviews, if none of your learners actually did anything in the products at the end of the day, it is about changing behaviors, that is the value to the business. And, you know, driving value with your product, because of course, if your learner’s aren’t finding value with your product, then that is not going to lead to Retention and Expansion and all of these business metrics. So I think that just at an individual course, level to tell you, Hey, is my content successful? Yeah, you can get that data. I mean, wow, it’s really, I think, helpful to see. And there’s two things to keep in mind there. I think you want to look at like, yes. Did they? Did they do this at all? Close to the course. Right? Because you can influence it. And there’s ways to I think, see a spike in data. So if you add a practice activity, or something like that, where it’s like, Hey, go and do this. Now. You can oftentimes see oh, hey, they did it. Right, really close to this course. But I think more important is did they continue to do it over time? Right? Did that usage continue to go up into the right and your example before? Because you, you don’t want to get flogged in like, yep, they did it? Once, right, like, because I told them to in this exercise, but again, it’s about changing behavior.
Dave Derington 46:40
It’s like a behavioral analysis chart Do you have? Do you have an example of what you would? Like? I don’t know, that you could share with in words, not envision? How would that Bres.
Christy Hollingshead 46:52
So in most tools, you’re going to use some type of retention analysis. So what I will do is, you know, my course is sort of the first event I’m looking at, in that retention analysis, and then whatever the key behavior is, that’s going to be my second event, or my return event. And I’m just looking at interactions over time since since taking sort of that course. Now, one of the other things you want to do is, when I’m also comparing sort of data with trained and untrained users, is you also want to make sure that you’re only looking at like active users of the product. And I think this is really important, because you start to get much closer to feeling confident that, hey, it was the training specifically, that actually drove this behavior if you’re looking at users who are regularly active in your product. And this is even easier when it’s, you know, not core functionality of your product. But it’s one of these like secondary or third sort of levels of value. Like that starts to become really powerful, and much harder to deny that, hey, like, across all of our active user base, our power users, right, the people who are discovering this content, who are using it regularly, a good chunk of them happen to be trained users
Dave Derington 48:16
into, okay, so I want to share something here, that I’ve looked at, that may be a good idea, maybe you can help unpack this a little bit more and make it clear on how to do this. But what I had done to get to that is, at the point, I’ve been using Google Data Studio and Tableau and some other platforms, and we pull data out of our data lake. So what, okay, let me let me tell you an example of how I got there. So I had my LMS data, and I got access to that in my, you know, my BI tool. And then I had my CRM data. Now I know the accounts, the people, and I can connect that. So we call that a join. Right? So now I know that Jane and Judy and Joe and John from company, Jaguar, Inc, whatever had either done or not done training, one of the things, the easiest thing I did is say, Well, I don’t care about the people right now let’s look at the account level, kind of like an educational attach rate. I may be bastardizing the term and using a quiet right, but this is what I did ago. All right, for his are the accounts that did take training here, the counselor take didn’t take training, here’s the courses that I’m looking at, what is the overall adoption and retention and whatever, comparatively. And that’s easy to do, because I’m not looking at like really low level details. But I can come back and say, Oh my God 50% more usage and adoption of people that are training that’s across the board. And the and these are real kinds of numbers. Like you can calculate this pretty easily. And when you take that to your leadership, they go, why are we not selling more training or wearing it? Let’s just give it away for free. My favorite Exec is like, I don’t care about making money on this. We just need to educate our customers. Well, thank God but if you did Have that if you if you now you’re supporting your line of business financially, and you need to go, I need X dollars to support my team, then you start getting to that, how am I gonna get the more money? So is that is that like a initial good example you would recommend? Do you have better ones?
Christy Hollingshead 50:18
No, I do. Yeah, I think that’s a great example. Definitely like is one that execs see the value in. Particularly, you know, again, most organizations have some type of definition of an active user. So they’re not going to get into the nitty gritty of like, the feature adoption, you know, levels that I was just talking about, but they definitely tend to care about, hey, what percentage of our users become active users, right? The Dow, the Mau, the wow.
Dave Derington 50:52
Yeah, you Yeah, yeah. I like the weekly active daily active. Yeah. So
Christy Hollingshead 50:57
you know, again, I think, exactly to your point, if you can show, hey, after taking training, you know, how many of my users convert into whatever that definition is a weekly active user, or a daily active user? That’s a really powerful one. And then I think the other kind of interesting ones a different split. So it’s not just of users who took training, but you’re actually looking at the entirety of your active user base, then you’re splitting that group out into how many of them were trained and untrained. And so if you see, oh, 70% of our entire active usage, pace, was trained. Like, that’s a pretty powerful metric. Right? If you see that it’s lower. Yes. I mean, this is where it gets, you know, maybe check this metric before you share it. But you know, if of your active usage base, only 30% are trained, you know, that is a little less powerful, right? Like users are able to find out how to use your platform, how to get value, how to use it regularly, without training now, you know, the big thing that’s important there is you still helped 30% of your users get there. So even if it’s lower, even if it’s less, you know, then then half of those users, you still, like supported those users, you gave them the resources, they needed to be able to join those active usage, and out ranks, if you will. So those are the users who maybe didn’t get it on their own that you helped there. But those two tend to just be the the most powerful ones in terms of the level that those execs care about. I mean, my team shows those metrics to our board, actually. Which is pretty impressive to have an education metric and your board deck. And yeah, and it’s because it just it tells such a clear story about the power, power and value of education.
Dave Derington 52:51
Absolutely. And those kinds of things help even more, for example, I would take those kinds of things and be able to slice it further. So one thing I wanted to add is segmentation. So you may be actually looking at segmentation across, you know, small, medium, large accounts, you know, enterprise, you may be looking at industry segmentation, or like, how am I How was my education actually, you know, hitting the different cohorts of people that I might have within the world looks like, I might have education or air, you know, aerospace or whatever. And so what you’re saying, I think, is the greatest way to start, let’s look at high level aggregate data, that if I see my program is actually helping an aggregate, I can share that with the board, they go, here’s a check, keep doing what you’re doing, you’re doing great. And if I can give my marketing team information to say, Oh, well, you have 30%, if I get to 70, over the next year, let’s do a campaign to promote our university or live training or whatever, let’s do workshops, then. So basically, what you’re doing is saying, but what you know, what really makes me feel good about this conversation, Christy is that we haven’t gotten into saying, this is hard, or it’s really scary. We did a little bit. But it’s not. It’s the hard part is saying, where’s my data at? Go talk to people? What am I trying to do talk to product, talk to leadership, get everybody aligned? And you know, as I you know, I’ve moved into a new role at service rocket now. And I’m thinking a lot more about connecting to all, you know, what, what do I say to people that are coming in and building a program for the first time about data? This is the spirit of that, like you’re giving good practical information, and it’s really about the baseline stuff. Are people going to take? Are they taking training? are they consuming it? One thing I really love to do is like, I would offer my CSMs the ability to go and look at a dashboard or report focused on their accounts. And then they can get into like you could even do things like provide ways to incense your, your broad teams to point to your training. And if I can say look, Julie in customer success is enterprise CSM, and she realizes that she can refer people that are trained And our university. And now I can tag Julia as well as doing those referrals I can have that be part of her bonus. It’s data is the foundation of our education program in any customer education professional that’s out there. If you’re not looking at this yet. You know, one thing that I would ask you going forward is like, obviously, you’ve shared gospel here on today’s show. But is there any other place that you would recommend people just starting out or even people wanting to improve? To understand how better to be a data, you know, educational data analyst, I could say, or you hire an ops person, or what are your switcher sage advice to really getting the?
Christy Hollingshead 55:41
Yeah, it’s it’s tough. I don’t know that. You know, as an industry, we do have like the here’s how you do it. Here’s your resource. To be frank, I think it is, you know, my most helpful thing, like I already said, is, find the teams at your company who have data and try to learn from them, what are they looking at? What are they caring about? And then how can you apply that to your program? You know, there’s so many ways to do this, the one thing we didn’t even touch on is bringing in your business data or your revenue data, writing as well, which is a whole other piece. I mean, I could talk about this stuff, I think, at length, but yeah, I don’t want people to feel like it’s hard. I think it is overwhelming, it is easy to feel like well, I don’t know how to do all of this. I’m just getting started. But yeah, you have probably a ton of data that’s useful. It’s just about learning how to make insights from that. And yeah, you’ll you’ll do more over time.
Dave Derington 56:48
Absolutely. Well, with that, I think we’re at a good place. And actually, it’s sounding like we might need to have a another, another session, you know, maybe sometime in 2022, and talk about the business data. That’s huge. Like how, how do we actually connect all of this stuff. And my ideal is, there’s a process that I work through, and a guide book that I work through, that tells me as a customer education person, the things that I should be doing. And here’s some base reports that I should start with. But I think what we’ve done like the call to action here, that you just made was talk to people talk to product, talk to your leadership, understand what your North Star is. See what your data sources that you have are, and then start with the basics. You know, start with completion rate retention, or completion rates and registrations start, start looking at volume, start looking at if there’s problems that you’re seeing, use those and go back to your product team and talk to them or figure out what’s wrong with your own material. But it’s it what you’re saying out of this is data is about a conversation. And data is a continuous loop of going back and looking at how did we do and it’s the not hard parts as we do have the data, the hard parts is when you get to wanting to automate it in building the dashboards and things like that. And that’s, that’s where is that? Is that one of the things that he can help us with on?
Christy Hollingshead 58:08
Yes, definitely very powerful tool. The last thing I just want to mention is yes, the data is important, but data is only you know, valuable if you act on it, and you make changes to the data. So while we talked a lot about getting the data here, I do want to go back to the beginning. You know, iterate, iterate, iterate, use the data to make changes to your program and you know, measure again, use the data, like just having it doesn’t do a whole lot of good.
Dave Derington 58:41
I love it. That’s a great way to lead out anything else you want to tell us about you or your company and we’ll wrap up
Christy Hollingshead 58:48
No, but feel free to reach out to me on LinkedIn. Christy Hollingshead – I’m happy to chat with anyone and anyone about data for days on end.
Dave Derington 58:59 I’ve everybody’s really looking forward to this episode. So yeah, reach out to Christy. She’s an authority on this and one of the best people I like to work with. So let’s wrap up. Christy again. Thanks for joining us today. This is this has been wonderful this is I’ve been looking forward to this for a really long time and I appreciate getting you on the show. If you want to learn more, remember that C lab has a podcast website at customer dot education. You can find all of our stuff there we transcribe every episode so there’s value voluminous notes. Now you can have everything you can if you’re you’re hanging on every word that Christie is saying here, you got a copy of it. And on on Twitter, I’m at Dave Derington. I’m also mainly on LinkedIn you can see me there special thanks to Alan Coda for the wonderful music that we have here. And if this helped you out, remember subscribing in Apple podcast, overcast Stitcher, Spotify can go on your pod catcher of choice, this podcast is out there if it’s not we’ll get it there. And please please consider leaving us a positive review. We thrive on those we’re trying to spread the word and our job here at SEALAB is to help conversations like this and help source expertise and get you all up to speed quick. So to our audience, thanks for joining us. Go out there, educate, experiment and find your people. Thanks everybody.