Adam Avramescu: [00:00:00] Oh, okay. The only voice you can do after having a adult sized IPA,
Dave Derington: [00:00:08] like the one last night,
Adam Avramescu: [00:00:08] that’s how the bartender sold us. He said you could have a adult sized IPA or a child sized IPA. And. No one ever chooses the child size.
Welcome to CELab customer education lab, where we take education, myths and misconceptions and give them a rightful spanking across the bum I’m out of ever Moscow.
Dave Derington: [00:00:37] Now I’m Dave Derington
Adam Avramescu: [00:00:39] and today we are happy to welcome you to national Chucky, the notorious killer doll day.
Dave Derington: [00:00:46] That’s a real day.
Adam Avramescu: [00:00:48] Very spooky.
Dave Derington: [00:00:49] I can’t believe that it’s a national day of, it’s also what greasy food day. It’s
Adam Avramescu: [00:00:54] also greasy food day, but we don’t want to, we don’t want to share too many of them because we got more in the can. Okay. the reason why it is. Pooky spooky days because we’re getting close to Halloween over here. And, I can date this podcast because we are recording this the day after we gave our session at DevLearn.
Dave Derington: [00:01:11] Yay.
Adam Avramescu: [00:01:12] What’s DevLearn Dave
Dave Derington: [00:01:14] dev learn is actually a pretty amazing conference and it’s really all about learning professionals. I think they target learning and development, but. As customer education people, there’s a lot to learn here. So I’ve been here all week, learned a lot, really looking forward to that.
We’re going to talk today about our presentation.
Adam Avramescu: [00:01:32] Yep. And we’ll have a, we’ll have another episode coming up where we do more of a summary of the DevLearn conference itself, but a really cool conference because it’s not necessarily about the, executives who hold the L&D budget. It’s about the practitioners who are actually creating amazing things.
So we’re going to go through our presentation that we gave. And, you may not have all the same visual elements that we had in front of the classroom. But what we decided to do was a live episode of CELab. So we’ll share the hypothesis that we are tearing down. But the session that we chose to do is on measuring ROI by collecting, connecting, and visualizing data.
Dave Derington: [00:02:14] All right, let’s go ahead and do this. Adam, why don’t we kick us off?
Adam Avramescu: [00:02:17] all So we introduced ourselves to the crowd by mentioning that we are part of the world of customer education, which. Is a little bit different from the typical L&D world that a lot of folks came from. And in fact, there were some customer education people in the room with us too, which was really exciting to see.
but part of how we frame that up, we talked a little bit about our podcast. but we also talked about the fact that for those of us in customer education, We are in a world where we are continuously renewing our customers, especially in SaaS customer education, where we have software, that software is on a license and the license has got to renew.
So education is not a one-time thing. It’s an always
Dave Derington: [00:02:59] continuous learning is the phrase that continues to come up. I see what I did there, continuing to develop new content, continuing to learn. It’s all very,
Adam Avramescu: [00:03:08] yeah. And we’ve got to always keep it updated over time, but yeah. we don’t have the advantage that a lot of folks have when they’re doing internal learning, especially at a smaller company where you know, all those learners, you have control over them.
ultimately the organization can, audit them or fire them if they don’t take your training. Not many people actually do that, but some do and compliance-based environments, we don’t have that same level of touch, so we have to be able to do things at scale.
Dave Derington: [00:03:33] Exactly. So we’re going to kick this one off and we’re going to begin to talk.
like we did it at the conference about data. So let’s kick this off. Adam, what is hypothesis number one,
Adam Avramescu: [00:03:45] hypothesis. Number one is that data will speak for itself. Or if you’re a grammar, nerd data will speak for themselves.
Dave Derington: [00:03:55] It’s good that we’re using good grammar on this show.
Adam Avramescu: [00:03:57] We should always use good grammar.
Dave Derington: [00:03:58] We’re wondering whether this is provable or not. And I would say probably not
Adam Avramescu: [00:04:04] tend to agree. we think this is false data. Doesn’t actually speak for itself or themselves. And to do that, we wanted to share a few quick stories about how data might seem objective, but ultimately it’s used in the service of storytelling.
So data can tell stories. And for a quick example, we pulled a reference from a Forbes article that Forbes article showed a table. That was, and we can link this article as well, but it showed a table that had the education levels listed out. So from doctoral degree, all the way down to less than a high school diploma and was correlating that with unemployment rates and median weekly earnings.
And when you saw it expressed as a table, you could see that the numbers were going down or going up as the education level went down or went up and you could see that there was. Some sort of correlation, but it wasn’t necessarily that visual or that interesting. we then shared a different version of that exact same report, the exact same data, but visualized to actually show it as a bar graph.
And then Dave, what could you see when you showed it as a bar graph? Instead of just looking at the table?
Dave Derington: [00:05:08] When you’re saying this is a bar graph, you’re seeing how different things are between each one of the education levels. So when I looked at the next chart, I go, Oh my gosh. If I got like a professional degree, the monthly salary was significantly higher than just maybe just a bachelor’s degree or no degree at all.
So it’s really telling a story in impactfully, visually that, Hey, this data has something that I want to learn about.
Adam Avramescu: [00:05:33] Yeah. There’s something about the way. That visualizing data can actually tell a more emotionally resonant story. So here, when you start to look at the fact that it’s not just a little bit of a step up in salary from degree to degree, it actually becomes very, multiplicative.
It’s like an order of magnitude higher that tells I think a much more meaningful story. So the point that we tied that back to is when you were in the room with your executives, arguing for budget or arguing for the influence of your training team or customer education program, you can’t just bring in the numbers and expect them to resonate.
Especially if those numbers are just on training activity data. So that brings us to hypothesis number two.
Dave Derington: [00:06:12] Okay. And that hypothesis is your training activity data. We’ll buy you that coveted seat. At the table, do we think that’s true?
Adam Avramescu: [00:06:20] I don’t think that’s true. I don’t think we can just rely on a training activity data
Dave Derington: [00:06:23] that is not as absolutely not true.
And just like we saw in the previous example, if you’re going to be in a room with your executives and defending, trying to explain your ROI, the data alone is not going to buy you anything. So what we’ll, what we really need to do is of course, Like we discussed, we need to present a good visual representation of the data.
We have to make that clear and. often the reason that we as education folks show them training activity data is because that’s all we’ve got. That’s what we’re going to focus on collecting, connecting, and visualizing that data. But before we do, let’s do another story on him.
Adam Avramescu: [00:07:02] Sure. So this is a parable and I actually may have shared this on this podcast before, but it’s one that I’ve heard from, Ruben Tasman.
Who’s. Been actually very active in the e-learning Guild who sponsors a devil or an earth there they’re the ones that actually put it on the parable that he shares is a story where you are the L&D leader at your company. And you’re going in to argue for budget and to do that, the activity data that you show is, Hey, we have this many people in the organization, come in and take.
The required security training or the required safety training. and we had this many people attend and this many people pass and I need the budget next year to do more of these. can you give me some money please? Now, meanwhile, the floor manager or the floor supervisor is walking in and she is also tasked with managing, the safety of the workers who are working on the floor, the same people who you’re trying to train on safety, but if she comes in and says, Hey, our error rate is still going up.
Our defect rate is still increasing and it increased by 5% this year. So we need the budget. We need that for more on the job training, which might not come through the L&D team, or we might need that just to enhance our safety mechanisms. Who do you think is actually more likely to get that budget?
Is it you coming in with your trading activity data or is it her coming in with the actual data about what’s happening on the floor?
Dave Derington: [00:08:22] I would say it’s probably going to be the line supervisor.
Adam Avramescu: [00:08:25] I would say so too. And that’s the problem that training professionals are in a lot of the time where we’re trying to tell a story about our influence on the business, but we’re telling that story.
In the wrong way, because we’re talking about things that happen in our classroom, not the outcomes that we’re affecting in the real world.
Dave Derington: [00:08:41] Okay. Let’s take a sidestep and let’s go back into the world of digital marketing. So Adam, you and I have spent a decent amount of time in digital marketing, outside of what we’re doing with training, maybe inside it.
and we want to put this into perspective now. Let’s. Perceive a funnel, get a visualization of a funnel on your mind. And there’s four levels. Now this is in marketing. So we are doing that conversion funnel right at the top of the funnel. We have a band that says awareness. Then we have a band that says interest and that’s the next step down then?
It’s consideration. If a customer is, or a potential customer is interested in buying your product where they’re going to have a discussion with the salesperson and then finally conversion. So we think a lot about funnels in the business world as we do sometimes as well in education. So in sales and marketing, you’re trying to get to that conversion.
That’s getting that person to buy something with you and you can’t get into that funnel though, unless we go through every step of it. We have to begin with the awareness. We need to make a great website, right? We need to, may need to do advertising. ultimately if somebody is interested, then what are they going to do?
They’re going to fill out a form or they might pick up the phone or they might talk to your chat bot. any of those things could be, do be happening. you may also have that sales development rep cold calling, emailing, et cetera. Once you get people into that funnel, then you’ve got data. Now what’s important about this.
A lot of us will look at that last conversion. You’re going to measure the conversions.
Adam Avramescu: [00:10:08] That’s what the business cares about. Ultimately how much did we sell? I don’t care how many people visited the website. If. Those visits, don’t actually lead to some sort of conversion.
Dave Derington: [00:10:16] Exactly. And what’s more what’s important here is that number is, can, is very important, right?
What is our conversion rate? But what we’ve learned in digital marketing is that every step in the way have proxy metrics, right? We start up with how many people were interested in going to our site. That’s Google analytics, how many people have interest in there? How many people fill out that form? So every step of the way we’re able to do some measurement.
Ultimately what we’re going to do next is. Turn this into something for us as educators. We want to sit down and understand what you’re going to measure, how you’re going to measure that and how you tell a story about
Adam Avramescu: [00:10:51] your day. Yeah. Which is important because the reason that funnels exist in digital marketing.
I think anyway is because marketing used to be perceived as a frivolous function in the business. It was, Oh, you’re going to go to the trade show and hand out pens, and then something’s going to happen. And then we’re eventually going to make some sales. that’s not really true for digital marketing.
Digital marketing actually gave marketing a seat at the table along with the sales team. And the reason that it did was because they found a way to measure those proxy metrics that would ultimately lead to the thing that the business really cared about. I E. Sales and the way that they did that was by measuring those leading indicators before the purchase.
So before you ever get to conversion awareness, interest consideration, those metrics, those are all leading indicators, ultimately of the sale. the reason that those marketing funnels exist. Is to help, define those leading indicators that are ultimately going to lead to conversion. And it gives the marketer something to measure and to report on that is actually going to help them define where drop-off exists.
Now we as learning professionals, We actually have a tool that works this way. Although a lot of us don’t think of it like this. It’s Kirkpatrick’s four levels. And often if you go and search Google image search, Kirkpatrick’s four levels right now, and chances are the way you will see it displayed is, it might be some building blocks sitting on top of each other.
but it also very likely might be. A pyramid. Now I would argue that viewing it as a pyramid, leads us into kind of a pyramid scheme of bad data. I’ve made that joke so many times that it never lands. It never lands. I got a real grown in the room, but, we start by measuring the data that’s most available to us.
The reaction data, most of us do surveys or smile sheets or some way of knowing. When people came to a course or viewed an article or whatever our training is, what did they think of? It? That’s the easy part. That’s the easy part. And then most of us have some way to measure learning. Was there a test?
Was there a pre and post eval, something like that, but then it gets harder once you go further up the pyramid, right? you’re climbing up that mountain and your energy is, starting to flag. You can’t really measure behavior or results. That well, a lot of the times we just don’t have the resources to measure those things.
But what if we start to think of it a little bit less like a pyramid and invert that pyramid and turn it into a funnel? that means we’re going to start by measuring reaction, but only in service of, so what do they learn? Anything we’ll measure learning, but, so what did they change their behavior?
And we can measure behavior so white. If it generated results. So even if we’re still not able to measure all of those things all the time, at least thinking about it as a funnel, instead of a pyramid starts to orient us around what those end results are. Now, this talk is about ROI and we have ways to measure ROI in our field as well.
There’s the Jack Phillips ROI method, and there’s a whole ROI Institute. Built on top of measuring the effect of training programs, but even the ROI Institute says you won’t always get to that level with all of your programs. Why? one reason is because it’s a big investment to measure ROI of all your programs.
You’re not always going to be able to do it because there are a lot of calculations that go into it and often it’s resource intensive to measure ROI. The other reason is because if you look at the steps in the process to measure ROI, one of them is. After you collect your pre and post program data, you have to isolate the effects of the program, which means you have to do something that’s really tricky in our world.
It was tricky for the marketers to do. You have to find some way to reduce all the noise and isolate the effects of your program. And frankly, training is messy. We’re not always going to be able to do that. We’re not always going to be able to isolate the things that we did. So that brings us back to, our friend Ruben Tasman, who gave us the parable at the very beginning, check out his book.
It’s called learning on demand. Really cool. but he says we have to stop thinking about training as if it’s going to be deployed locally. And what that means to me is that we have to rely on analytics more to judge the success and failures of our training, but we have to be able to tie those systems together as well.
So the way I interpret that is before we can really report on the effectiveness of our programs, we have to find ways to collect the right data in service of the right goals. And we have to find ways to connect and tie it together so we can report on it at scale. But how do we do that,
Dave Derington: [00:15:18] Dave? Okay. That brings us to hypothesis three.
So we’re talking about data and we’re covering again, those three aspects of collecting, connecting, and visualizing each gets progressively more. Interesting difficult. Maybe. So let’s start with collecting. And the hypothesis we brought to the audience here was your learning management system has all of the data that you need to collect.
Now, what do we think about that? Yeah, Adam,
Adam Avramescu: [00:15:45] I don’t like it so much. I don’t know that most LMS has have all the data you need to collect.
Dave Derington: [00:15:50] No. so LMS does in fact capture a lot of information, much of the time, the tools we use LMS in particular. our tasks with capturing that data, they do a pretty good job that we have email.
We have names, we have other profile data and going beyond that, now we know who’s taking the training. One of the things that we immediately want to know, how long did somebody take on a module? what was their score when they completed a quiz or exam? did they complete something? and did they have any feedback?
Was the material that they. Consume good. Did they like it? Do they hate it? Do they find something wrong with it? All that stuff is super critical, but is that enough?
Adam Avramescu: [00:16:27] We pulled the room and what we found out, really it validated this point for most people in the room. th their LMS is not an Island.
no. LMS is an Island. they were connecting their data, just like we do to other systems of record, like their CRM, like Salesforce or to their HRS for those who are. Working on, HR based training, but the LMS isn’t the only source of data that’s flowing into those systems. A lot of them are also trying to get their survey data out of a system like Qualtrics, or maybe their video data out of a system like Wistia.
So there are several different systems usually that have to report into some source of record where you’re going to ultimately do your reporting and data visualization.
Dave Derington: [00:17:08] So let’s go a little further again, the task here of collecting Adam, that we argue is you need to start somewhere. You look at the landscape of all of your data.
You just mentioned all of these different resources,
Adam Avramescu: [00:17:19] right? I might need to start. It sounds really complicated. It
Dave Derington: [00:17:21] is very complicated. So where you start, where I always start, I get, I tend to get stressed out. If I don’t have something listed on a piece of paper where I can see anything. Or where I can see everything.
So in this case, we introduced the concept of the data dictionary. again, for our audience, not a lot of folks were aware of this and you may not have heard this term as well. Data dictionary really is just a centralized repository of information. Now this is all the data, it’s about your data.
What fields in your, or what objects in your LMS are important to you? what meaning do these things have and what relationships is the data and data have to one another, right? there’s a lot here. So basically what we’re saying is we want to list everything out. We want to look at all the sources of data.
We have put them in a spreadsheet, put them on a piece of paper, determine how you’re going to use that data. And ascribe some meaning to it. Additionally, you might also want to think about the format of that data. It might come in a particular way, that might be X API or something like that.
Adam Avramescu: [00:18:22] Yeah. And this is the reason why a lot of folks in the learning world are using X API and reporting into a learning record store. we’ll talk about that again in a moment. But I think for a lot of people who have only been exposed to those concepts, but not the more general versions of what they represent, the idea of taking.
Some source of data using some standard, to pass it in. Through an endpoint into a database, you don’t necessarily know why you’re doing what you’re doing. So it does sound like a data dictionary is valuable. but Dave, it sounds like a lot of work.
Dave Derington: [00:18:52] It is a lot of work at first. But the important thing behind this effort, the building your data diction building your list of.
Of software, is that you’ve listed all the important things that you can use. And you can share that with other people. Now, this becomes particularly important. If you’re working with others to help you get to visualization, we’re going to talk about later. Okay. So I want to end this on one last point, which is, again, what we are trying to do as educators is get out of our little comfort zone of thinking about all of our learning data, the standard stuff, the easy stuff, and start speaking the language.
Of the business. So let’s go a little further on this. We, we talked about levels of data analysis and, we had a chart or a table, and we talked about the audience that you’re working with, what kind of data that you have and what the goal is right. For using that data. We began with an L&D team.
And for those of us custom customer education folks, that’s our team. That means, Hey, in my LMS, I want to know how many people are enrolled, who they are. What is the completion rate? This is the stuff I talked about previously, things like drop-offs and smile sheets that is going to help us refine and optimize our program.
That data is super important to us. Our leadership might not care, but for us, we can say, Hey, this module is not doing so well. This one is great. but then we start going down the line and we start thinking about other people that we know, and we work with every day and how do we affect or impact other people in our business that might be managers that might be customer success managers in our case?
I like to use that example because let’s say we’re working with a customer success manager. when I was at Gainsight, I had, Oh, I had built tables and things in the product itself that we’re able to show. What who had taken training, what training have they done? And CSMs really wanted that. And they loved that because then they, when they got into call, they knew, Hey, Adam had taken this training and he taken all the training.
In fact, and I have a confidence level that this person is going to understand the material that’s really powerful, so we can get activity by, by it. Customer cohorts, right? Different kinds of segments that you talk about in customer success, you can get evaluation scores. NPS is a really big thing.
and so on, there are a lot of metrics that your CSM, so your customer success team is going to care about. And then let’s finally get down to exactly right. We’re going to try and get to that seat at the table. And the big problem is we need to give our executives something that means something to them.
What does that mean? what impact do we have on attach rates? Did we affect our C-SAT scores? Did we bring down the overall time to first value when a customer onboards and they complete that onboarding, these are all really important things. In particular. We have things like, support too.
So if you’re working with a support team and they’re getting a lot of calls, we can aid them in call deflection. again, we’re talking here about who’s using. the technology, how much are they using it and are they satisfied with it? so at the end, not all data two is created equal. So in this data dictionary, and we’re talking about, again, you’re going to be starting to look at all these different cohorts, just like I talked about before and define who we’re going to surface reports and to whom
Adam Avramescu: [00:22:09] now, as you’re starting to collect data, you’re going to find that to get to those deeper levels of analysis.
You are going to have to connect some systems. I E it’s. Really difficult to do this in a spreadsheet where if you’re going to say, Hey, I want to take, all of my training data and find my trained versus untrained customers, and then find the impact on C-SAT or attach rate. But chances are, there is some data that you’ve already collected even before you’ve connected your systems.
And you can still use that. Most of that is still useful for you to optimize your own program. So for example, you probably have some sort of value metric that you use, like upvotes or course completions or survey scores. You probably also have some sort of metric that indicates discoverability like enrollments or page views or search terms.
those pieces of data in isolation don’t really mean a whole lot, but paired, they help you make decisions about your own program. So for example, if something is both highly discoverable and highly valued, how if I.
Dave Derington: [00:23:05] Those are hard.
Adam Avramescu: [00:23:07] Words are hard. If something is highly valuable and highly discoverable, that’s something you might want to invest in higher production values, new formats or something to really keep promoting with your customers.
That’s flagship content. If something’s neither valuable nor discoverable, that’s time to slaughter your darlings. It’s hard to keep all of your programs going at once. Every time you have a piece of content in the world, you need to maintain it. So these are good candidates to just get rid of. But the most interesting one is when you start finding disparity between the two, if something is highly valuable, but not really discoverable, you can start to make decisions about how to add search terms or tags, how to surface it and your navigation, how to put it above the fold in your LMS or other places where it might be displayed, or if you’re using in product education.
Those might be things to add in product links to as well. And finally if it’s discoverable but not valuable. now you have a heat map essentially, of things that you can fix. So for example, at Optimizely, we had a spreadsheet that we used. It was not a super sophisticated tool, but it was a way that we put all of our most viewed, but least upvoted articles.
I E the ones that had the lowest upvote or downvote ratio, and we heat map those. So the burning ones that had a really high negative rating. That was our heat map that we went to go and update next quarter, or next week, we also relied on other data that our other peer teams used. So one team that we supported, quite often was support.
So we always looked at their dashboards for what the most common ticket types were, so that we could start to prioritize which pieces of content we could create to address those top ticket types. And finally, Still a pretty much in isolation. We could look at the articles that were freshest or at least fresh what had been updated most recently, or at least recently.
And then who could we rely on to help update it? So just by looking at dashboards of, how old articles were or the last update date, you don’t have to connect any systems to look at that. You also don’t have to connect any systems to look at who are the people who are offering articles most frequently so that we could reward them for our efforts and continue to draw them in.
But that only gets you so far, Dave, how do you start to really connect those systems?
Dave Derington: [00:25:15] let’s continue with this. let’s think about, what the next step is in this. our fourth hypothesis is you need a data scientist to connect your systems and your data. What do we think about
Adam Avramescu: [00:25:27] that?
I think a lot of us don’t have data scientists at our disposal.
Dave Derington: [00:25:32] A lot of us don’t and it was interesting in the audience. There were a couple of people, one person in particular had, had an analyst or a data scientist on their team, loved it, which was amazing. And they were just beaming. so to do any sort of measurement, we’re going to start looking past operational metrics, And we’re gonna start getting into the bed of the business data. We’re starting to look more broadly how we’re affecting other programs. You can’t just collect your data. You have got to connect it. So going back to the last hypothesis, we were talking about our LMS. There’s a lot of stuff you’re almost can’t collect, right?
If you’re lucky and you have other things in there, great, you can get more, but chances are you’re going to be using other systems. So you’re gonna have a strong need to get all that other data into a place that you can access afford it. so let’s continue. Let’s talk about how this landscape of.
Software. we’re in the, we’re in SaaS,
Adam Avramescu: [00:26:22] right? we have data coming in from so many different systems and everyone uses the best of breed tools.
Dave Derington: [00:26:26] It’s amazing. And you have, gosh, I could sit down and I showed the audience a chart and you could have things like Gmail and HubSpot and Salesforce and Zendesk and Wrike and MailChimp.
All of these different systems are out there and are they talking to each other by default? They don’t. That’s where we need to start thinking about how to bring all these, how to connect all these things together. Sure. You could hire a data scientist and they would help you make sense of this.
But we in our talk presented a process and we do this today, presented a process that you can follow on your own. So to begin. There’s two really important things that you need to do. One of them is you’ve got to find a home for your data, a singular home, right? You’re going to get all of the data that you’re looking at and other systems and bring that into one place.
Otherwise, how are we going to do? And a simple use case, we would do a pivot table or something like that in an Excel
Adam Avramescu: [00:27:20] who doesn’t love a good pivot table.
Dave Derington: [00:27:22] I love me a good pivot table.
Adam Avramescu: [00:27:24] like some V lookups.
Dave Derington: [00:27:26] Ooh. How about a cross tab?
Adam Avramescu: [00:27:29] Like Excel magic. We love Excel magic.
Dave Derington: [00:27:32] So number two, you’re going to use tools to connect those data sources, and then you’re going to bring them in.
You’re going to use a term, like ingest that in your repository. So once we have that home, we’re going to connect all the data sources and then we’re going to start working with that data. let’s talk about this a little bit more. Okay. So the next slide we presented was again, on that topic of getting your data into a centralized location.
And we brought up on the screen, several different things. There’s this mess of data. You need to get all this stuff into place in terminology that you might hear about, or things like a data warehouse or a data Lake, or just a database,
Adam Avramescu: [00:28:10] a lot of folks in the room. Especially people who come to DevLearn.
They’re very interested in thinking about tools and technologies like X API and learning record stores. And we were trying to draw the connection here. That X API is an API. A learning record store is a data Lake essentially, right? Where it’s a repository and a database that X API statements go into it so that they can be made reportable.
So it’s not different technology than what we use in other places. We want it to draw that parallel.
Dave Derington: [00:28:43] and Megan Torrance. And one of the discussions that I attended talked all about, like the intro to X API and how this all works achieve and said, Adam, That SAP is API, right? It’s just a standardized format.
Adam Avramescu: [00:28:55] We’ve got to have her own show.
Dave Derington: [00:28:56] Yeah, we definitely want to, so yeah, you could be using an LRI so you could be using something else, but really you could just start manually. The first step I always like to take is I’ve laid out all my data and now I’m actually starting to work with it. and in just a basic spreadsheet and just a basic series of tables that I can start to do stuff with.
So from there, we talked about common types of connections, right? Connection methods. We listed out six of them and we’ll go over them really briefly. But this helps you to understand like the landscape of interconnectivity. Okay. So the landscape of connectivity here is you again, we talked about a spreadsheet.
That’s easy. we all know that we love that we can download CSVs, create new tabs, do some pivots and do some magic, but that’s not going to scale with you. If I have to do that report every, Oh, heck. when I was at Gainsight, I did this and I updated it once a month and it took me several hours to put all this data together and do a pivot before I started to automate the process.
So yeah, it’s, non-trivial, it takes some time. There’s still ROI on that. I started to doing my, started doing my reporting just in Excel and went from there. So from that point I started talking about other connection types. one of my favorites is I-PASS iPASS
Adam Avramescu: [00:30:08] while it reminds me of episode 15 of this very show where you talked all about I-PASS.
Dave Derington: [00:30:12] I talked all about that. So you can go back to episode 15 and be a great call for call out for you, integration platform as a service is what that is. there’s many options out there. What we presented to the audience was, if I, if this, then that, if Zapier, those are. somewhat, those are on the easier end of the spectrum of the products that you can buy there.
They’re more consumer grade, but there’s other options out there. Basically when I pass solution, does, is it connects point to point brings data in, can do that based on different activities or events. So if somebody started a course, it could fire off a message and say, Hey, Adam took this course today. So that’s a really useful technology, to bring data, to connect data together.
So let’s go over a few others. we talked about native integration. What comes to mind first is most of the LMS providers I’ve had give us a Salesforce native,
Adam Avramescu: [00:31:02] and if they don’t, then we should all be a little bit worried.
Dave Derington: [00:31:05] We should be. integration is really helpful because if I have all of my same learning data in Salesforce, and I’m a heavy Salesforce user, guess what?
I can write reports on that platform. I can make dashboards on that platform and I can start connecting data about my customers. Back to my learning data,
Adam Avramescu: [00:31:22] which is why, if you’re a customer LMS, you probably do have a Salesforce integration because that is the system of record for most customer facing organizations.
And if not Salesforce, a different CRM, some people use sugar or other CRMs that are out there that said not all native integrations are created equal. So one point that we wanted to make here is that it’s really important that as you’re integrating or implementing a new LMS, you should really look at what those integrations look like.
And. Work with your business informatics or business intelligence or business systems team. They go by different names, but they all do some version of this, that the people who work in your data warehouse and, rights equal all day. they’re the ones who should be looking at the quality of these integrations to figure out, can they do what you actually want them to do?
Just because someone says that they have an integration doesn’t mean that it’s going to do what you want. And the thing to especially be worried about is if someone starts saying, Oh, we can just write an API for that. There is no, just write an API. That’s a lot of hard work, but we still have a couple of spots on the journey before we get to full API.
Yeah.
Dave Derington: [00:32:22] let’s bring it full circle here. So of course we have X API, which we didn’t go into detail in our audience, but for those of you on the show, this is a standard. This is a, an in an education industry standard. That is basically, it is an API. but it works with learning record stores.
Pretty commonly.
Adam Avramescu: [00:32:41] Yeah. If any of you work with SCORM, currently, SCORM is a standard. It’s not an API, it’s a standard. X API is generally a lot more flexible in terms of what it can report.
Dave Derington: [00:32:53] And I sat in a session, we learned about X API statements, and there’s a lot of goodness in there
Adam Avramescu: [00:32:57] indeed.
Dave Derington: [00:32:58] in the last two things, are ETL tools.
Now ETL stands for extract transform load. The times you’d want to use that as really with big data because that’s. Not like an I-PASS solution where you’re putting bits of information back and forth. This is I’m pushing huge amounts of information. So we in our industry is probably not going to hit this, but it is out there.
And sometimes these ETL tools double as an I-PASS tool. And then finally you stole my thunder a little bit on the API or
Adam Avramescu: [00:33:26] all those things you can get rid of it and editing if you want. We can pretend I didn’t say it.
Dave Derington: [00:33:31] No, we’re going to leave it in. That’s what we have fun. APIs are hard. I’m just gonna admit it.
API APIs. if you’re not using an iPASS solution though, I-PASS solutions are there to connect through those APIs automatically. So somebody has done the work for you, but let’s say, I’m not going to use that. And my security team doesn’t trust me. I am simply going to write the good myself or hire somebody to do or have my development team do it.
there might be times when you indeed want to do that. I’ve done that before, but I’m going to tell you that it’s. really super hard. And it’s not for the faint of heart.
Adam Avramescu: [00:34:02] Yeah. So we visualize that and we showed the spectrum of going from the easiest solutions to pop in. Excel’s easy, like it’s accessible, you can use it, but you have less control ultimately over how flexible you can then connect data to each other and how robust that reporting is going to be all the way over to API APIs, which are the hardest for you to do personally, but are going to ultimately give you the most customization and flexibility.
But the point that we really wanted to make here was for you as a customer education leader, you need to make data connection, a priority. A lot of people go and argue for data collection, which is great, but then they have a bunch of data sitting around and they have no idea what to do with it. solution one to that is you have to have a good idea of what your funnel is going to look like, what your results are supposed to be.
And we’ll get into visualization in a moment. But the other piece is you really have to work on getting the resources from your BI team or business systems, team, or whoever they’re called in your organization to work with you, to do one of these types of connections. That’s the team that’s going to be able to work with those ATLs.
Dave Derington: [00:35:04] Absolutely. And another story to add into this is this takes a lot of time why Adam, you said make this a priority. Is. Pretty much every company I’ve come into, whether I’m doing the work myself or I’m working with other teams, the bigger the company that you have, the more siloed and slow things are going to go.
Cause there’s more processed. That’s not necessarily a bad thing. It’s reality. So when you come to an environment and you’re saying, I’m going to get all this data and I’m going to start connecting it together. so you want to start early. You want to make sure that you give yourself enough time, to anticipate.
those in your organization, we’re going to be asking for reports that you might need. Maybe nobody’s doing that, but it’s going to come along sooner than later.
Adam Avramescu: [00:35:43] Yeah. Usually those teams have to plan their work in advance. Cause they’re getting requests from all over the business to connect various data to each other.
I’ll tell a story too. This is the podcast exclusive story. I didn’t tell this at the conference. two days ago I was in the elevator bank at our office and I was, I was waiting for the elevator. With, one of our data scientists who was in a conversation with another person. So I was overhearing her conversation and she was using a metaphor of a big black box that most people don’t know how to get into and, how to work with, and I just chanced it and I said, Are you talking about our data warehouse?
Ooh. And she was like, ha how did you know that this stuff is hard to work with? So you don’t necessarily want to be doing it yourself. if you can use Excel or an iPASS tool yourself. Great. Otherwise ask for help. Cool. So this brings us back to where we started, which is the data interpretation is not objective it’s storytelling.
So that’s our final hypothesis. we would say that hypothesis is true. data interpretation is not object, so we’d like to share some ways that we visualize data now. This is not conducive to podcasting because we’re talking about an inherently visual medium, but Dave and I will try to walk you through some of the storytelling that we use as we visualize this data.
Dave Derington: [00:37:03] Yeah. We’re going to take a go of it. Maybe we’ll make a YouTube version of this as well.
Adam Avramescu: [00:37:07] I’m also doing a great dance right now that you also can’t see on the podcast.
Dave Derington: [00:37:10] we try out, we try. Okay. at next, what we did is presented a series of reports and I’ll try to visualize these mentally as well as I can.
the slide I was presenting showed three different reports, and this came from my tenure at Gainsight, where we were fortunate enough Gainsight. Gainsight’s a really interesting and cool product that allows you to, you have all of your CRM data. You have all of your. Whatever you can ingest whatever kind of data that you want, and you also have your product data.
So really cool. The first chart that I showed was a time to first value chart. Did we affect, could we affect as educators? How people got onboard, How long did it take a company from when a deal was closed? One, two, when they launched, And they went through that onboarding, learn the product and use the product and got up to speed.
I’m going to come back to that in a minute, but that’s an interesting chart. The next thought we had, and this was more experimental. We wanted to see how training impacted sales. So you’re thinking about that. Like how does training, how could training impact sales? what we had found, and I found this at a couple of different places, if you don’t have a lock down behind some kind of a portal, at one of the companies that I’ve worked at, there’s a single sign-on integration.
You can only get access when you have access to the product. There’s good and bad both ways, but if you added more to have. A wide open portal for training. the net benefit that I’ve seen is a lot of companies when they’re doing their due diligence to learn about your product will spend time saying, Hey, I wonder if they have training and what does it look like?
And how’s that going to help my team?
Adam Avramescu: [00:38:53] That’s one of the benefits of having a, an open training ecosystem.
Dave Derington: [00:38:57] Yeah. And I think that’s important and it’s really cool because people can say, Oh yeah, you’ve got training. And I was able to go look at it and use it. So we did a diagram and we correlated a closed one opportunities.
And if those accounts, and again, this is a light correlation, this isn’t causation or anything really technical, but we were able to see some lift. We were able to see that a number of counts went and looked at training before that close date. And that’s the key they had training before the close date that affected that sale.
Was it a hard. correlation now, but that’s really interesting to see that. And that was a good argument for me to say no, I would prefer that we keep our training open for this reason.
Adam Avramescu: [00:39:39] And that’s data storytelling that helps you influence and make a decision based on the data you have. I love that.
Dave Derington: [00:39:44] Yeah. Will have a harder time, arguing with you about data. And let me tell you one more story. it, now I’m looking on my screen at a. It’s basically a table and the table represents all of my learners and the curriculum that they had been enrolled in. When did that, what did they complete? What was the quiz score?
When was the last time they touched the training material? So it’s all this information. One row per course, per person. So if you took three courses with me, Adam, you would see a record for all of them. So I very early on decided this would be a really good thing to have because my customer success managers were asking me all the time, Hey, Dave did, take training and I could go into my LMS and look at that, but, Oh my goodness, it’s so much better.
If I can bring that all together and put that into an automatic report so that as training happens, this data is all there and anybody can look at it. Let me tell you a little story about this data. So again, we’re looking at this table where you can look to see who’s done training. I had an interesting case where one of my CSMs came up to me and said, Dave, we’ve got an escalation.
And the VP of this company says, Hey, we didn’t train some person. And that person is failing. They’re not doing a good job. They don’t understand what they’re trying to get done, where we’re having mistakes. So I go, All right. And I asked them to open up this screen, this report type in the name. And they said to me, ah, at that, Dave’s not there.
Is there a problem? Is there a mistake? No, absolutely not. This data is validated. We have processes. I’ve worked with everybody to make this work it’s accurate. It means that person never took training. So yeah, the plot thickens. And then that was really, that was a CYA moment where I could have been in really hot water.
Because this was a contractual obligation to train somebody. And I was able to go back to that leader and say, I’m so sorry, but this person has never taken training if they had, they would have been in my system. So that is a really, I think that’s an impactful story because here I was in interacting with leadership, even my leadership and an external leadership, and I was able to show visually, no, they haven’t taken the training.
yeah, really impactful.
Adam Avramescu: [00:41:55] When earlier, yeah. When you talked about the different levels in your data dictionary, that’s a really good example of something that you would want to be able to share with your CSMs and with other peer manager of teams. Okay. Absolutely.
Dave Derington: [00:42:05] Who took training and when did they take it or not?
and I’m going to go back to one of the slides. And again, I’m looking now at a slide that is about time to first value. So I have basically a form with two boxes. One of the boxes says. What is the average time to value for accounts where nobody has consumed any training, none at all, they’ve onboarded they’ve gone through, and somehow they got around it.
And on the right of the same screen, I have another box that says the average time to value in days, for those accounts that have consumed training. Now, what you’ll see in this case is the box of without says 135 days. Okay. It’s about four months. The box that represents those that have taken training was 36 days a little over a month.
Adam, what would that say to a leader? What would it w if I brought that diagram and that, in fact I did, if I brought that chart to my leadership, what do you think they would say,
Adam Avramescu: [00:43:02] gosh, why are we wasting so much time? With customers and onboarding when we could just be training them and getting them through onboarding quicker, they’re going to get value quicker.
And they’re probably not going to hate us leave later.
Dave Derington: [00:43:15] that’s really important if I can prove, and I could actually take this data and go back to marketing and sales and say, look, let’s argue for sales in your contract. Let’s say maybe make sale, make training for free or just included.
These kinds of things really make a difference.
Adam Avramescu: [00:43:29] Yeah, for sure. When you start to be able to tie those metrics again, to the things that the business really cares about, and you can show and tell that story of how it relates to churn or renewal or expansion. Now you’re cooking with gas.
Dave Derington: [00:43:40] Now you’re cooking with gas.
Okay. Let’s do one more. on my screen, I had showed another two boxes and these boxes were like, what do you call it? A speedometer dial.
Adam Avramescu: [00:43:49] Yeah. Dashboard style speedometer. Yeah. You know what I’m
Dave Derington: [00:43:53] talking, right? we’re looking at a net proponent. Let’s say that again. We’re looking at net promoter score NPS and in customer success.
And in SaaSA general, we look at NPS a lot. I’ve always measured NPS off of my learners from my coursework, but in aggregate, when you’re looking at all of the metrics that overall NPS, you still can go back and say, okay, if I have my data in Salesforce, and I know about my accounts and I can see my survey data from Qualtrics or, survey monkey or whatever it is, and I’m seeing.
Let’s look at what’s the difference between accounts that have not consumed training and what their overall NPS is. And those that have consumed training what their NPS is now that the chart that I’m showing here are the NPS for those that didn’t have any training at all, was the three.
Pretty low. And for those who did it was up in about nine. So that is again really important because again, it’s not hard correlation, but there is a correlation, there are saying that for the accounts that have consumed training there and happier.
Adam Avramescu: [00:44:53] Yeah. And I think that three in that nine, for those of you who are MPS aficionados, that’s probably actually a C-SAT score and not an NPS
Dave Derington: [00:45:00] score.
Yeah, I would probably about a 30 and a 90 and
Adam Avramescu: [00:45:03] the 30th, something like that. negative 20 and a positive 63, whatever you actually see in your business, obviously you’re going to have that metric. Now someone asked us afterwards, and I think this is a fair distinction to make. does this mean that you’re using NPS to measure your actual trainings?
Because that’s been shown to be an invalid way of measuring training reactions? it’s been validated to measure whether someone is loyal to your business or not. And just to be clear, what we’re not saying here is these are the NPS scores for people’s reactions to training. This is when people did or didn’t take training.
What is their NPS for your business? that’s a good distinction to make here. the next one we’ll move over to one that, we saw from Pat Durrani who, he led the, he led CEDMA the customer education management association. So going left to right on this one. Again, we have some speedometers that we’re looking at with our red, yellow, and green zones.
And the first pedometer shows the percentage of churned accounts that were trained. And that’s sitting at about 5%, so only 5% of turned accounts ever got training. what about, a little bit better accounts that are still with us, but turn dollars. So they downgraded or they’re just spending less for some reason.
that 15% of them were trained and still not great. We probably don’t like that correlation there, but now let’s flip it. The third speedometer, it looks at the percent of customers that consume training. Who then renewed at or above their ACV, their annual contract value. that’s at 91%.
That’s a really impressive correlation. So when customers took training 91% of the time, they renewed at their contract value, or they even expanded, that’s powerful. That’s what your business cares about. but the kicker, this is the one I love the fourth speedometer is. Okay. But then what percentage of renewable accounts.
We’re trained only 17%, 17% 17. Yeah, it is. It is really low. And you know what I’ll say, what I’ll say here is it’s not that much higher than the percentage of accounts returned dollars that were trained. But what we are showing here because of the sequencing of these dashboards. And again, this is data-driven storytelling that when you look at those untrained figures, and then you look at the impact of training, what you’re doing is you’re helping anticipate the question that your executive is going to ask, which is, gosh, I see the training has some sort of impact here.
How many of our accounts were actually trained? Oh, not many. Okay. Let’s get more people training. the next one, this is what I’ve talked about before actually, and it’s in my book as well, but when we released Optimus at Optimizely, we showed customer tickets that were coming in over time. And for basically the year before we released Opta, versus the reason why we started working on it, in fact, Our customer tickets were rising and rising beyond our ability to actually support them by hiring more agents.
And in fact, we probably didn’t want to hire that many more agents because that would have been really expensive and inefficient for the business. most of you who have a support team in your business probably know that efficiency is key. Now we basically plugged in the Optimus release date to this trend and showed that we released our diverse at the peak of our inbound customer tickets.
And within the month that we released it, that trend started to reverse. And in fact, it reversed so much that within eight months tickets were back to a historical low eight months later, they were at the same level in December, 2014, that they were in July, 2013. And for our fast-growing software business, that’s an incredible impact.
But of course, An executive would have the question. how do we know that correlates to our accounts also growing, we don’t want to see the tickets are falling because we also have fewer people engaging with our product. So we took that a step further. Once we were able to connect our data by measuring our enterprise customer contact rate, I E number of tickets submitted per 100 enterprise customers.
To our enterprise account growth. So on one hand you can see the number of enterprise accounts growing over time, and then you can see the contact rates still falling over time. So we can tell a story that we are still continuing to drive efficiency among that enterprise customer base. And we continue to monitor this over time by looking at our customer cohorts who did or didn’t take training.
So you could see customers who did or didn’t enroll in our Academy. Customers who did or didn’t get certified who did, or didn’t use our knowledge base. And then we looked at that against product adoption. So we used a key adoption metric, which was running experiments in our platform. And you could see that again, there’s a clear correlation between when customers did take the Academy.
Did you certification did, use our knowledge base. They were much more likely to run experiments than if they didn’t use those resources. So we can see that these factors contribute. To a successful customer. Again, we are not saying that this is causation, but the correlation is strong and the correlation is still compelling to executives.
Dave Derington: [00:50:05] And again, going back to how our data is messy. Sometimes correlation is the best we’re going to be able to do
Adam Avramescu: [00:50:10] well, that’s it? And that’s the same reason the marketing teams use a funnel. there are predictive marketing tools out there that do some sort of regression analysis for you. There are data scientists who work on, marketing, doing a regression analysis on marketing data.
You can do these things, but you’re not always going to be able to do it with the resources that you have. And we shouldn’t necessarily. Go, as far as to say we need every piece of data we ever report on to be causal, because that’s not necessarily the point now that said you can do some interesting causal things.
And one of the most available to us is AB testing. So at optimize lead, we did a ton of AB testing at Slack. We also do a lot of AB testing. And, one other thing that we do at Slack is we look at the propensity for user sophistication. So not just whether people are using our product, but. What features are they using in our product and how well, how advanced are they in terms of that usage?
and how does that reflect other activities that they may have performed? but again, you don’t need a data scientist to do a simple AB test. you might need a sample group to be able to say, Hey, I know that what I’m doing is effective. And I didn’t just test this on three people, cause that’s probably not a meaningful sample size, but once you have a big enough sample size, if you’re looking in your product, for instance, one thing that we did was we looked at, whether we would want to serve as guiders in our product.
So some of you use digital adoption platforms like Pendo or user IQ, or walk me, or what fix there are more out there. some of you do performance management or performance support tools, right within your product. We wanted to see what happened when we surface those, versus when we didn’t. And we had a goal that we were attracting too.
Now without guiders, this was our, our baseline population. There was a 13% chance that they would accomplish their goal. That’s not very high. And for anyone out there, who’s listening to us in software, who has leaders who say, we don’t need customer education because our product should be intuitive enough.
That’s exactly the type of figure that you want to be able to show to them to say, I don’t think the product is so intuitive that people are doing what they need to do on their own. Now. That was the control group for the, variation group. We actually showed them some guiders that might help them, use the product better.
And for them, 51% of them accomplish the intended goal, that was crazy. And it showed the value of us servicing customer education in product at a moment of need. And again, this isn’t going to work for everything, but when you are able to show some sort of causal connection, it just makes what you do that much more powerful.
And you can visualize that again with bar graphs. So the last activity that we do it was, we asked people to think about what their dream dashboard would be. And I used the, I told the story here that, an optimized lead, for instance, before we ever had these dashboards in place, I drew them on the whiteboard and I showed them to our business intelligence team to say, we don’t have all of our data connected yet, but this is how we’re going to want to look at it.
When I moved to Checkr, we did the same thing. when I came to Slack, we luckily had some more sophisticated measurement in, in place, but there are still going to be dashboards that we’re going to want to show. And I’m probably going to draw them on a whiteboard before I ever ended up implementing them in a system.
so our call to action to you. Dear listeners is to whiteboard your own dream dashboard. What is it that you want to measure? And how would you want to tell that story in a dashboard? How are you going to tie the work that you do to the business impact that you want to have? And that’s going to help you start taking the data you collect on day to day, your reaction data, your learning data, all of your good Kirkpatrick level one and two, and frame it in a story that’s going to be compelling to your executives.
So you can actually show the business impact of your program and ultimately prove ROI. Awesome. So Dave,
Dave Derington: [00:53:55] awesome.
Adam Avramescu: [00:53:56] What else can we do?
Dave Derington: [00:53:58] We’ve done it all. We’ve done it. We’ve done it all. No, we can’t do it anymore. We all need to sit down and build out our dashboards and think about our reports. It’s a lot of work.
Yes. But getting to that ROI, by being able to visualize stories is just so impactful,
Adam Avramescu: [00:54:15] but we can’t visualize until we connect our data.
Dave Derington: [00:54:18] And we can connect our data too. We have that all collected.
Adam Avramescu: [00:54:21] So we got to collect, connect and visualize.
Dave Derington: [00:54:25] Amen.
Adam Avramescu: [00:54:25] So if this helped you out and you can help us out by subscribing in your pod, catcher of choice, if you’re not listening to this in a podcast, or maybe you’re listening to this on our site, customer.education.
so whichever one of those, you didn’t do go do the other one. If you were listening to the podcast or go to our site, if you’re on the site, subscribe, and please leave us a good review too, while you’re got it.
Dave Derington: [00:54:47] All right. And finally, as we like to always close out our podcast, I am @davederington at,
Adam Avramescu: [00:54:54] I am at also on the Twitter.
Dave Derington: [00:54:58] So reach out anytime and to our audience. Thanks for joining us. Go out, educate experiment. Find your
Adam Avramescu: [00:55:05] people.