Este post también está en español.
Preparing my talk “Full loop analytics framework” for ILA 19 in Medellin, which as the title implies is focused on analytics, I’ve been reading a lot on the subject.
Here’s a small compendium of a few things that I found interesting.
Context is key
In his article “Observations on Data, Metrics and Goals”, Dan Hill says “Goals are three things: a metric, a value and a moment in time. When a goal feels wrong, it´s usually missing one of these”. For instance, a goal could be “increase sales by 10% in the next 3 months”.
I would add a fourth component: context. This is key in order make sense of the goal. Following the example above, is 10% realistic? Is it too much or too little? If the growth of sales month to month, for the last 3 months has been 1%, then increasing it by 10% is not realistic. Having context for a metric helps you see the big picture and helps you have the right conversations.
Hope vs reality
He also says: “Explain variance to forecast, not variance to hope”. And I thought it was a great observation, especially when considering the previous point.
The problem is, even though forecasts should be based on historic trends and other business/market data, often times goals are based on hope rather than forecast. Following the previous example, it may be that the 10% increment that was set as a goal may effectively be in the forecast, but that said forecast was created based more hope than reality.
And if we take into account the context component of the goal, we can see that this 10% is not only not realistic in terms of forecast, but also not realistic in terms of reality. Again, adding context to metrics helps to understand (and explain) results as well as to have the right conversations.
…
From Jack Martin’s presentation (former Head of Product at Roomgo) “How to identify relevant KPIs” for Product School I also found a few key points I wanted to highlight.
Funnels: focus on abandonment
When analyzing a funnel don’t just focus on those who completed the process. Rather, look at those who fell off because that’s where you’ll get the most valuable information.
The value that those users can bring include:
- Their business potential: the users that fell through already wanted to convert, so we have higher chances of helping them and higher power of influence over those who never came to our site or started the process.
- They hold valuable information: they can help us uncover problems with our existing product or flow so we can make improvements for these and future users.
A holistic view is key
Focusing on short term goals, which are usually results oriented, can lead us to making the wrong decisions.
For instance, if we have a goal to increase the CTR to product pages, and we do so by eliminating some product information from the homepage so visitors are forced to go to the product page to get the information they need.
We may increase the CTR with this tactic but at the expense of creating friction for the users, which now have a harder time finding the information they are looking for to make a purchase decision, and at the risk of them not doing the effort to find it and therefore abandoning the process. If on top of that our competition does have product information easily accessible, we have an even higher risk of losing the sale.
So we may increase the CTR in the immediate term, but since this comes at the expense of the user experience we may be hurting sales in the short to mid term.
In short, if we don’t have a holistic view we may mistake the tree for the forest, and not realize that one small improvement in a specific part of the process is hurting the entire system.
KPIs and Business questions
This point is quite obvious but often overlooked: all metrics must answer a business question, otherwise you don’t know how to interpret the KPIs.
For instance, an increase in visits to the help pages of your site, is good or bad? If we analyze the KPIs in isolation, we may think it is a bad thing because it means more people are having problems with our product.
But if we have a goal of having more people use digital channels over the call center, then an increase in visits to the help pages is a good thing, as long as the volume or length of calls decreases.
In other words, it is important to peg KPIs to business questions and not analyze them in isolation because we may arrive at the wrong conclusions.
Cohorts
When we make changes it is important to use cohorts when measuring the impact because if we use aggregate data we may the wrong results.
For instance, if we make changes to the checkout flow we should make a cohort with the existing users (those who made a purchase in the past) and another one with new users when measuring performance.
This is important, because existing users may have trouble with the new flow at first, while new ones may find it easy to interact with. So if we put both users in the same group when analysing results we may not get accurate results.
Este post también está en español.
Preparing my talk “Full loop analytics framework” for ILA 19 in Medellin, which as the title implies is focused on analytics, I’ve been reading a lot on the subject.
Here’s a small compendium of a few things that I found interesting.
Context is key
In his article “Observations on Data, Metrics and Goals”, Dan Hill says “Goals are three things: a metric, a value and a moment in time. When a goal feels wrong, it´s usually missing one of these”. For instance, a goal could be “increase sales by 10% in the next 3 months”.
I would add a fourth component: context. This is key in order make sense of the goal. Following the example above, is 10% realistic? Is it too much or too little? If the growth of sales month to month, for the last 3 months has been 1%, then increasing it by 10% is not realistic. Having context for a metric helps you see the big picture and helps you have the right conversations.
Hope vs reality
He also says: “Explain variance to forecast, not variance to hope”. And I thought it was a great observation, especially when considering the previous point.
The problem is, even though forecasts should be based on historic trends and other business/market data, often times goals are based on hope rather than forecast. Following the previous example, it may be that the 10% increment that was set as a goal may effectively be in the forecast, but that said forecast was created based more hope than reality.
And if we take into account the context component of the goal, we can see that this 10% is not only not realistic in terms of forecast, but also not realistic in terms of reality. Again, adding context to metrics helps to understand (and explain) results as well as to have the right conversations.
…
From Jack Martin’s presentation (former Head of Product at Roomgo) “How to identify relevant KPIs” for Product School I also found a few key points I wanted to highlight.
Funnels: focus on abandonment
When analyzing a funnel don’t just focus on those who completed the process. Rather, look at those who fell off because that’s where you’ll get the most valuable information.
The value that those users can bring include:
- Their business potential: the users that fell through already wanted to convert, so we have higher chances of helping them and higher power of influence over those who never came to our site or started the process.
- They hold valuable information: they can help us uncover problems with our existing product or flow so we can make improvements for these and future users.
A holistic view is key
Focusing on short term goals, which are usually results oriented, can lead us to making the wrong decisions.
For instance, if we have a goal to increase the CTR to product pages, and we do so by eliminating some product information from the homepage so visitors are forced to go to the product page to get the information they need.
We may increase the CTR with this tactic but at the expense of creating friction for the users, which now have a harder time finding the information they are looking for to make a purchase decision, and at the risk of them not doing the effort to find it and therefore abandoning the process. If on top of that our competition does have product information easily accessible, we have an even higher risk of losing the sale.
So we may increase the CTR in the immediate term, but since this comes at the expense of the user experience we may be hurting sales in the short to mid term.
In short, if we don’t have a holistic view we may mistake the tree for the forest, and not realize that one small improvement in a specific part of the process is hurting the entire system.
KPIs and Business questions
This point is quite obvious but often overlooked: all metrics must answer a business question, otherwise you don’t know how to interpret the KPIs.
For instance, an increase in visits to the help pages of your site, is good or bad? If we analyze the KPIs in isolation, we may think it is a bad thing because it means more people are having problems with our product.
But if we have a goal of having more people use digital channels over the call center, then an increase in visits to the help pages is a good thing, as long as the volume or length of calls decreases.
In other words, it is important to peg KPIs to business questions and not analyze them in isolation because we may arrive at the wrong conclusions.
Cohorts
When we make changes it is important to use cohorts when measuring the impact because if we use aggregate data we may the wrong results.
For instance, if we make changes to the checkout flow we should make a cohort with the existing users (those who made a purchase in the past) and another one with new users when measuring performance.
This is important, because existing users may have trouble with the new flow at first, while new ones may find it easy to interact with. So if we put both users in the same group when analysing results we may not get accurate results.
