Active Insight Methodology



Organizations are increasingly dependent on their digital channels to drive sales and customer acquisition. This means it is more important than ever to ensure customers have a seamless and trouble-free experience when trying to transact on web or mobile. The result of not delivering an outstanding digital experience has a major impact on the bottom-line for modern companies. However, how do you ensure you are delivering the best possible digital experience to your customers at all times?

UserReplay uses “Active Insight,” a service that ensures its CEM technology continues to drive value on a continuous basis. This simple, repeatable methodology will ensure any organization can drive increased revenues and conversions through their digital channels using their CEM technology.


The Active Insight Methodology has 5 Key Steps:

  1. Discover what the key obstacles to conversion are by replaying the actual experiences your customers have.
  2. Analyze the data captured about customers who experience the issues identified in the Discover step then quantify and prioritize the issues that need to be addressed.
  3. Rectify issues based on the prioritization developed in the Analyze step. Session replay is again important here to reproduce issues quickly and efficiently.
  4. Track the online experience data to verify issues have been rectified and to alert if these issues re-occur.
  5. Share the results and wins that are achieved through application of your CEM technology within your organization – especially with executives and senior management.


Active Insight Diagram

Discover Analyze Rectify Track Share
Discover the issues that are stopping customers from being able to successfully complete transactions.

The Key Aspects of Customer Experiences

There are many aspects to the experiences your customers have in your digital channels that define whether customers transact with you. Issues in any of these aspects can cause a downgrade in customer experience and these issues can prevent conversion. The key aspects include:

  • Usability
  • Performance
  • Technical Stability of the Platform
  • Application Issues
  • Clarity of Navigation

Why Implement the Active Insight Approach

If you find even marginal improvements in the above areas, this can ultimately lead to big improvements in conversion. In fact, it is proven that discovering just a small number of customer experience issues can potentially deliver a huge impact on the bottom line. Our customers achieve major increases in conversion and revenue when they start with UserReplay.

But equally importantly by embedding the “Active Insight” approach into their operations, either on their own or with the support of UserReplay analysts; they achieve continuous improvement in their performance. They also ensure that problems introduced by website updates or browser upgrades are immediately identified before sales are lost.

The Active Insight Methodology in Detail

The five steps of the Active Insight Methodology are outlined below:

1. Discover

One of the biggest challenges that face e-commerce professionals is identifying the real reasons why customers abandon during online processes such as checkout. Existing web analytics technologies quite often provide a picture of where the key abandonment points are in a process but do not provide the visibility into individual customer experiences that will answer the very important why question. This missing piece of the jigsaw can be provided by a CEM technology such as UserReplay.

UserReplay provides the capability to replay customer sessions in 3 different ways (visual, narrative and technical) to help pinpoint the reasons why customers abandon. UserReplay also provides an analytics layer, which ensures that users can be directed quickly and efficiently to the sessions that are most relevant to replay.

The analytics within UserReplay is powered by the concept of flagged events. These flagged events are essentially counters that fire when a particular condition occurs within a user session. Flagged events can capture a variety of activity or conditions within user sessions such as:

  • Steps within a particular online process
  • Particular pages that are visited e.g. “Contact Us” page
  • Error messages that are seen by the user
  • Pages that take longer than a defined time period to load
  • Repeated activity within a session e.g. multiple attempts to pay or log-in

In order to discover the reasons for abandonment from a process such as a checkout process, the first step is to create flagged events in UserReplay for the key mandatory steps within the process. These flagged events can then be organized into process funnels that quickly visualize the process and what the key abandonment points are.

Process Funnels within UserReplay are designed so that the user is able to easily drill-down to the sessions that abandon at particular points in the funnel. Normally, it is most relevant to first look at users who abandon towards the end of the funnel.

This is the “low hanging fruit” of visitors who have shown maximum intent to complete the process but have not been able to. For example, a visitor may click on the “Complete Order” button but never receive an order confirmation page. Replaying and analyzing some of these user sessions will begin to identify some of the reasons why customers are abandoning.

Our customers achieve major increases in conversion and revenue when they start with UserReplay.

Monitoring Ratios and Identifying Problems

Once flagged events have been set up they can also be used to monitor Customer Experience on an on-going basis. Monitoring ratios can be particularly useful. For example if the ratio of users who progress from one step of the checkout journey to another changes, it is important to identify this, look for commonalities between the new abandoners and replay some of the impacted journeys.

Sometimes this can be very simple but very impactful, for example this approach led to one of our customers identifying that customers could not check out using iOS8 on iPads almost as soon as it had been released.

UserReplay also means it is simple to investigate website problems reported via customer service, voice of customer tools, and social media. Often these issues are suspected rather than definite problems. For example, it might be believed that baskets are occasionally emptying themselves, or form fields are sometimes invalidating correct entries, or a new UI design is confusing a subset of customers.

Customers might occasionally be complaining about double payments, or payments being taken without the order being processed. Such obscure problems may be impossible to find in testing, and the log files often do not help either. In these cases the IT department is liable to blame user error, and the customer services department is likely to blame IT. Often nothing gets done.

With UserReplay’s 100% capture these problem journeys can be immediately identified and diagnosed with no need to replicate them in testing. In our experience if you have a “hunch” that there is a problem, even if it is hard to replicate, it usually is real.

Once some abandonment reasons have been identified, you can move onto the next step of the methodology which is to analyze how often these happen within the general population of users and what business impact they have

Problem journeys can be immediately identified and diagnosed with no need to replicate them in testing.

2. Analyze

The next step in the methodology is to understand how serious the issues that you discover actually are and how much impact they are having on the business. To do this, we use UserReplay’s flexible reporting capability to query the rich set of data we capture about the user.

There are some key questions that you can answer with UserReplay at this point:

  1. What is the normal conversion rate for users who reach the conversion step affected by the issue? E.g. if the user experiences the issue at the payment step, which is step 4 of a checkout process, what is the conversion rate from step 4 to the final conversion step?
  2. How many users were affected by the issue in a given time period?
  3. How many users who were affected by the issue also converted?
  4. How many people should have converted for the users affected by the issue?
  5. How much are these lost opportunities worth in terms of annual revenue and margin?

Let us drill down into how we would do this in a bit more detail:

What is the normal conversion rate for users who reach the process step affected by the issue?

If you imagine this in terms of a scientific experiment, the segment of users that reach the process step at which our issue occurs is known as our “Control Group”. This control group is the wider population of users against which our issue is going to be measured.

The first data we need to acquire from UserReplay should be to find out for a given period (normally a typical day) how many user sessions are in the “Control Group”. This is easy to do in UserReplay as this value is readily available via the “Flagged Events Summary” in the UserReplay portal. This, of course, is dependent on having created the flagged event for process steps as per the Discover step.

Retrieve the number of visitor sessions affected by the issue by using the Flagged Events Summary report.

Next, we need to understand how many of these users went on to complete the process and hence convert. This data is also easy to retrieve from UserReplay via the Reporting capability. The report required would need to retrieve the data set of all sessions in the period we are focused on that reached step A and the conversion step within their session.

Normal Conversion Rate = (Total number of sessions where user proceeds to the checkout page and then converts, in this example 51406 per day) / (Total number of sessions where user converts, in this example 65223 per day) * 100 = 79%

How many users did the issue affect?

Having understood our control group, we now need to quantify the group of users who were affected by the discovered issue. We can call this our “Issue Group”. Again, it is simple to retrieve the number of visitor sessions affected by the issue by using the Flagged Events Summary report within the UserReplay portal.

For increased accuracy, we could quantify the number of users who experience the issue who actually pass through the process step we based the Control Group on. Again, using UserReplay this is straightforward using the flexible reporting capability.

In the case of our example UserReplay advised that 3093 users per day were found to attempt to proceed to the checkout page but instead ended up immediately back at the home page. The problem was real. How many uses converted in the Issue Group?

We now need to find out how many of the users affected by the issue actually managed to complete their transaction. This is easy to find out in UserReplay by running a report that returns the numbers of users who trigger the flagged event for the issue and who also see the conversion step in the same session.

In our example 1256 users from the 3093 experiencing the problem converted anyway. In fact one heroic user kept trying to check out repeatedly and eventually converted despite being bounced back to the home page no less than 27 times! Overall the conversion rate for those impacted by the issue was 40% rather than the usual 79%.

How many lost conversion opportunities were there for the issue?

We know the normal conversion rate for the Control Group. This is the conversion rate that our users affected by the issue SHOULD have generated. We can apply this conversion rate to the number of users affected by the issue to find out how many of those users should have converted. It is then a simple equation to find out the number of lost opportunities in a single time period (e.g. day):

(Issue Users * Control Group Conversion Rate) – (Actual Conversions from Issue Group)

(On our example 3093 per day * 79% = 2443) – 1256 = 1187 per day

What is the yearly opportunity cost for resolving the issue?

We now have all the raw data we need to monetize our issue. We know the daily lost conversion opportunities so we can now extrapolate this across a full year and put a revenue figure against it.

The required calculation is as follows:

(Daily Lost Opportunities) * (Average Order Value) * 365

At this point, it may be appropriate to apply a margin percentage rate to the lost revenue number so that the impact on profit of the issue can be calculated.

On our example the cost was 1187 per day * $50 average basket size * 45% gross margin = $26,707 per day or $9.75m per year, quite a financial hit for a bug that allegedly did not exist.

As you can see, the issue group is a relatively small group of the overall number of users who reach the payment step. However, when the analysis is completed we can see this issue could be worth nearly $10m in lost revenue.

The business analysis process above is a totally repeatable process that can be used to quantify the impact of issues in your digital channels

3. Rectify

Of course, the revenue opportunities outlined in the Analyze step cannot be brought to fruition without the issues being fixed and rectified. The Analyze step provides the data for the development team to be able to prioritize issues based on their real impact to the business. They can then take steps to rectify the most important issues first.

UserReplay can also help with rectification. The first step is often to look for commonalities in the issue group. For example the issue group may all be on a particular device or operating system. Visually replaying these journeys may rapidly identify the issue and how to fix it. For example, conversion may have dropped because the “next” button dropped below the fold line for a particular mobile device. This would be immediately obvious on visual replay.

In the case of our example the common feature was that the same server in the server farm had served the entire issue group. By looking at the technical details of the request and response headers from the broken journeys it was possible to identify what was going wrong. In our example the server operating system had been upgraded inconsistently to the others in the farm – a simple but very financially rewarding fix.

4. Track

Once issues have been rectified, it is important to monitor and trend these issues over time. You can then see if the issue really has been fixed and also identify if it re-occurs.

The Real-Time Dashboard capability within UserReplay is a great way to do this. This provides the capability to create dashboard widgets that show a trend of flagged events over time. Further to this, alerts can be created that will trigger on thresholds being breached for the flagged event.

In this case, no requests to checkout should direct to the home page, so the threshold is zero.

There then can be assurance that this or a similar problem can never go unnoticed again.

5. Share

It is important that the gains achieved through the use of this methodology are communicated within the organization. There are a number of benefits for doing this:

  1. Provides visibility of common issues so that they can be designed out of the application and tested for in future releases.
  2. Providing senior management and executives the visibility of improvements being made, along with their associated revenue gains, means they will see the clear business case for on-going and further investment in this type of analysis.
  3. Provide clear, undisputed evidence to e-commerce platform suppliers that can ensure their support and development efforts are focused towards improvements that can benefit your company’s bottom line. This is a win-win for the supplier and your company.
  4. Personal gains for the staff performing this analysis as they can clearly demonstrate to their managers that they are making a difference to the bottom line of the company.

Having got the Active Insight process running don’t forget the other benefits of UserReplay: Enhancing customer service, resolving customer disputes, fixing technical problems and combatting fraud.


Utilizing a simple but very effective methodology can maximize the value of session replay/customer experience management technology within an organization. This value is generated through regular discovery and resolution of issues that are impacting conversion.

Not all of these issues will necessary be affecting large amounts of customers but may have a disparately high effect on revenues. By focusing on those issues that have real revenue impact you can ensure the marginal gains generated by these issues end up having a huge impact on the bottom line.

Get a live demo!