A/B tests for launcher and chat window (beta)

Which launcher label brings more conversations? Does a side panel work better than a floating window? With A/B tests you show different visitors different variants and measure which one performs better.

A/B tests are marked as Beta. You will find them in the side menu under Setup → A/B tests, address /app/ab. They apply to the whole workspace; each test belongs to one project. Creating and changing tests requires the right to change the project.

What you can test

Under What is tested? you choose Launcher (button), Chat window or both. Per variant you can change:

• Colours – primary colour, accent, background, bubble surface, text, secondary text, lines, plus corner radius and font.
• Launcher – shape, size, size on phones, label, second line, display, icon.
• Style – all switches from Chat window, such as window size, side panel, background or animation.
• Texts – label and second line of the launcher (launcher.text, launcher.subtext), greeting and entry point of the home screen (home.greeting, home.start, home.start_sub), title and second line in the header (header.title, header.subtitle_online). 300 characters at most per text.

Whatever a variant does not change stays as set in the project.

Creating a test

Under /app/ab/new:

• Name (120 characters at most) and Hypothesis (1000 characters at most) – what you expect and why.
• Variants – 2 to 5, A to E. A is always the control, that is, your project unchanged. Variant names 60 characters at most.
• Weights – share per variant from 0 to 100, adding up to 100. A must be greater than 0.
• Participating visitors – 1 to 100 % of your visitors.
• Device – all, desktop only or mobile only.
• Languages – up to 20; empty means all.
• Pages – up to 20 patterns each for “only on” and “not on”, 200 characters at most each. “Not” beats “only”.
• Primary metric – what success is measured by:
• open – share of visitors who open the chat. The direct effect of the launcher.
• msg – share of visitors who write a message.
• lead – share of visitors who leave contact details (email).
• goal – share of visitors who reach your own goal (see below).
• csat – share of positive ratings (thumbs up).
• Smallest effect to detect – 1 to 100 %, default 10 %. From this your account calculates the required sample.
• End – an end date or a number of visitors.
• End automatically on a clear result – not before 7 days and only once the required sample is reached.
• Apply the winner to the project brand automatically – the winning variant becomes your project's setting.

Variants can only be changed in a Draft. At most one test runs per project at a time (running or paused). After the start, the test reaches your visitors within a minute.

Who sees which variant

Assignment happens on the server and is fixed for each visitor: anyone who has seen variant B once sees B again on their next visit.

Pages whose snippet carries an own brand, that is, a data-rc-brand with more than colour, radius, and shape and size of the launcher, do not take part in the test. See Colours, font and theme. If consent in the project is set to required, nothing is counted without the visitor's agreement.

Viewing a specific variant

Add ?rc_ab=B to the address of a page where the widget runs to see variant B. Nothing is counted when you do this.

https://www.example.com/preise?rc_ab=B

Reporting your own goal

For the primary metric goal, you report the goal from your page, for example on the thank-you page after a purchase:

ReactiveChat('abGoal');

You can query which variant a visitor sees like this, for example to pass it on to your own analytics tool:

var v = ReactiveChat('abVariant');   // { test, variant, qa } or null

ReactiveChat('on', 'ab', function (e) {
  // e.test, e.variant, e.qa
});

If your code might run before the widget has loaded, put the queue line from Technical integration in front of it.

Results

Under /app/ab/{id}/results you see per variant:

• the rate of the primary metric and of all other metrics,
• the difference from control A with a 95 % confidence interval,
• p-values, with Bonferroni correction when there are several variants,
• the probability of being the best variant,
• a check whether the distribution of visitors matches the weights,
• the required sample at 80 % power.

The figures can be downloaded as CSV (/app/ab/{id}/csv).

If a test collects no data

• Is it running? Only one test per project can run.
• Does the page match? Check device, language and page patterns.
• Own brand in the snippet? Then the page does not take part.
• Consent set to “required”? Without agreement nothing is counted.
• Called with ?rc_ab=…? These calls deliberately do not count.