Control groups

Every day, Aampe sends out thousands of different messages across your user base, learns your individual user preferences based on who responds to what, and then automatically adjusts your messaging timing, content, and copy to fit those learned preferences.

With so many moving pieces, how are you supposed to know if Aampe is showing an improvement over your old messaging strategy?

That’s where our control groups come in.

How our control group works

Aampe’s systems measure attribution by taking aspects of both switchback holdouts and synthetic controls, combining them with a statistical method called Coarsened Exact Matching (CEM).

CEM works by matching users who got an “intervention” (in our case, a message) with a user who didn’t get that same intervention but is similar to the target user in other ways.

So where a synthetic control tries to model the entire outcome of a message, our approach is to model each individual user who received the message in order to pick a comparable individual control user.

So we find a user who got a message, then we randomly pick a second user who received a message at a slightly earlier period in time. We make sure the matched users are similarly active on the app (so we don’t match an active user to one who hasn’t been on the app for a month, for example) and that the messages each user received had similar content. We then measure the behavior of both the original and the “control” users as if they had both received the original user’s message.

This allows us to not just show your performance vs. a non-messaged user but to show your actual performance alongside what it would have been if you hadn’t used Aampe:

❗️

The costs of a secondary control group

Aampe is able to support external control groups very easily, but we recommend against it for the following reasons:

• creating proper control groups is a lot of work for your team
• randomization of population allocations is often done incorrectly
• removing users for the control reduces the potential uplift for your marketing efforts

As a result, we have found that customers who choose to do their own control groups generally don't see much gain relative to the effort.

If you would like to create a control group, the implementation from Aampe's perspective is very simple.

Step 1: When sharing your contact data, add a new "__control_group" boolean field

Step 2: For any contact that you would like Aampe to ignore, set "__control_group" to True.

Aampe's models will ignore any user labelled as such.