It’s been a month now since the UCL Centre for Behaviour Change 2018 conference “Going Digital & Beyond” and I have been reflecting on a key theme that struck me as being so important and relevant in the work I do at Motivait. Mostly, it is the idea that intervention effectiveness and engagement have distinctly different sets of design and outcome criteria.

For any intervention, programme or solution to be effective in achieving its aims, it must be engaging for users (for this blog, I’ll focus on digital interventions). Whilst this is hardly breaking news, it is easy to fall into the trap of thinking user engagement and user effectiveness are the same thing and therefore only designing exclusively for one or the other.

The effectiveness of an intervention is determined by whether it positively changes desired behaviour(s) and delivers intended outcomes. Effectiveness is determined by a range of factors; from how the intervention is delivered, the population and context, target behaviour(s), the extent of relevant behaviour change techniques & psychological theory applied to the intervention content, to name but a few.

Using an example, if we take employee job-strain as a commonly experienced challenge for organisations, we may use the Job Demands-Resources model as a relevant psychological framework to understand and reduce job strain. Then, we could translate its principles into the content design of a workplace wellbeing app to help managers & employees learn how to reduce employee job-strain –let’s give it a name and call it MyWorkBeing.

While the intervention’s content has been designed using relevant theory, it does not necessarily mean that this alone will attract users and sustain their engagement to the point where it is actually achieving its outcomes. In our made-up example MyWorkBeing app, it needs to be engaging enough, for long enough, to help employees’ change behaviours associated with job-strain, in the long-term, not just for a week.

There are multiple factors that influence engagement with interventions. Aside from variations in individual differences (e.g. motivation), these include the extent to which persuasive communications is used to attract users, to usability and UX, to specific engagement designs such as gamification elements and how specified they are in behaviour change techniques terms (i.e. the ‘active ingredients’ from behavioural science that regulate changes in behaviour).

So thinking about our MyWorkBeing app, it could include features such as onboarding tutorials to help users understand the apps usefulness, self-help education videos such as mindfulness, goal-setting, feedback and monitoring to cope with personal stressors, to the use of crowd-creation and crowd-rating mechanisms to generate and rate employee ideas on how the working environment could be re-designed to support wellbeing (e.g. better communal areas to incentivise social lunches, to work process changes).

When it comes to evaluating digital behaviour change interventions, it is important that engagement and effectiveness are evaluated separately without confusing their criteria. Engagement with digital interventions is considered to be measured in behavioural terms (amount, depth, frequency of use, etc.) via system analytics and in subjective experiential terms (attention, interest, vigour, satisfaction, etc.) via self-report measures.
Here we can observe what levels of engagement with the intervention, as revealed from the above metrics, is bringing about the desired changes (i.e. effectiveness criteria); this is referred to as the ‘optimal dose’.

Using our example, what levels of user engagement with the MyWorkBeing app is resulting in reductions in job-strain related behaviours (e.g. lunching at desks, email access out of hours) and outcome measures (e.g. absenteeism, wellbeing self-report measures)? We could find that those users who set and regularly monitor their goal progress and frequently watch self-help video features, to have better outcomes compared to those who only engage with the crowd-creation & crowd-rating features.

The take away message here is that engagement and effectiveness are equally important and should both be designed for carefully and evaluated with their different criteria. Simply, if users are not engaging with something, it’s not going to be effective. By the same token, if people are engaging, but the content design (or some of it) is not relevant to users, it’s also unlikely to be effective.