Continuous Experimentation

Transcriptie

1 Continuous Experimentation Kevin Anderson Chapter Lead Data Analytics 9 October 2018

2 Why experimentation is of such value to ING

3 We live in a VUCA world Volatile Uncertain Complex Ambiguous 3

4 4

5 What we can learn from other companies

6 I believe we are the best place in the world to fail (we have plenty of practice!), and failure and invention are inseparable twins. To invent you have to experiment, and if you know in advance that it's going to work, it's not an experiment. Jeff Bezos, CEO Amazon

7 If you want to gain a competitive advantage, your firm should build an experimentation capability and master the science of conducting online tests. Ron Kohavi, Manager Analysis & Experimentation, Microsoft

8 over 100 teams run experiments over a concurrent tests running data you have data you need It's not about how much data we have, but about the data we need to prove the thing that you want to prove. Experiments are a way to create data. Lukas Vermeer, Experiments at Booking.com

9 What we can learn from other companies also from non-techs

10 Based on knowledge from Behavioral Science the Dutch tax authorities improve tax returns

11 We think it is important that financial companies experiment. AFM, Dutch Authority for the Financial Markets

12 all experimental companies share the same mindset: We don t know what will work Experiments keep us humble

13 ING should also take this path & start now

14 What we mean with continuous experimentation

15 15

16 There are two types of experiments launch optimize pilot roll out A/B testing a.k.a. split testing, bucket testing, online controlled experiments 16

17 How does an A/B test work? visitors randomly distributed A control B variant conversions version B 14% more conversions conversions 17

18 Experiments: small window into the black box between activities and results Product Owner Customer Journey Expert UX DEV engineer goals new design new features content changes activities more sales better Customer Effort Score results impact OPS engineer Data Analyst new algorithm campaigns cause & effect? less errors more conversions lower costs 18

19 Some examples

20 pilot New My ING 20

21 A/B test Mobile Approval transactions 21 squad Wookiee tribe Fraude & Security My ING

22 A/B test Logoff My ING +22% clicks

23 rollout pilots A/B test Experiment API setup manage evaluate ING.nl (open) My ING / D20 (closed) Mobile Banking App (not yet available) Branches Call 23

24 A control B variant 24

25 Experiment API setup + dashboard

26 hypothesis template 26

27 select conversion point automatic data capture tagging 27

28 Content Management Experiment API Digital Analytics 28

29 We already have our own experimentation platform but usage is low

30 Werk aan de boven en onderstroom systemisch transitiemanagement, Plan B 30

31 Why are there so few online controlled experiments? v1.1 3 NOT AWARE OF VALUE 2 NO NEED 1 NO PRIORITY VERY FEW ONLINE CONTROLLED EXPERIMENTS Not enough online traffic Not enough conversions 6 NOT RELEVANT (for some journeys) 5 NO EXPERIENCE 4 TECHNOLOGY lack of experimentation mindset / culture

32 DoAnalytics: verander de mindset

33 [naam experiment] experiment Aanleiding Het probleem dat wij proberen op te lossen is:. Dit draagt bij aan [SMART doelstelling]. We weten al dat [eerder opgedaan relevant inzicht]. Gebaseerd op [onderzoek of theorie] dat [eerder opgedaan relevant inzicht]. Gebaseerd op [onderzoek of theorie] Idee Experiment Meten Wij geloven dat [het idee]. We verwachten dat klanten daardoor [effect op klantgedrag], omdat [waarom]. Om te toetsen of wat wij geloven ook waar is, gaan we [experimentvorm / onderzoeksmethode] Daarbij gaan we meten: [concrete metric, denk ook aan negatieve effecten]. De hypothese is waar als: Vervolg Hypothese Als [ik dit doe], dan zorgt dat voor [gedragsverandering], onder [deze groep], [deze omstandigheden], omdat [waarom], en dat leidt tot [verbetering in KPI]. Als de hypothese waar is, dan [beschrijf de vervolgstappen] Als de hypothese niet waar is, dan [beschrijf de vervolgstappen] 33

34 34

35 35

36 effect 36

37 37

38

39

40 Tips Creëer urgentie en noodzaak Maak impact binnen belangrijke projecten Maak het platform zo eenvoudig mogelijk Train en help collega s starten Deel de successen 40

41 Thank Kevin Anderson