Workshop Context Driven Testverbetering

Workshop Context Driven Testverbetering Amersfoort, 31 mei 2018

Vakmanschap Meer Plezier Oprecht Kennisdeling Lokaal Persoonlijk Focus

Roadmap voor Context Driven Testverbetering https://improvement.polteq.com 4

Context Driven Testverbetering heeft een aanleiding zoals een behoefte (+) of een knelpunt (-), krijgt doelstellingen mee om deze op te lossen, binnen een gegeven aandachtsgebied, met een aanpak, die zich laat sturen door de context, als basis voor een onderzoek, dat leidt tot een rapport, met verbetervoorstellen, die worden geïmplementeerd en waarvan het effect wordt geëvalueerd

Roadmap Context Driven Testverbetering Start Aanleiding voor een voorzien van Doelstellingen die worden geëvalueerd tegen de CDTV initiatief gericht op een Aandachtsgebied gestuurd door de Context leidt tot een Aanpak voor een Onderzoek dat Verbetervoorstellen oplevert voor de Implementatie die leidt tot een Effect

We beginnen bij de hoofdvraag Hoe kan Testservices bijdragen aan de doelstellingen van de afdeling? Hoe kunnen we het testen op een hoger plan brengen in de agile teams? Testen duurt te lang, hoe kan dat sneller? Waar en hoe kunnen we testen verbeteren in ons ontwikkelproces? We willen testen verbeteren, maar waar moeten we beginnen? Kunnen jullie bevestigen dat deze testgroep inderdaad erg goed is?

Wat is uw hoofdvraag?

Wanneer zijn we succesvol? Op basis van het antwoord wil men visie ontwikkelen op testen en op de services van Testservices Minder fouten in productie Lagere testkosten Verbetervoorstellen in logische volgorde Inzicht in vergelijking met andere organisaties

Wanneer zijn we bij u succesvol?

Wat is het aandachtsgebied? Testen bij de afdelingen Logistiek, Kantoorautomatisering en Life cycle management Zes agile teams Testautomatisering Testen van mobile apps Automatisch testen van mobile apps CI/CD 13

Scope One or more test levels unit, system, integration, One or more test types functional, performance, security, Project, department, product group, company, One or more agile teams Center of Excellence Specific testing aspects performance test test automation mobile testing.

Wat is de context? Logistiek: zeer hoog risicoprofiel Kantoorautomatisering: agile, testautomatisering Life cycle management: devops 16

Not on this list V-model Prototyping Rapid Software Test Management Lean (Acceptance) Test Driven Development TMap (Next) Wat is uw context?

Cloud Mobile SOA Wat is uw context?

INTERMEZZO: 7000 testers, of 9000? Laatste telling 26000... Out-ofcomfortzone TPI, een 10 jarig jubileum bij Huawei, China.

TPI original is best streng 23

Beijing 2011 Upgrade naar TPI Next Complexiteit MBT Outsourcing Requirements Agile Terugkerende thema s 25

Initial Controlled Efficient Optimizing 1 Stakeholder commitment A B B C F H H K M M 2 Degree of involvement A B C E H H J L L 3 Test strategy A A B E F F H K L 4 Test organization A D D E I I J J K L L 5 Communication B C C D F F J M M 6 Reporting A C C F G G K K 7 Test process management A A B B G H J K M 8 Estimating and planning B B C C G H I I K L L 9 Metrics C C D G H H I K K 10 Defect management A A B D F F H J K L L 11 Testware Management B B D E I I J L L L 12 Methodology practice C D E F H J J M M 13 Tester professionalism D D E E G G I I K K M 14 Test case design A A E F I I J K K M TPI Next transparanter, motiveert meer 15 Test tools E E E F G G I L M M 26 16 Test environment C D D E G H J J L M M

Xi an 2012 Te goed! Kan het wat minder? 27

Key areas Initial Controlled Efficient Optimizing 1 Stakeholder commitment A B B C F H H K M M 2 Degree of involvement A B C E H H J L L 3 Test strategy A A B E F F H K L 4 Test organization A D D E I I J J K L L 5 Communication B C C D F F J M M 6 Reporting A C C F G G K K 7 Test process management A A B B G H J K M 8 Estimating and planning B B C C G H I I K L L 9 Metrics C C D G H H I K K 10 Defect management A A B D F F H J K L L 11 Testware management B B D E I I J L L L 12 Methodology practice C D E F H J J M M 13 Tester professionalism D D E E G G I I K K M Context: testen was even klaar met verbeteren 14 Test case design A A E F I I J K K M 15 Test tools E E E F G G I L M M 16 Test environment C D D E G H J J L M M

29 42, or: what happens in Xi an, stays in Xi an...

Chengdu 2014, out of comfortzone Eerste keer: TPI alleen Agile TPI (TI4Agile) Michael Bolton 30

TI4Agile kijkt naar Testen en Agile (samen)werken

Hangzhou 2017 Waar is TPI gebleven? I4agile 33

Wat is de aanpak? De aanpak van het assessment wordt gedifferentieerd per domein vastgesteld. Testen of breder? Training? Staged of Continuous? Gelinkt aan SPI of CMMi? Formeel of informeel? Schaalbaarheid? Studie? Zelf of extern? 34 Verbetersuggesties? Budget? Top down? Bottom up? Interviews? Groepsessies? Mee testen? Meetings bijwonen?

Wat voor aanpak past bij uw organisatie?

Kies / combiner wat past en bijdraagt Algemene modellen TPI Next TMMI STEP CTP GQM Specifieke focus TI4Agile TI4Automation CTPI SFAI-web Belbin Meer details na de pauze! Heuristieken Exploring Test along Heuristieken - Discussies - Bevragen Impliciete kennis Experience based Mindmapping Brainstormsessies Out-of-the-Box Good Practices

Het onderzoek: een beeld ontstaat Praatplaatjes Leidende thema s Basis voor testbeleid bouwt op Testafdeling in zijn kracht zetten Bijsturen en inzoomen 38

Wat is het resultaat? Management presentatie Testservices presentatie Ondersteunend rapport Concrete voorstellen voor actie afgestemd op doelgroep 40

Intermezzo: Ervaringen met implementatie van Context Driven Testverbetering 42

Testverbetering take-1: Top-down Verbeteringfase Actie Context Effect Voordoen Test strategie 1.0 V model Meedoen Trainen Management draagt uit Hulp aanbieden Geen Zelf doen Reviewen hulpvraag van testers Testen stond nu wel op de kaart! Q er in reviewen Papieren tijger

Testverbetering take-2: Bottom up Verbeteringfase Actie Context Effect Voordoen Test strategie 2.0 Agile scrum Meedoen Trainen Organisatie zoekend: Helpen bij test Zelfsturend vs refinement zelf Zelf doen Reviewen management Indringer! Team beslist zelf hoe met testen om te gaan! Agile testen geïntroduceerd CDTI aanjager discussie over zelfsturend / zelf management Early adopters kwamen met hulpvraag over implementatie van teststrategie

Testverbetering take-3: Samen Verbeteringfase Actie Context Effect Voordoen Teststrategie 2.0 Management is Meedoen Aansluiten bij SM agile aware gilde Scrum teams nog weinig Samen met team volwassen verbeterbacklog opstellen Zelf doen Gebruik retrospective Agile teststrategie duidelijk Test (verbetering) ingebed in de Agile scrum methodiek Men kon zelf verder Management buy-in Team buy-in

CDTImplementatie - conclusies De CDTI implementatie verandert de context Verschuivende weerstandshobbel - Fase 1: team - Fase 2: management - Fase 3: weerstand weg Weet wat er anders nog speelt Pas de implementatie daarop aan Door de implementatie wordt je onderdeel van de context Dwing jezelf om scherp te blijven

Implementatie Er is meer uit het assessment gekomen dan we verwacht hadden. Bepaalde uitkomsten herkenden we en op dat gebied waren we zelf ook al kleine stapjes aan het zetten. Andere bevindingen waren nieuw voor ons en het was nuttig dat we daarop gewezen werden Nieuw testmodel - sluit beter aan bij Agile en Waterval Laagdrempeliger, beter toegankelijk voor een breed publiek Inzicht geven in wat we wel/niet doen Duidelijke scheiding van taken en verantwoordelijkheden Draagt bij aan beter voortbrengingsproces Belangrijkst van alles: de zaadjes die zijn gepland 49

Implementatie We zijn voortdurend bezig om onszelf te verbeteren en hebben ook echt de drive om zaken aan te pakken. Het rapport geeft ons steeds weer een extra duwtje in de goede richting. 50

CDTV helpt... met modellen en aanpakken 51

Waarom een model? Om.. aan te sluiten bij het testproces zoals TPI Next + TMap Next aan te sluiten bij andere verbetermodellen zoals TMMi + CMMi) te kunnen vergelijken afdelingen, de industrie, een niveau tegen een industriestandaard te meten TestSpice (Duitsland) makkelijker buy-in te krijgen model geeft gezag te focussen op een specifiek gebied TI4Automation, TI4Mobile niets te missen eigenlijk geldt dit voor alle modellen Objectiviteit Dekking

Waar bestaan modellen vaak uit? Levels Areas ontwikkeling Checkpoints

Een greep uit bestaande verbetermodellen Model Focus Strong in TPI Next TMMi TestSPICE Test process in traditional organizations Test process in traditional, large organizations Test process in industry oriented organizations Focus test improvement Formal test maturity measurement Link to standards and SPICE STEP Test & Evaluation Drive test improvement CTP Testing Drive test improvement GQM Software metrics Quality improvement T5 (imbus) TI4Agile (polteq) Testing in agile context Test improvement in agile context TI4Automation Automated testing (checking) Implementation of test automation TI4Mobile Mobile testing Improve mobile testing Belbin Team roles Improve team work Cloutest Testing cloud services Organizing testing in cloud context TOGA Test Outsourcing Governance Approach Improve testing in outsourcing context

CDTV modellen en aanpakken Modellen voor (test)verbetering Andere CDTV-hulpmiddelen Testen in Agile context TI4Agile Roadmap voor (test)outsourcing TOGA Testen en ontwikkelen in Agile context I4Agile Testen van cloudservices Cloutest Testen van mobile apps TI4Mobile Productrisicomodel Internet of Things IoT risico s Testautomatisering TI4Automation CDTV in een notendop Compact TI Testen als discipline TPI Next Bovenstaande modellen en hulpmiddelen zijn ontwikkeld door 55 Polteq, behalve TPI next dat een een merk is van Sogeti.

TPI Next Continuous representation Focus on test process Covers entire test process Improvement sequence Improvement suggestions TMap Next terminology Book (e-book) Within the context of software development process

Nieuwe modellen Waarom hebben we die gemaakt? Context! Welke zijn er? TI4Agile TI4Automation TI4Mobile I4Agile - Uitbreiding met development - Uitbreiding met projectmanagement - TODO: business (analysis) integreren in bestaande delen

TI4Agile Testing in Agile context Consistent with Scrum Many suggested practices Covers agile team aspects More strict on testing skills https://improvement.polteq.com/en/ti4agile/ TI4Agile has been developed by Polteq

TI4Automation Improvement of test automation Goal driven Not tool driven TI4Automation roadmap Reference model Focus: added value of test automation TI4Autmation has been developed by Polteq https://improvement.polteq.com/en/ti4automation/

TI4Mobile Basics Improvement of testing mobile apps Mobile awareness Test process in mobile context Special topics: Mobile device lab Device analytics API testing https://improvement.polteq.com/en/ti4mobile/ Advanced Expert TI4Mobile has been developed by Polteq

I4Agile Agile https://improvement.polteq.com/en/i4agile/ Testing Development Project management

CDTV helpt... bij de introductie van Agile 67

Transitioning to agile Process Waterfall Development Transition Agile Development Measure of Success Conformance to plan Response to change, working code Management Culture Command & control Leadership, self managing teams Requirements & Design Coding & Implementation Test & Quality Assurance Planning & Scheduling Big & upfront documentation Code all features in parallel. Test later Big, planned / test late detailed / fixed scope, estimate time & resources Continuous, just in time elaboration Code & unit test, deliver serially Continuous & concurrent testing starts early Two-level plan / fix date, estimate scope (Scaling Software Agility: Best Practices for large Organisations DeanLeffingwell 2007)

Structured Not structured Evolution of Agile testing Pioneering Agile Forming. Norming Performing Structured Time

Agile testing levels Agile testing maturity levels Forming Norming Performing Set the basis and the first steps towards working in an Agile manner Adopt a process that facilitates the Agile view on working Continuously improve the way you work by living the Agile way

Assessment model Key area Forming Norming Performing 1 Stakeholder commitment 1 2 3 4 1 2 3 4 1 2 3 2 Planning & Estimation 1 2 3 4 1 2 3 4 1 2 3 4 3 People 1 2 3 4 1 2 3 4 1 2 3 4 Interaction 1 2 3 4 1 2 3 4 1 2 3 4 5 Teamwork 1 2 3 1 2 3 4 1 2 3 4 6 Environments 1 2 3 1 2 3 1 2 3 4 7 Test process 1 2 3 1 2 3 4 1 2 3 8 Test management 1 2 3 4 1 2 3 1 2 3 9 Test profession 1 2 3 4 1 2 3 4 1 2 3 10 Test automation 1 2 3 1 2 3 4 1 2 3 4 11 Regression & E2E testing 1 2 3 4 1 2 3 1 2 3 4 12 Defect management 1 2 3 1 2 3 4 1 2 3

Stakeholder commitment Recognise the value of a team and of each role in the team Create an environment in which the team can work effectively Deliver acceptance criteria Participate in acceptance Prioritise backlog, risks and defects Deliver context for user stories Allow freedom of choice Facilitate Agile/SCRUM

TI4Agile Stakeholder commitment Forming Norming Performing Stakeholders start projects Stakeholders help projects Stakeholders participate in projects

Level of commitment The path to commitment Institutionalization Positive perception Adoption Internalization Understanding Contact Awareness

People

TI4Agile People Forming Norming Performing People are motivated People embrace changes in the product People initiate process changes

People Feedback State something positive related to the subject State your criticism objectively Don t use the word but State the effect Suggest an improvement Thank you for delivering a lot of information in the meeting. I noticed that you were talking a lot, this provided me little room to give my opinion. You might want to ask others for their input in the future.

People T-shaping Know the context Know your own strengths Assist others Leave your comfort zone 78

Stakeholder commitment Forming 1. The principal stakeholder is defined and known by the team(s) 2. Stakeholders deliver the committed resources 3. Stakeholders actively acquire information on the progress of the project 4. Stakeholders are willing to adapt their way of working to the (test) process

Stakeholder commitment Norming 1 The stakeholders provide a mandated representative (PO) to the team 2 The stakeholders define business value and provide a prioritized product backlog 3 The stakeholders define acceptance criteria for the items on the product backlog 4 Stakeholders attend the review meeting

Stakeholder commitment Performing 1 The product owner is actively participating in the team activities 2 Stakeholders trust and value the team(s) 3 Stakeholders share responsibility for the quality of the product

People Forming 1. People are well trained and/or experienced in their functions 2. People are willing to put in extra effort when needed (commitment) 3. People can explain their value in the project context 4. People take full responsibility for their work

People Norming 1 People understand project (management) terminology and know the purpose of the different meetings 2 Scrum master keeps track of the process 3 Scrum master removes roadblocks outside the team 4 People have a positive attitude towards change

People Performing 1 People proactively provide feedback 2 People know how to handle feedback and use the feedback to improve 3 People are able to help with tasks outside their main area of expertise (T-shaping)

Assessment model Key area Forming Norming Performing 1 Stakeholder commitment 1 2 3 4 1 2 3 4 1 2 3 2 Planning & Estimation 1 2 3 4 1 2 3 4 1 2 3 4 3 People 1 2 3 4 1 2 3 4 1 2 3 4 Interaction 1 2 3 4 1 2 3 4 1 2 3 4 5 Teamwork 1 2 3 1 2 3 4 1 2 3 4 6 Environments 1 2 3 1 2 3 1 2 3 4 7 Test process 1 2 3 1 2 3 4 1 2 3 8 Test management 1 2 3 4 1 2 3 1 2 3 9 Test profession 1 2 3 4 1 2 3 4 1 2 3 10 Test automation 1 2 3 1 2 3 4 1 2 3 4 11 Regression & E2E testing 1 2 3 4 1 2 3 1 2 3 4 12 Defect management 1 2 3 1 2 3 4 1 2 3

CDTV helpt... bij Mobile 86

TI4Mobile Improvement of testing mobile apps Mobile awareness Test process in mobile context Special topics: Mobile device lab Device analytics API testing Basics Advanced Expert TI4Mobile has been developed by Polteq

TI4Mobile maturity levels Mobile expert Mobile advanced Mobile basics

1 st Level Mobile basics Basics The level of mobile awareness across the project, people within a team and organization. This is enabling the conditions that are required for mobile testing. Mobile basics aspects Testability of the mobile app Available testing skills Mobile test environment Usage of Analytics API testing

2 nd Level Mobile advanced Advanced All the mobile perspectives are understood and used to increase the effectiveness of the test process. Mobile Advanced aspects Team shares mobile test knowledge Mobile perspectives in the Product Risk Analysis Dedicated test role in a co-located development team Device lab for all types of tests Development heartbeat corresponds to market demands End-users are involved in the development cycle

3 rd level Mobile expert Expe rt Testing is so organized that it is constant capable of giving a quality feedback from unit to end-user and from backend to app for all perspectives within hours Mobile Expert aspects Multiple test layers (environments, architecture, users) Regular updates of the device usage market analysis User sessions as integrated part of the development cycles Passion or innovation time to incorporate trends

The key areas of TI4Mobile

The key areas of TI4Mobile Basics Advanced Expert Defect analysis Organizational mobile awareness Test environment Test device policy Mobile API Mobile testing skills Users & Personas Mobile analytics

Organizational mobile awareness Basics Advance d Expe rt

CDTV helpt... bij Test Automation 95

TI4Automation Improvement of test automation Goal driven Not tool driven TI4Automation roadmap Reference model Focus: added value of test automation

Test Automation levels Test automation maturity levels Forming Norming Performing Set the basis and the first steps towards automated tests Facilitate integration of test automation in the test process Continuously improve test automation by applying lessons learned

Assessment model Key area Forming Norming Performing 1 Automation strategy 1 2 3 4 1 2 3 4 1 2 3 4 2 Automation architecture 1 2 3 1 2 3 4 1 2 3 4 3 Automation standards 1 2 3 1 2 3 4 1 2 3 4 Automation scripts 1 2 3 1 2 3 4 1 2 3 5 Team 1 2 3 1 2 3 4 1 2 3 6 Test environments 1 2 3 4 1 2 3 4 1 2 3 4 7 Test data 1 2 3 1 2 3 4 1 2 3 4 8 Tooling 1 2 3 4 1 2 3 4 1 2 3 9 Tool integration 1 2 3 1 2 3 4 1 2 3 10 Planning & estimation 1 2 3 1 2 3 4 1 2 3

Key areas Forming Norming Performing 1Automation strategy Scope defined Priorities defined Regression Strategy All components All test levels Selection and Test automation Strategey based Strategy End to end strategy defined agreed with stakeholders incorporated addressed scheduling defined risks addressed on ROI frequently reviewed strategy defined 2Automation architecture High level architecture available Test abstraction for each part of the architecture Tests use their respective abstractions Abstraction(s) guarantee read and write access to the test object All tests use their respective abstractions Intelligible test output available Test coverage visible in test results Reusable examples available Autonomous execution possible Continous Integration supported Performance strategy linked to automation strategy Metrics supported 3Automation standards 4Automation scripts 5Team Information on application of standards available Scripts communicate their intent Team experienced in structured testing Distributed version control used Scripts can run atomically Test automation code tested and debugged Scripts deliver consistent test output Configuration instead of magic numbers Scripts can take different data sets Team experienced in Team understands Team can development and applies the automation strategy explain rationale of automation strategy Coding convention consistently applied Scripts and linked data stored separately Team has domain knowledge Code reviews practiced Rubber ducking practiced Defacto design patterns applied Scripts can be Effort according Scripts contain no test executed in separate groups to test pyramid logic Shared understanding of added value of automated testing Mutual support in automation efforts Team continuously improves automation Refactoring Continuous integration frequently practiced practiced Common development practices applied Team continuously improves skills and knowledge Test fixtures control setup and data actions Team provides improvement suggestions for testability 6Test environments Environments available when needed Tooling has access to environments Environments enable consistent execution 7Test data Test data available Automated data selection possible Scheduled environment changes Test data enables consistent execution Environments continously available Test data linked to scripts Environments support all automated test levels Test data honors privacy regulations Environments support testing all components Test data consistent across components Environments continuously kept up-to-date Test data acceptance environment production like Acceptance environment production like Test data can be reset to original state Validity of environments frequently checked Test data automatically replicated in BPT Environments support end to end automation Test data continuously kept up-todate Environments can be duplicated and scaled Validity of test data frequently checked 8Tooling Information on tooling available Unattented execution supported Tooling manages scripts Test results can be verified Tooling used for detailed logging Tooling used for CRUD actions Tooling gathers metrics Tooling used for tracibility to requirements Tooling used for recovery Tooling supports Tooling used for end to configurable plugins end testing 9Tool integration Test reporting integrated with surrounding tools Import from external sources facilitated Export to external targets facilitated Automation uses defect management system Automation logs defects automatically Test results linked to component and environment Configuration management supported Defect life cycle automated Tool integration fully automated Automated scheduling of tests based resolved defects 10Planning & Estimation Automation activities planned and estimated Estamation substantiated Planning and estimation agreed with stakeholders Automation planned in small tasks Script review planned Sequence based on business value and risk Devations from stategy discussed and agreed Automation activities monitored Metrics and lessons learned are collected Metrics and lessons learned are used for improvement

TI4Automation Team Levels Forming Norming Performing The team applies the automation strategy The team gives mutual support in automation The team continuously improves skills and test automation

Keep the goal visible

Team development (Bruce W. Tuckman Stages of group development)

Team Forming 1. The team has sufficient knowledge and experience in structured testing 2. The team has sufficient knowledge and experience in development 3. The team understands and applies the chosen test automation strategy

Team Norming 1. The team can explain the rationale behind the chosen test automation strategy 2. The team has sufficient domain knowledge 3. Other skill groups understand the added value of the test automation efforts of the team 4. All relevant skill groups actively support each other

Team Performing 1. The team strives to continuously improve their skills and products 2. The team proactively gathers relevant information about good practices in test automation 3. The team provides constructive feedback on the testability of the test object, including improvement suggestions

CDTV helpt... met heuristiek 106

CDTV helpt... met heuristiek How do you feel? Ever felt this before? How about your work? Doing sports? How is your family? In the morning? When was that? Need to walk? At what level? Similar complaints? Before breakfast?

CDTV helpt... wrap up 110

Insanity: Doing the same thing over and over again and expecting different results. Albert Einstein

The assessment approach Volg het model Haal kennis op via Interviews Idea raising sessions Heuristics Study Walk around Test along Attend meetings Feedback and react Speak to management first Follow the problems (exploration) Ask open questions Assessment Heuristics Observe the work (Don t) Jump to conclusions The Period Of The Great Confusion

Report Management presentatie Presentatie voor de (test)groep Ondersteunend rapport Concrete voorstellen voor actie afgestemd op doelgroep Assessment results - Model based results - Unbound results - Positive points too! Recommendations - Model based improvements - Unbound improvements - Related to goals/objectives Reporting heuristics A picture is worth a 1000 words Iterate with major stakeholders Identify quick wins

Report Management presentatie Presentatie voor de (test)groep Reporting heuristics Ondersteunend rapport Assessment results - Model based results - Unbound results - Positive points too! Recommendations - Model based improvements A picture is worth a 1000 words Iterate with major stakeholders Identify quick wins - Unbound improvements - Related to goals/objectives Concrete voorstellen voor actie afgestemd op doelgroep

Continuous Improvement is better than delayed perfection. Mark Twain