Usability Engineering & User Experience

Transcriptie

1 Usability Engineering & User Experience Christof van Nimwegen, 2 juni 2017 Course book Chapter 8: DATA ANALYSIS, INTERPRETATION AND PRESENTATION Measuring the User Experience (additional literature) CHAPTER 4: PERFORMANCE METRICS CHAPTER 6: SELF-REPORTED METRICS

2 Quantitative and qualitative data Quantitative data expressed as numbers Qualitative data difficult to measure sensibly as numbers, e.g. count number of words to measure dissatisfaction Quantitative analysis numerical methods to ascertain size, magnitude, amount Qualitative analysis expresses the nature of elements and is represented as themes, patterns, stories

3 Quantitative and qualitative

4 Quantitative and qualitative: After the gathering phase..

5 Simple quantitative analysis Averages Mean: add up values and divide by number of data points Median: middle value of data when ranked Mode: figure that appears most often in the data Percentages Graphical representations give overview of data Number of errors made Internet use Number of errors made Number of errors made User < once a day once a day once a week 2 or 3 times a week once a month Number of errors made User

6 Simple quantitative analysis: be careful with Researcher bias in interpretation Think of honest presentation, don t fool the reader Not scrutinizing enough (Always study the distribution) M=3.5?

7 Simple quantitative analysis: Prepare the data Data entry Outliers?

8 Analysis: Tools to support it Spreadsheet simple to use, basic graphs Statistical packages Qualitative data analysis tools Categorization and theme-based analysis Quantitative analysis of text-based data Nvivo and Atlas.ti support qualitative data analysis CAQDAS Networking Project, based at the University of Surrey (

9 Analysis: Tools to support it

10 Look!

11 Simple qualitative analysis Recurring patterns or themes Emergent from data, dependent on observation framework if used

12 Simple qualitative analysis: Video

13 Simple qualitative analysis Categorizing data Categorization scheme may be emergent or pre-specified

14 Simple qualitative analysis Looking for critical incidents Helps to focus in on key events Advantages Focuses on important issues e.g. safety critical events may bring major benefits Useful for identifying rare events that might not be picked up by other methods Disadvantages routine incidents may not be reported poor as a tool for general task analysis critical incidents often rely on memory, incidents may be distorted or even forgotten if the incident is collected long after an event.

15 Qualitative analysis: Theoretical frameworks Basing data analysis around theoretical frameworks provides further insight Three such frameworks are: Grounded Theory Distributed Cognition Activity Theory

16 Grounded Theory

17 Grounded Theory Aims to derive theory from systematic analysis of data Based on categorization approach (called here coding ) Three levels of coding Open: identify categories Axial: flesh out and link to subcategories Selective: form theoretical scheme Researchers are encouraged to draw on own theoretical backgrounds to inform analysis

18 Distributed Cognition The people, environment & artefacts are regarded as one cognitive system Used for analyzing collaborative work Focuses on information propagation & transformation

19 Activity Theory One cannot pull apart human activity from the sociocultural context within which the activity is conducted Explains human behaviour in terms of our practical activity in the world Provides a framework that focuses analysis around the concept of an activity and helps to identify tensions between the different elements of the system Two key models: one outlines what constitutes an activity ; one models the mediating role of artifacts

20 Activity Theory: mediating role of artifacts

21 Presentation: The right choice.?

22 Presentation: the right choice.?

23 Chapter 4: performance metrics

24 How sacred is it?

25 How sacred is it?

26 Performance metrics A bit farther from UX, a bit closer to UE Adequate size needed They tell what, not why 1. Task success 2. Time on task 3. Errors 4. Efficiency 5. Learnability

27 Performance metrics: Confidence intervals 95% confidence interval: % confidence interval:

28 Performance metrics: Task success Binary: yes or no, 0 or 1 Levels of success Vary e.g. user s experience in a task Vary level of optimality Issues How to define success S trikes and out? Call it at a certain point?

29 Performance metrics: Time on task Informs on efficiency Often: the faster the better (not always) What about engagement?

30 Performance metrics: Errors When to measure them? What is an error Collecting and measuring them Analyzing presenting errors Issues to consider

31 Performance metrics: Efficiency Steps: path measures

32 Casus: Destination Feedback Hypothese Destination feedback leidt tot een minder plan-gebaseerde aanpak en slechtere prestatie dan wanneer dit achterwege gelaten wordt Als bepaalde informatie geëxternaliseerd is, wordt de gebruiker verleid om GEEN plannen te maken maar om op de interface te vertrouwen. Als informatie geïnternaliseerd moet worden, wordt de gebruiker getriggerd meer na te denken voor hij iets doet.

33 Casus: Performance metrics - Destination feedback Destination feedback Slots in roster (that are possible AND available) turn green onclick Not the best/smartstes slots, simply the one that are possible NO destination feedback User must look at the limitations and think for him/herself and plan ahead? (casus = geen tentamenstof)

34 Casus: Methode experiment Procedure 43 deelnemers, 17 (40%) mannen, 26 (60%) vrouwen Leeftijd 19 tot 32 jaar Volgt, of heeft onlangs HBO of hoger gevolgd. Experiment duurde maximaal een uur, 5 euro beloning Gekeken naar: Diverse prestatie maten (tijd,zetten) Strategie Kennis Meningen

35 Casus: Time on task No difference in average time needed Time before first move (s) Time between moves (s) NO FEEDBACK DESTINATION FEEDBACK 0 NO FEEDBACK DESTINATION FEEDBACK NO FEEDBACK waits longer then DESTINATION FEEDBACK (p<.05) NO FEEDBACK: M=19.8 DESTINATION FEEDBACK: M=15.3 NO FEEDBACK takes longer between moves than DESTINATION FEEDBACk (p<.05) NO FEEDBACK: M=4.8 DESTINATION FEEDBACK: M=3.9

36 Casus: Wait a minute. No difference in average time needed between the NO FEEDBACK and DESTINATION FEEDBACK NO FEEDBACK takes more time before first move NO FEEDBACK takes more time between moves

37 Casus: Efficiency (path) Geen verschil in gemiddelde totale tijdsduur!! Superfluous moves NO FEEDBACK 4.3 DESTINATION FEEDBACK DESTINATION FEEDBACK make more superfluous moves than NO FEEDBACK (p<.05) DESTINATION FEEDBACK: M=2.5 NO FEEDBACK: M=4.3

38 Casus: Efficiency - Strategy Use of "smartest strategy" (per 5 trials) NO FEEDBACK DESTINATION FEEDBACK NO FEEDBACK uses smart strategy MORE than DESTINATION FEEDBACK (p<.05) NO FEEDBACK: M=2.4 DESTINATION FEEDBACK: M=1.5

39 Casus: At last: data!!

40 Casus: At last: data!! Meestal kwantitatieve data (anders kun je niet vergelijken): tijd, aantal, Statistische data Alleen bij (voldoende) grote getallen Nulhypothese (H0): wat je hoopt te verwerpen Alternatieve hypothese (H1): wat de conclusie is als H0 verworpen wordt Gemiddelden en standaard deviatie Statistische significantie 40

41 Performance metrics: Learnability More self-service for all of us (travel, banking products, tax, e- government, e-health, insurances,.) In casus?

42 Performance metrics: Learnability in Casus Vertical: Superfluous moves (from ideal path) NO FEEDBACK DESTINATION FEEDBACK (casus = geen tentamenstof)

43 Performance metrics: Learnability, bv. transfer?

44 Chapter 6: Self reported measures UX: Shift from more quantitative to more qualitative Accent on performance measures is a human factors legacy Delight, joy, trust, fun etc. receive more attention

45 Self reported measures Subjective data, preference data? User s perceptions When: during, post-task, post-study Social desirability bias (e.g. phone more positive than form) What? attitudes Beliefs Opinions.

46 Scales 2kg*10 = 20 but.. Code as: =.5*1? Is 1 20%, or 0%

47 Likert or Semantic Differential? Likert scale + is easier to answer (seems) - It has a negative polar and positive polar Semantic Differential + without prejudgment (the word, e.g. I enjoy using this system) - The divisions can be a problem

48 Post task ratings E.g. Ease of use After scenario Measure Expectation Measure Ask the users how easy/difficult they think each task is going to be, before doing them. Ask them how easy/difficult it actually was immediately after each task.

49 During task ratings (concurrent)? More honest?

50 Post session ratings After the complete range of interactions Lab = free again Overall barometer Aggregated individual task ratings System Usability Scale (SUS) <50: not acceptable 50-70: marginal >70 Acceptable

51 Post session ratings: SUS

52 Post session ratings: SUS

53 Post session ratings Computer system Usability Questionnaire (QSUQ) 19 statements, 5-point likert Questionnaire for User Interface Satisfaction (QUIS) 27 scales (10 point) Usefulness, Satisfaction and Ease of use Questionnaire (USE) 30 (7-point) likert rating scales, in 4 categories Product Reaction Cards 118 cards, choose the ones describing your experience, then explain top 5

54 Post session ratings: Net Promoter Score Net promotor score (NPS) how likely is it that you would promote, 1-item scale starting at 0 Sauro (2010): SUS and NPS correlate r=0.61, p<0.001 Sus scores: Promotors: 82, Detractors: 67

55 Online services (live sites) Voice of the Customer (VoC) studies Comparable to postsession metrics, but typically done on live websites. Selected sample gets pop-up survey at a predefined moment E.g: WAMMI (specifically websites). Visitor experience is measured by asking visitors to the website to compare their expectations against what they actually find on the website. WAMMI compares visitor-satisfaction for the site being evaluated with values from a reference database, American Customer Satisfaction Index Opinion lab

56 Online services (live sites) Opinionlab Usabilla Issues: Only few questions possible Self-selecting

57 Opdracht 3A

58 Opdracht 3 1. Bedenk onderzoeksvraag: een vergelijking 2. Maak materiaal - 2 webpaginas, A en B: HTML 3. Host dit op eigen serverruimte 4. Richt Google Analytics en Hotjar in 5. Modificeer HTML met GA & Hotjar code snippets 6. Zoek 30 proefpersonen 7. Start experiment 8. Verzamel, preparer en analyseer data: SPSS 9. Rapportage

59 Opdracht 3 - onderzoeksvraag (voorbeeld) Is er een verschil in leessnelheid tussen online tekst met een gele en een grijze achtergrond?

60 Opdracht 3 - Materiaal (Voorbeeld) Geel Grijs A B

61 Opdracht 3 wat gaan we verzamelen We gaan data verzamelen: 1. HTML/PHP - Timestamps 2. Met Clicky - Timestamps - Browser - Geolocatie 3. Hotjar - Videos

62 Opdracht 3 - Server inrichten

63 Opdracht 3 Clicky en Hotjar inrichten

64 Opdracht 3 - Modificeer HTML

65 Opdracht 3 zoek proefkonijnen 30 (of meer) deelnemers per team die een paar minuten willen besteden aan A of B (maar dat weten ze niet) Zorg standaard mail met instructies, en verwijzing naar de URL (een mail voor versie A en een mail voor versie B).

66 Opdracht 3 - wachten.

67 Opdracht 3 Verzamel, prepareer, analyseer

68 Opdracht 3 - rapporteer

69 Vragen?