Information Provision for Situation Awareness

Transcriptie

1 Technical Report Information Provision Indicative MOSAIC Experiment Editor: Authors: Marc de Lignie Release Date: 21 November 2009 Reference: D-CIS RR /

2 Marc de Lignie Copyright 2009 D-CIS Lab, all rights reserved Permission to copy or use solely for non-profit purposes is hereby granted subject to inclusion of this copyright notice. Contact information: Name Affiliation Address Marc de Lignie Vts Politie P.O. Box 238 Nederland NL-3970 AE Driebergen The Netherlands D-CIS Lab / TU Delft Kees van der Meer D-CIS Lab / TU Delft D-CIS Lab / TRT-NL Niek Wijngaards D-CIS Lab / TRT-NL P.O. Box 90 NL-2600 AB Delft The Netherlands P.O. Box 90, NL-2600 AB, Delft, The Netherlands P.O. Box 90, NL-2600 AB, Delft, The Netherlands Kees.vanderMeer@icis.decis.nl Masja.Kempen@icis.decis.nl Niek.Wijngaards@icis.decis.nl Recommended citation: De Lignie, M., Hu, B.B., van der Meer, K., Kempen, M. and Wijngaards, N. (2009). Information Provision : Indicative MOSAIC Experiment. D-CIS Research report series, no. D-CIS RR/ , see publications.decis.nl. Acknowledgement: The research reported here is part of the Interactive Collaborative Information Systems (ICIS) project at the D-CIS lab, supported by the Dutch Ministry of Economic Affairs, grant no: BSIK The authors wish to thank Jeroen Gerritsen, Rianne Gouman and Eddy van der Heijden (D-CIS Lab / TRT-NL) for their help in conducting the experiment. The authors wish to thank Inspectie OOV for making a realistic incident management scenario available. The authors are grateful for the participants willingness to participate in the experimentation. 2

3 Information Provision D-CIS TR/ Abstract The MOSAIC project aims at enhanced situation awareness and reduced information overload to public safety officers (police, fire brigade, medical transport) in a complex safety incident. In this first MOSAIC indicative experiment, a realistic safety incident, a ship collision with many persons and poisonous gas involved, was simulated by messages on this incident. These messages correspond to usual messages sent to the local police commander at the location of incident. In the experiment, the original message set (Set-0) was reduced in consecutive ways. In Set-1 messages on victims and events related to the safety incident were combined. In Set-3 only information pertaining to operational police tasks remained (all information on the informative police task was removed). The normal operational background task of the Control and Command Room (CCR) was simulated by a computer game. The effects of the information reduction on situation awareness and information overload were measured by a questionnaire with questions on the situation, an open evaluation, and background task performance. Eighteen police officers participated in the experiment. The results showed that removal of messages containing information that was sent before reduces the information overload that is experienced. Aggregating numbers of victims and certain types of events is a most effective way to decrease the number of messages while improving situation awareness. Aggregating geographical data by plotting was, unexpectedly not effective; training beforehand might have improved this. To conclude, reduction of the number of messages proved to be effective in improving information processing of police officers. 3

4 Marc de Lignie Contents Abstract...3 Contents Introduction Scenario & Message sets Ship-collision scenario Operational Task Message sets Hypothesis Experimental Design Ways of measurement Questionnaire: subjective data Questionnaire: objective data Other data Results & Analysis Results Analysis Discussion Conclusion Appendices...20 A Introductionary handouts for participants...21 Appendix B. First questionnaire...23 Appendix C. Final questionnaire...26 Appendix D. Debriefing handout...33 Appendix E. Ratings for open questions...34 Appendix F. Transcripts of answers to open questions

5 Information Provision D-CIS TR/ Introduction This document describes the MOSAIC indicative experiment, executed in The MOSAIC project aims at enhanced situation awareness while reducing information overload for executive public safety officers: police officers, fire brigade officers, and GHOR (medical transport) officers, by task-relevant information provision. The MOSAIC indicative experiment is part of the MOSAIC project. It is an experiment to find out whether the foreseen use of MOSAIC could be valid in practice. The MOSAIC project is a joint effort of vts Politie Nederland and D-CIS Lab. The experiment entails 18 participants who are supposed to be engaged in fighting a complex incident or crisis. They receive a large number of messages referring to that incident, while performing a background task; the background task represents the operational engagement cq. disturbance that is ubiquitous in public safety officers work. The participants are asked periodically to answer questions on the incident. Their performance of the background task is also measured. A training disaster scenario from the Dutch Inspection OOV is reused; it is analysed and adapted for MOSAIC information provision, and in this experiment played as a Microsoft PowerPoint presentation. The methodology for information relevance assessment and user modelling is described in the document Context-based Information Delivery for Mobile Users in Emergency Situations by Beibei Hu. 1 The dominant communication channel between the command & control room (CCR) and executive public safety officers is the C2000 channel. This communication channel is not investigated in the MOSAIC experiment. For our convenience a suitable and comparable communication channel was used. Our intent is to compare the actor s situational awareness with and without the MOSAIC information provision. Ideally, the users (participants) have to act upon the information (aggregated or not) they receive in a simulated environment. The effects of better information provision can then not only be measured by interviewing a participant, but also by observing the differences in actions in the simulated environment. Moreover, an ideal experiment would support the following communication and interaction modes: Oral information using C2000; in the indicative experiment this is simulated by information messages from the location event generators to the command & control room (CCR; in Dutch meldkamer ). External messages to the command & control room are aggregated and pushed to the local executive public safety officers Local executive public safety officers can read a personalized overview of all available information (pull) In the current indicative experiment, though, rather than modelling multiple devices, it is easier to convey all information to the user s Personal Assistant Liaison agent which simply presents the information on-screen. Ever more information is available on the context of the incident that could be relevant to the public safety officers: there are databases with data on events, on permissions, on cars, on buildings; there are web pages, web cams, cameras, pictures from Twitter; and so on and so forth. The current experiment addresses the comparison between receiving all information and receiving aggregated (task-based) information. So, task-based information provision is performed to prevent information overload. In view of the increasing amount of information, this is a vital issue for busy public safety officers involved in incident response! 1 Beibei Hu. Context-based Information Delivery for Mobile Users in Emergency Situations. The First and Seventeenth International Conference on User Modeling, Adaptation, and Personalization(UMAP, 2009), Doctoral Consortium, Trento, Italy, June,

6 Marc de Lignie The information needs of the public safety officers (police, fire brigade, GHOR) in an incident is represented in a personalised information bubble in the incident information space (IIS). It indicates the content of the information for the first responders and the way, in which this information is made available, e.g. by an urgent call (from a management view this is push information) or made available upon request of the first responder (from a management view: pull information). In the experiment, the messages from the CCR are retrieved and relevance is assessed according to the current information in the personal information bubble. The concepts of improved situational awareness and reduced information overload have been made operational by finding out whether end users (with a specific role/task/location): 1. Consider information provision in MOSAIC more useful than the original information and/or state that they would perform better at their task in the scenario when assisted by MOSAIC; 2. Perform better at answering questions on the scenario after the session has ended; 3. Perform at least equal to the foregoing situation at a background task when information is task specified (as derived from the game results). These measurements imply that the end users need to be divided into two groups; one group that receives the aggregated information and a second control group that receives original information. Given the current status of the MOSAIC implementation and scenario realisation, the choice is made to not have a simulated environment in which the effects of actions of participants are incorporated in the unfolding of the scenario. Such simulations lead to additional scenario complexity (handling different endings and analysing results) on the one hand and much greater participant involvement on the other hand. The current choice is to have the participants engaged in a background task which sufficiently demands their attention, and to query the participants during and after the information provision about their current understanding of the (ongoing) scenario. This first indicative experiment has the following objectives: Validation of the MOSAIC ideas on information provision, or more specific: enhanced situation awareness and reduced information overload. Refinement and enhancement of personalized information filtering. Suitability of the used scenarios to realize enhanced situation awareness, reduced information overload and the corresponding personalized information filtering. The remainder of this report describes the set-up and results of this indicative experiment. Section 2 describes the incident scenario and message sets used as conditions in the experiment, together with our main hypotheses. Section 3 describes the experimental design. Section 4 analyses the results of the experiment. The report concludes with a discussion in Section 5 and a final conclusion in Section 6. The appendices show the materials provided to the participants and additional analysis (ratings of open questions and transcripts). 6

7 Information Provision D-CIS TR/ Scenario & Message sets To measure the usefulness of the MOSAIC ideas, including the information provision part, a suitable scenario was selected. The Rotterdam Rijnmond crisis scenario was chosen, tailored to a minor degree to our needs. This scenario of the Dutch Inspection OOV 2 involves a collision between two ships. The scenario was tailored to focus on the chief police officer at the location of the incident who receives a set of messages from the command & control centre ( meldkamer ). For this scenario, there are a number of sets of messages that differ in extent of the description of objects and events of the incident. The experiment is meant to enable comparison of the situation awareness of the chief local police officer. 2.1 Ship-collision scenario In Rotterdam at the river Nieuwe Maas, a ship-collision occurs. At the location of the Boelewerf / Werfkade, a freight ship collides with a party-ship. As a result, a number of wounded people are aboard the party ship, while the containers from the freight ship are adrift in the river, blocking the shipping lane. As an extra complication, one or more damaged containers contain poisonous chemicals that evaporate, one of which reacts to moisture in the air, causing a cloud of poisonous gas (see Figure 1 for an impression). Locatie melder Plaats incident aan Boelewerf/Werfkade Uitzicht vanaf Boelewerf over Nieuwe Maas Gebaseerd op ADR scenario van de Inspectie OOV Figure 1.Impression of ship-collision scenario. These ingredients make a scenario via which command & control rooms in the Netherlands have been assessed. The variety of information (from the general public, water authorities, the involved ships, public safety officers, etc.) provides a useful background to conduct experiments with task-relevant information provision. In our experiment, the participant is a local commander of the police ( officier van dienst ) at the location of the incident. He or she is in charge of maintaining situation awareness, assessing risks and priorities for action, and overseeing the incident response and management. The participant has to engage in meaningful action in an operational background task, while acquiring and maintaining situation awareness. 2.2 Operational Task The rationale for the operational background task is to simulate that the participants are busy with their normal work tasks during incident response and management. In the experiment, 2 Inspectie Openbare Orde & Veiligheid, 7

8 Marc de Lignie these normal work tasks are simulated by forcing the participants to play the game Bubbleshooter 3, rather than real tasks performed in the crisis scenario. Figure 2. Snapshot of the free game Bubbleshooter from Absolutist.com. The main criteria for selecting Bubbleshooter to mimick the operational background the operational task are: The game pauses automatically when the player is not active. The game requires full attention yet can be left unattended to read (new) messages. The game can be restarted as often as required. The game does not cause stress or arousal. 2.3 Message sets The original message set accompanying the scenario contains 223 textual messages which describe the entire IOOV scenario. From this set an initial working set. Set-0 is derived manually containing 169 messages. Relevance assessment criteria to arrive at Set-0: 1) Repetitive incident facts are dismissed. 2) Noisy information and obvious incorrect information are dismissed. 3) New incidents that can be treated independently from the main incident are dismissed. The 169 remaining messages were categorized manually as follows (see Table 1): Messages relevant to the local police task, such as providing access for ambulance and fire brigade, traffic, evacuation, legal evidence, etc. Messages about identified victims, the numbers of which can be summed (or aggregated cq. combined). Messages about other combinable numbers (see Table 2). A total of 37 messages with combinable information also contain other relevant facts and cannot be dismissed. So 108 (=145-37) messages contain only combinable information, as listed in Table 1. Special messages, which are the remaining messages relevant to other public safety services. 3 A freeware game from Absolutist.com 8

9 Information Provision D-CIS TR/ Table 1. Message sets, including intermediate sets not used during experimentation. Relevant to police task Victims combinable Other combinable Special Total Set-0 (not used) Set Set-2 (not used) Set Table 2. Overview of combinable information elements. Combinable information elements Quantity Number of reports about floating containers 20 Number of reports about gas smell 64 Number of information requests from relatives 27 Number of information requests from effect area about safety 34 Total 145 In the current experiment only message Set-1 and Set-3 are used. Set-3 contains the messages relevant to the police task of the local police commander and some new messages are composed containing the aggregated numbers. Set-1 contains all messages, with the exception of 2/3 of the messages categorized as other combinable. This manual filtering procedure was conducted by, for every three messages, to keep the first message and filtering out the second and third message. 2.4 Hypothesis This experiment focuses on the effect of the aggregated message set on the situation awareness of the participant. Two message sets are used in the experiment: Set-1 and Set-3 (where Set-3 is the most reduced and smallest set). We hypothesize that with Set-3 people have the highest situation awareness and have more time to spend on their background operational task. H1: Participants with Set-3 can recall the situation more accurately than participants with Set-1. H2: Participants with Set-3 interpret the received data about the actual situation more accurately than participants with Set-1. H3: Participants with Set-3 spend more time on their operational task than participants with Set-1. 9

10 Marc de Lignie 3 Experimental Design In this experiment two situations were compared: Set-1: the first adapted message set. Set-3: the aggregated & task-relevant message set. For these two conditions (Set-1 and Set-3) each participant was randomly assigned to one of the conditions. The group of participants with Set-1 was located in a different room than the participants with Set-3. The setting of this experiment was as follows. Each participant sat behind a desk facing two LCD screens with only one keyboard and one mouse (see Figure 3). The arrangement in the rooms was such that the participants could not subversively talk or directly look at each other s screens. Figure 3. Computer systems and participants in one of the rooms. The positioning of the devices was such that the participant had the keyboard and mouse in front of him/her. The participant had to look slightly to the left or the right to look at one of the screens. The screen on the left presented the messages; the screen on the right contained the graphical interface for the operational task (i.e. Bubbleshooter). The screen on the left is the prime task. This screen runs an automatic PowerPoint presentation showing incoming messages from the appropriate message set. Each message is shown for 12 seconds, after which it disappears from the screen. There is an audible sign indicating arrival of a new message. The participants often needed to interrupt their operational task. 10

11 Information Provision D-CIS TR/ A total of eighteen participants was recruited from a pool of trainees from the course Police Duty Officer by the Police Academy in The Netherlands 4. This course aims at training police officers in charge at the incident location on a major incident. The MOSAIC experiment was conducted during two separate courses, with 8 and 10 participants respectively. The procedure of the experiment was as follows: Set-up of the participant s desks and testing each set-up. Plenary introduction to the experiment. Individual familiarization with the screens: reading a test message and playing Bubbleshooter. Before starting Session 1: start the performance monitor (Windows XP system utility) for measuring Bubbleshooter usage. Session 1: presentation of first half of the message Set-1 or Set-3, with participant (also) playing Bubbleshooter. Questionnaire 1: the message flow is paused, and the participants have as much time as they need to fill in a brief questionnaire. Session 2: presentation of the second half of the message Set-1 or Set-3, with the participant continuing playing Bubbleshooter. Questionnaire 2: the participants answered the questionnaire. Plenary debriefing. Hand-out of token of appreciation for the unpaid participants (a barbecue apron with the text If anyone observes me wearing this apron, call 112 immediately ). 3.1 Ways of measurement Questionnaire: subjective data A questionnaire was made with open and closed questions regarding: Usability of MOSAIC Realism of scenario The quality of the information in the messages; is the info judged as relevant for the sitrap and planned unit tasks for the incident Intrinsic quality of information aspects (correct, in-time, precise, complete, controllable) Relevance of provided information Missing information Information overload experiences Subjective ideas about Enhanced or Super situational awareness How distracting was reading of the information messages for the background task? Questionnaire: objective data Some questions pertain to detailed information in the messages. The correctness of the answers is presumably a measure of the situation awareness of the participant on the events and objects in the incident afterwards Other data Extra data are derived from keyboard and mouse activity logs, in order to measure the times spent on the various tasks; and background task performance, the score of the Bubbleshooter game. 4 Politie Academie, 11

12 Marc de Lignie 4 Results & Analysis 4.1 Results This section provides the raw results from the questionnaires and the performance measurements. These results are analysed in more detail in Section 4.2. Interpretation of the results follows in Section 5. The scores on the open questions are listed in Table 3. Each score is listed as an average (AVG) and standard deviation (STD) of 9 participants. All scores are normalized to values between 0 and 1. To test the confidence for the averages of Set-1 and Set-3 to be different, the unpaired one-tail two-sample t-test with unequal variances was applied. A confidence value of 90% means that in 9 out of 10 experiments Set-1 and Set-3 averages will be found to differ in the same way as measured. Table 3. Open questions, A1-A7 answered after Session 1, A8-A16 answered after Session 2. Set-1 Set-3 Session 1 AVG STD AVG STD CONF A1. Wat voor soort schepen waren betrokken bij de botsing? Wat was hun lading, omvang en naam? % A2. Wat was de toedracht van de botsing? A3. Waar gingen de schepen heen na de botsing? U kunt hiervoor bijgaande kaart gebruiken. A4. Wat zijn de brongebied(en) en effectgebied(en)? A5. Welke politietaken zijn van toepassing op deze bron- en effectgebieden? A6. Hoe verwacht u dat het incident zich verder ontwikkelt (best, realistic en worst case)? A7. Welke beslispunten denkt u dat daarbij naar voren zullen komen (met name t.a.v. prioritering, opschaling, veiligheid en advisering naar SGBO/ROT)? % % % % % % Session 2 AVG STD AVG STD CONF A8. Wat was het aantal zwaargewonden op en afkomstig van de schepen? % A9. Wat was het totaal aantal gewonden op en afkomstig van de schepen? A10. Wat was de afloop m.b.t. de man overboord? % % 12

13 Information Provision D-CIS TR/ A11. Wat waren de waarneembare eigenschappen en effecten van de stoffen die lekten uit het containerschip? A12. Hoe was de ontwikkeling van het effectgebied (omvang, richting)? U kunt hiervoor eventueel de kaart uit de eerste vragenlijst gebruiken. A13. Hoeveel omwonenden belden 112 met vragen om raad? A14. Hoeveel bezorgde familieleden en bekenden belden 112 om te informeren naar de toestand van slachtoffers en bewoners van het effectgebied? A15. Welke knelpunten deden zich voor? % % % % % Total first half Total second half Grand total The scores on the closed questions are listed in Table 4, again as averages and standard deviations of scores of 9 participants. For questions D1 and D3 the scores of one participant were removed as they were clearly outliers with respect to the rest of the participants. Participants could answer the questions on a 5-point scale with the meanings: 1 Very low 2 Low 3 Somewhat 4 High 5 Very high Calculation of the confidence of the Set-1 and Set-3 average scores to be different was done in the same manner as for the open questions. Table 4. Closed questions, answered after Session 2. Set-1 Set-3 AVG STD AVG STD CONF B1. Hoe relevant vond u de informatie uit de berichten voor de uitvoering van de politietaken? % B2. Hoe relevant vond u de informatie uit de berichten voor de coördinatie van de politietaken? B3. Hoe representatief vond u het experiment voor het werken onder crisisomstandigheden voor wat betreft het verdelen van de aandacht tussen het verwerken van informatie en het uitvoeren van directe taken? B4. Hoe waardevol was het direct ontvangen van informatie uit de meldkamer t.o.v. een reële situatie waarin u deze informatie (later) mondeling via C2000 ontvangen zou hebben? % % % 13

14 Marc de Lignie B5. Hoe waardevol was de miniplot van de locatie van de 112 beller t.o.v. de plaats incident bij elk bericht? C1. In plaats van veel van de berichten die u heeft ontvangen, hadden we u periodiek een overzicht kunnen sturen met meldingen over het aantal slachtoffers, het aantal inwoners dat om raad vraagt en het aantal bezorgde familieleden. In welke mate kan zo'n overzicht de informatie uit de oorspronkelijke meldingen vervangen (voor de politietaak)? C2. In plaats van veel van de berichten die u heeft ontvangen, hadden we u periodiek een geografische plot kunnen sturen met meldingen van een gaslucht en observaties van drijvende containers. In welke mate kan zo'n plot de informatie uit de oorspronkelijke meldingen vervangen (voor de politietaak)? D1. Hoe waardevol waren de overzichten met meldingen over het aantal slachtoffers, het aantal inwoners dat om raad vraagt en het aantal bezorgde familieleden? D2. Hoe waardevol waren de plots met meldingen van een gaslucht? D3. De genoemde overzichten in vraag D1 en de plots in vraag D2 zijn gebaseerd op een groot aantal meldingen waarvan u geen detail berichten heeft ontvangen. Bekijk nu kort een aantal van deze meldingen in de bijgeleverde printout. In welke mate kunnen het overzicht en de plot de informatie uit de oorspronkelijke meldingen vervangen (voor de politietaak)? % The scores on the performance measurements are shown in Table 5. For three out of 18 participants performance data were not available. The Bubbleshooter scores are the sums of scores of all Bubbleshooter games an individual participant played. The Bubbleshooter seconds gives the total duration of the perfmon.exe logs for which there was evidence that the participant was busy with the Bubbleshooter game. Table 5. Performance managements. Set-1 Set-3 AVG STD AVG STD Total seconds simulation Bubbleshooter seconds Bubbleshooter seconds percentage 65% 4% 77% 7% Bubbleshooter score all participants Bubbleshooter score experienced gamers Analysis A first look at the scores on the open questions in Table 3 reveals that the Set-3 participants have higher scores on the whole, but that for some questions the Set-1 participants have higher scores (questions A4-A6 and A11). Also for some questions the differences between 14

15 Information Provision D-CIS TR/ the two groups have a very low significance, say with a confidence level below 80% (questions A2, A12, A15). The open questions where the Set-3 participants scored better can be divided in three groups: 1 questions about recalling simple facts, such as names and numbers (A1, A8, A9, A13, A14), 2 questions about recalling events from the storyline (A3, A10), though with a lower confidence, 3 questions building on the interpretation of the received data (A7). Two of the questions where the Set-1 participants had higher scores are analysed in more detail in Table 6 and Table 7. Table 6 with the breakdown of the answers to question A4 shows that both groups scored similarly in pointing to the locations of the three involved ships as source areas for the incident, i.e. areas where the victims, floating containers and liquid and gaseous pollution originate from. However, the Set-3 participants were less aware of effect areas deserving attention, i.e. the areas where the floating containers and liquid and gaseous pollution moved into. To help understand these results in the number of messages was counted via which the participants were given information about the required answers. As can be seen from Table 6 the scores are correlated with these counts. Table 6. Breakdown of the answers to question A4. Fact to be mentioned in answer Mentioned in Set-1 Total score Set-1 participants Joapa ship as source area of a ship collision Apollo and Zuiddiep ships as source area of incident with wounded victims A gas cloud in northern direction as an effect area Floating containers downstream the Maas as an effect area Pollution downstream the Maas as an effect area Mentioned in Set Total score Set-3 participants The breakdown of the answers to question A11 shows that the participants of both groups had a comparable notion of the observable characteristics of the leaking chemicals, but that the Set-1 participants were much more aware of the harmful effects of the chemicals for the human victims. Again, the number of messages that gave evidence for the expected answers were counted; the scores correlate with the associated message counts. Table 7. Breakdown of the answers to question A11. Fact to be mentioned in answer Mentioned in Set-1 Total score Set-1 Mentioned in Set-3 participants Bad smell Mist Irritated skin and/or eyes Breathing problems Total score Set-3 participants The closed questions listed in Table 4 extracted the opinions of the participants regarding the relevance of the experiment for their work. From the answers to individual questions the following conclusions are derived: 15

16 Marc de Lignie B1 and B2: only the Set-3 participants found the messages highly relevant to coordinating the police activities. The groups agreed in their opinion that the information provided also has (some) relevance to actually carrying out police duties. These results are consistent with the experiment being aimed at the chief police officer at the incident location. B3: only the Set-1 participants found the experiment really challenging regarding dividing their attention between a background task and reading messages. This indicates a tuning issue of the experiment: making the experiment challenging for the Set-3 participants would require either a shorter simulation for them or creating a more demanding task for the Set-1 participants. B4: both groups confirm the value of the information used in the experiment. Indeed, also the SBG Intake & Noodhulp puts the immediate receipt of 112 calls by police officers high on their wish list. B5: both groups confirmed that it is useful to have the location of the 112 caller plotted relative to the geographic location of the incident. The scores could have been higher if they had been subject to an earlier try-out of the experiment where only the address of the 112 caller in the text message was provided in stead of the map. In the current experiment participants had severe problems with processing all the different street names. C1, C2, D3: the Set-1 participants confirm that they had rather received messages with aggregated information than the flood of all the individual messages. The support from the Set-3 participants the other way around is less convincing; this may have to do with the formulation of question D3 being too complicated (resulting in half of the answers being option 3 which could also mean I do not understand ). D1, D2: the Set-3 participants confirm that they find the messages with aggregated information relevant, especially the geographic plot with reports of bad smell (D2) and to a lesser extent the overviews with numbers of victims, people calling for advice and people worried about family (D1). Table 5 with performance measurements shows which fraction of the available time the participants played the Bubbleshooter game. It turns out that for the Set-3 participants the playing time was equal to the time that no messages were visible on the second screen (77% versus 79%). The Set-1 participants, however, only had 50% of time without messages, while they managed to keep playing the game for 65% of the time. This is consistent with the fact that for the Set-1 participants the messages were, on average, shorter and more often of a repetitive nature. It is also consistent with the scores of question B3 where only the Set-1 participants found the experiment stressful. 16

17 Information Provision D-CIS TR/ Discussion The discussion on the validity of the applied aggregation criteria leads to the following conclusions about the central hypotheses on aggregation, as formulated in Section 2.4: 1. The first hypothesis on a better recall of the situation by Set-3 participants is clearly corroborated by the experimental results, although the experiment points at the care needed regarding the filtering by novelty and the presentation of aggregated data. 2. The experiment is not conclusive on the second hypothesis on a better interpretation of the situation by Set-3 participants. This is due to the fact that only 4 out of 15 open questions tested this aspect (A5-A7, A15), two of which were hampered by Set-3 participants badly recalling the effect areas of the incident (question A4). 3. The third hypothesis on time spent on the main task is more or less trivial and indeed the Set-3 participants compensate the shorter time spent for reading by spending more time on playing the Bubbleshooter game. The scores of the Bubbleshooter turn out to be too dependent on the experience of the participants and are not considered useful as a performance criterion, although also here the Set-3 participants have higher scores on the game. Next to the results on aggregation, we can discuss the relevance assessment criteria applied to the messages, as described in section 2.3. The findings boil down to four factors, as follows: 1. Novelty of information (i.e.: information is shown only once) The effects of filtering by novelty were not explicitly tested in the experiment, because repetitive facts were filtered out for both Set-1 and Set-3. However, it was noted in Section 4.2 that information that was repeated (since the involved messages also contained new facts) very likely helped the recipients to better recall the information. Questions A4-A6 about the source and effect areas involved in the scenario resulted in higher scores for the Set-1 participants. Although the Set-1 participants had to filter the relevant information from a larger number of messages themselves, apparently the repetition of the bad smell in three separate messages helped them in building a mental picture of a developing gas cloud. The repetition effect also played a role in question A11 about the observable characteristics and effects of the leaking chemicals, where the Set-3 participants had to infer the large-scale effects on breathing, skin and eyes from a single message, while the Set-1 participants received multiple reports from victims with these symptoms. On the other hand, repetition increases the recipient s distraction from the main task. The optimal repetition factor for information that is important to be remembered and how repetitive information can be combined with new information has to be investigated further. This includes checking of studies in other fields, such as psychonomics and cognitive psychology. 2. Aggregation of numbers Aggregating numbers of victims and certain types of events turns out to be a most effective way to decrease the number of information messages while at the same time increasing the awareness of the situation. The aggregate numbers, such as the examples from the current simulation, help the public safety officer assess the severity and size of the incident, as also indicated in the closed questions. This result is highly relevant for crisis management. 3. Aggregation of geographical distribution data One would expect that aggregating reports about the geographical spreading of some event by geographically plotting the reports on a map would be as effective as aggregation of numbers discussed above. In the current simulation this applies to the spreading of a gas cloud due to the wind and the spreading of floating containers and leaking fluid chemicals downstream the river. The current findings do not support this expectation (question A4 negative, question A12 positive). However, it is possible that geographical aggregation can be as effective as numerical aggregation to reduce information overload and enhance situation awareness, provided that: 17

18 Marc de Lignie - the presentation of information takes into account (optimal) repetition, as discussed above, - the users are trained in recognizing and interpreting the geographic plots. In the current experiment the plots may have been too complex to process without previous training. 4. Relevance to the tasks of the recipient The effects of relevance assessment by task relevance are not explicitly tested, since the difference in numbers of messages between Set-1 and Set-3 is also due to the aggregation of facts. However, as scores of Set-3 participants are generally higher, it is at least safe to conclude that reduction of the number of messages with irrelevant information or information that can be aggregated is effective in improving the information processing of the recipients. Higher scores of the Set-3 group on the recall of simple facts such as names and events, which does not explicitly depend on the aggregation of information, indicate that task relevance is an important aggregation criterion. 18

19 Information Provision D-CIS TR/ Conclusion The first indicative experiment of the MOSAIC project had the following objectives: Validation of the MOSAIC ideas on information provision, or more specific: enhanced situation awareness and reduced information overload. Refinement and enhancement of personalized information filtering. Suitability of the used scenarios to realize enhanced situation awareness, reduced information overload and the corresponding personalized information filtering. 1. We found that removal of messages containing information that was sent before reduces information overload of the police officer. However, removal of all repeated information proves to be overdone; some repetition helps the police officer to build a mental picture of all aspects the safety incident. The optimum repetition factor for cases like these needs further exploration. 2. Aggregating numbers of victims and certain types of events turns out to be a most effective way to decrease the number of messages while at the same time improving the situation awareness of the participant. This result is highly relevant for crisis management. 3. The results do not support our expectation that aggregating geographical data by plotting them (on a map) would be very effective. Probably the optimum to repetition of messages with geographical information is different from the optimum from other types of information (see the foregoing note). It is also possible that police officers will have to be more trained to process geographic plots. 4. Basically, reduction of the number of messages with irrelevant information or information that can be aggregated is effective in improving the information processing of the recipients. The high scores of the Set-3 group on the recall of simple facts such as names and events, indicates that task relevance is an important aggregation criterion. In conclusion, the experiment is successful in showing useful differences in presenting reduced information (deduced and aggregated) versus unreduced information. In general our hypotheses have been validated by the results of the experiment. This is important support for the MOSAIC underlying ideas on the use of information filtering to effectuate enhanced situation awareness and reduced information overload. The set-up of the indicative experiment, albeit constrained in its flexibility, was useful to acquire comparable performance results. The original scenario was slightly adapted to our focus on the Police officer on duty. It proved to be sufficiently realistic to engage police participants. MOSAIC continues its research and experimentation efforts to further investigate and demonstrate a Multi-Officer System of Agents for Informed crisis Control. 19

20 Marc de Lignie 7 Appendices The appendices in this technical report are listed below. Most of them are written in Dutch. A. Introductionary handouts for participants B. First questionnaire C. Final questionnaire D. Debriefing handout E. Ratings for open questions F. Transcripts of answers to open questions 20