Engineering Psychology and Cognitive Ergonomics: 8th International Conference, EPCE 2009, Held as Part of HCI International 2009, San Diego, CA, USA, ... Lecture Notes in Artificial Intelligence)

Citation preview

Lecture Notes in Artificial Intelligence Edited by R. Goebel, J. Siekmann, and W. Wahlster

Subseries of Lecture Notes in Computer Science

5639

Don Harris (Ed.)

Engineering Psychology and Cognitive Ergonomics 8th International Conference, EPCE 2009 Held as Part of HCI International 2009 San Diego, CA, USA, July 19-24, 2009 Proceedings

13

Series Editors Randy Goebel, University of Alberta, Edmonton, Canada Jörg Siekmann, University of Saarland, Saarbrücken, Germany Wolfgang Wahlster, DFKI and University of Saarland, Saarbrücken, Germany Volume Editor Don Harris Cranfield University, School of Engineering Department of Systems Engineering and Human Factors Cranfield, Bedford MK43 0AL, UK E-mail: d.harris@cranfield.ac.uk

Library of Congress Control Number: 2009928834

CR Subject Classification (1998): I.2.0, I.2, H.5, H.1.2, H.3, H.4.2, I.6, J.2-3 LNCS Sublibrary: SL 7 – Artificial Intelligence ISSN ISBN-10 ISBN-13

0302-9743 3-642-02727-X Springer Berlin Heidelberg New York 978-3-642-02727-7 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12706648 06/3180 543210

Foreword

The 13th International Conference on Human–Computer Interaction, HCI International 2009, was held in San Diego, California, USA, July 19–24, 2009, jointly with the Symposium on Human Interface (Japan) 2009, the 8th International Conference on Engineering Psychology and Cognitive Ergonomics, the 5th International Conference on Universal Access in Human-Computer Interaction, the Third International Conference on Virtual and Mixed Reality, the Third International Conference on Internationalization, Design and Global Development, the Third International Conference on Online Communities and Social Computing, the 5th International Conference on Augmented Cognition, the Second International Conference on Digital Human Modeling, and the First International Conference on Human Centered Design. A total of 4,348 individuals from academia, research institutes, industry and governmental agencies from 73 countries submitted contributions, and 1,397 papers that were judged to be of high scientific quality were included in the program. These papers address the latest research and development efforts and highlight the human aspects of the design and use of computing systems. The papers accepted for presentation thoroughly cover the entire field of human-computer interaction, addressing major advances in knowledge and effective use of computers in a variety of application areas. This volume, edited by Don Harris, contains papers in the thematic area of Engineering Psychology and Cognitive Ergonomics, addressing the following major topics: • • • •

Cognitive Approaches in HCI Design Interaction and Cognition Driving Safety and Support Aviation and Transport

The remaining volumes of the HCI International 2009 proceedings are: • • • • • •

Volume 1, LNCS 5610, Human–Computer Interaction––New Trends (Part I), edited by Julie A. Jacko Volume 2, LNCS 5611, Human–Computer Interaction––Novel Interaction Methods and Techniques (Part II), edited by Julie A. Jacko Volume 3, LNCS 5612, Human–Computer Interaction––Ambient, Ubiquitous and Intelligent Interaction (Part III), edited by Julie A. Jacko Volume 4, LNCS 5613, Human–Computer Interaction––Interacting in Various Application Domains (Part IV), edited by Julie A. Jacko Volume 5, LNCS 5614, Universal Access in Human–Computer Interaction––Addressing Diversity (Part I), edited by Constantine Stephanidis Volume 6, LNCS 5615, Universal Access in Human–Computer Interaction––Intelligent and Ubiquitous Interaction Environments (Part II), edited by Constantine Stephanidis

VI

Foreword

• • • • • • • • • •

Volume 7, LNCS 5616, Universal Access in Human–Computer Interaction––Applications and Services (Part III), edited by Constantine Stephanidis Volume 8, LNCS 5617, Human Interface and the Management of Information––Designing Information Environments (Part I), edited by Michael J. Smith and Gavriel Salvendy Volume 9, LNCS 5618, Human Interface and the Management of Information––Information and Interaction (Part II), edited by Gavriel Salvendy and Michael J. Smith Volume 10, LNCS 5619, Human Centered Design, edited by Masaaki Kurosu Volume 11, LNCS 5620, Digital Human Modeling, edited by Vincent G. Duffy Volume 12, LNCS 5621, Online Communities and Social Computing, edited by A. Ant Ozok and Panayiotis Zaphiris Volume 13, LNCS 5622, Virtual and Mixed Reality, edited by Randall Shumaker Volume 14, LNCS 5623, Internationalization, Design and Global Development, edited by Nuray Aykin Volume 15, LNCS 5624, Ergonomics and Health Aspects of Work with Computers, edited by Ben-Tzion Karsh Volume 16, LNAI 5638, The Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience, edited by Dylan Schmorrow, Ivy Estabrooke and Marc Grootjen

I would like to thank the Program Chairs and the members of the Program Boards of all thematic areas, listed below, for their contribution to the highest scientific quality and the overall success of HCI International 2009.

Ergonomics and Health Aspects of Work with Computers Program Chair: Ben-Tzion Karsh Arne Aarås, Norway Pascale Carayon, USA Barbara G.F. Cohen, USA Wolfgang Friesdorf, Germany John Gosbee, USA Martin Helander, Singapore Ed Israelski, USA Waldemar Karwowski, USA Peter Kern, Germany Danuta Koradecka, Poland Kari Lindström, Finland

Holger Luczak, Germany Aura C. Matias, Philippines Kyung (Ken) Park, Korea Michelle M. Robertson, USA Michelle L. Rogers, USA Steven L. Sauter, USA Dominique L. Scapin, France Naomi Swanson, USA Peter Vink, The Netherlands John Wilson, UK Teresa Zayas-Cabán, USA

Foreword

Human Interface and the Management of Information Program Chair: Michael J. Smith Gunilla Bradley, Sweden Hans-Jörg Bullinger, Germany Alan Chan, Hong Kong Klaus-Peter Fähnrich, Germany Michitaka Hirose, Japan Jhilmil Jain, USA Yasufumi Kume, Japan Mark Lehto, USA Fiona Fui-Hoon Nah, USA Shogo Nishida, Japan Robert Proctor, USA Youngho Rhee, Korea

Anxo Cereijo Roibás, UK Katsunori Shimohara, Japan Dieter Spath, Germany Tsutomu Tabe, Japan Alvaro D. Taveira, USA Kim-Phuong L. Vu, USA Tomio Watanabe, Japan Sakae Yamamoto, Japan Hidekazu Yoshikawa, Japan Li Zheng, P.R. China Bernhard Zimolong, Germany

Human–Computer Interaction Program Chair: Julie A. Jacko Sebastiano Bagnara, Italy Sherry Y. Chen, UK Marvin J. Dainoff, USA Jianming Dong, USA John Eklund, Australia Xiaowen Fang, USA Ayse Gurses, USA Vicki L. Hanson, UK Sheue-Ling Hwang, Taiwan Wonil Hwang, Korea Yong Gu Ji, Korea Steven Landry, USA

Gitte Lindgaard, Canada Chen Ling, USA Yan Liu, USA Chang S. Nam, USA Celestine A. Ntuen, USA Philippe Palanque, France P.L. Patrick Rau, P.R. China Ling Rothrock, USA Guangfeng Song, USA Steffen Staab, Germany Wan Chul Yoon, Korea Wenli Zhu, P.R. China

Engineering Psychology and Cognitive Ergonomics Program Chair: Don Harris Guy A. Boy, USA John Huddlestone, UK Kenji Itoh, Japan Hung-Sying Jing, Taiwan Ron Laughery, USA Wen-Chin Li, Taiwan James T. Luxhøj, USA

Nicolas Marmaras, Greece Sundaram Narayanan, USA Mark A. Neerincx, The Netherlands Jan M. Noyes, UK Kjell Ohlsson, Sweden Axel Schulte, Germany Sarah C. Sharples, UK

VII

VIII

Foreword

Neville A. Stanton, UK Xianghong Sun, P.R. China Andrew Thatcher, South Africa

Matthew J.W. Thomas, Australia Mark Young, UK

Universal Access in Human–Computer Interaction Program Chair: Constantine Stephanidis Julio Abascal, Spain Ray Adams, UK Elisabeth André, Germany Margherita Antona, Greece Chieko Asakawa, Japan Christian Bühler, Germany Noelle Carbonell, France Jerzy Charytonowicz, Poland Pier Luigi Emiliani, Italy Michael Fairhurst, UK Dimitris Grammenos, Greece Andreas Holzinger, Austria Arthur I. Karshmer, USA Simeon Keates, Denmark Georgios Kouroupetroglou, Greece Sri Kurniawan, USA

Patrick M. Langdon, UK Seongil Lee, Korea Zhengjie Liu, P.R. China Klaus Miesenberger, Austria Helen Petrie, UK Michael Pieper, Germany Anthony Savidis, Greece Andrew Sears, USA Christian Stary, Austria Hirotada Ueda, Japan Jean Vanderdonckt, Belgium Gregg C. Vanderheiden, USA Gerhard Weber, Germany Harald Weber, Germany Toshiki Yamaoka, Japan Panayiotis Zaphiris, UK

Virtual and Mixed Reality Program Chair: Randall Shumaker Pat Banerjee, USA Mark Billinghurst, New Zealand Charles E. Hughes, USA David Kaber, USA Hirokazu Kato, Japan Robert S. Kennedy, USA Young J. Kim, Korea Ben Lawson, USA

Gordon M. Mair, UK Miguel A. Otaduy, Switzerland David Pratt, UK Albert “Skip” Rizzo, USA Lawrence Rosenblum, USA Dieter Schmalstieg, Austria Dylan Schmorrow, USA Mark Wiederhold, USA

Internationalization, Design and Global Development Program Chair: Nuray Aykin Michael L. Best, USA Ram Bishu, USA Alan Chan, Hong Kong Andy M. Dearden, UK

Susan M. Dray, USA Vanessa Evers, The Netherlands Paul Fu, USA Emilie Gould, USA

Foreword

Sung H. Han, Korea Veikko Ikonen, Finland Esin Kiris, USA Masaaki Kurosu, Japan Apala Lahiri Chavan, USA James R. Lewis, USA Ann Light, UK James J.W. Lin, USA Rungtai Lin, Taiwan Zhengjie Liu, P.R. China Aaron Marcus, USA Allen E. Milewski, USA

Elizabeth D. Mynatt, USA Oguzhan Ozcan, Turkey Girish Prabhu, India Kerstin Röse, Germany Eunice Ratna Sari, Indonesia Supriya Singh, Australia Christian Sturm, Spain Adi Tedjasaputra, Singapore Kentaro Toyama, India Alvin W. Yeo, Malaysia Chen Zhao, P.R. China Wei Zhou, P.R. China

Online Communities and Social Computing Program Chairs: A. Ant Ozok, Panayiotis Zaphiris Chadia N. Abras, USA Chee Siang Ang, UK Amy Bruckman, USA Peter Day, UK Fiorella De Cindio, Italy Michael Gurstein, Canada Tom Horan, USA Anita Komlodi, USA Piet A.M. Kommers, The Netherlands Jonathan Lazar, USA Stefanie Lindstaedt, Austria

Gabriele Meiselwitz, USA Hideyuki Nakanishi, Japan Anthony F. Norcio, USA Jennifer Preece, USA Elaine M. Raybourn, USA Douglas Schuler, USA Gilson Schwartz, Brazil Sergei Stafeev, Russia Charalambos Vrasidas, Cyprus Cheng-Yen Wang, Taiwan

Augmented Cognition Program Chair: Dylan D. Schmorrow Andy Bellenkes, USA Andrew Belyavin, UK Joseph Cohn, USA Martha E. Crosby, USA Tjerk de Greef, The Netherlands Blair Dickson, UK Traci Downs, USA Julie Drexler, USA Ivy Estabrooke, USA Cali Fidopiastis, USA Chris Forsythe, USA Wai Tat Fu, USA Henry Girolamo, USA

Marc Grootjen, The Netherlands Taro Kanno, Japan Wilhelm E. Kincses, Germany David Kobus, USA Santosh Mathan, USA Rob Matthews, Australia Dennis McBride, USA Robert McCann, USA Jeff Morrison, USA Eric Muth, USA Mark A. Neerincx, The Netherlands Denise Nicholson, USA Glenn Osga, USA

IX

X

Foreword

Dennis Proffitt, USA Leah Reeves, USA Mike Russo, USA Kay Stanney, USA Roy Stripling, USA Mike Swetnam, USA Rob Taylor, UK

Maria L.Thomas, USA Peter-Paul van Maanen, The Netherlands Karl van Orden, USA Roman Vilimek, Germany Glenn Wilson, USA Thorsten Zander, Germany

Digital Human Modeling Program Chair: Vincent G. Duffy Karim Abdel-Malek, USA Thomas J. Armstrong, USA Norm Badler, USA Kathryn Cormican, Ireland Afzal Godil, USA Ravindra Goonetilleke, Hong Kong Anand Gramopadhye, USA Sung H. Han, Korea Lars Hanson, Sweden Pheng Ann Heng, Hong Kong Tianzi Jiang, P.R. China

Kang Li, USA Zhizhong Li, P.R. China Timo J. Määttä, Finland Woojin Park, USA Matthew Parkinson, USA Jim Potvin, Canada Rajesh Subramanian, USA Xuguang Wang, France John F. Wiechel, USA Jingzhou (James) Yang, USA Xiu-gan Yuan, P.R. China

Human Centered Design Program Chair: Masaaki Kurosu Gerhard Fischer, USA Tom Gross, Germany Naotake Hirasawa, Japan Yasuhiro Horibe, Japan Minna Isomursu, Finland Mitsuhiko Karashima, Japan Tadashi Kobayashi, Japan

Kun-Pyo Lee, Korea Loïc Martínez-Normand, Spain Dominique L. Scapin, France Haruhiko Urokohara, Japan Gerrit C. van der Veer, The Netherlands Kazuhiko Yamazaki, Japan

In addition to the members of the Program Boards above, I also wish to thank the following volunteer external reviewers: Gavin Lew from the USA, Daniel Su from the UK, and Ilia Adami, Ioannis Basdekis, Yannis Georgalis, Panagiotis Karampelas, Iosif Klironomos, Alexandros Mourouzis, and Stavroula Ntoa from Greece. This conference could not have been possible without the continuous support and advice of the Conference Scientific Advisor, Prof. Gavriel Salvendy, as well as the dedicated work and outstanding efforts of the Communications Chair and Editor of HCI International News, Abbas Moallem.

Foreword

XI

I would also like to thank for their contribution toward the organization of the HCI International 2009 conference the members of the Human–Computer Interaction Laboratory of ICS-FORTH, and in particular Margherita Antona, George Paparoulis, Maria Pitsoulaki, Stavroula Ntoa, and Maria Bouhli. Constantine Stephanidis

Table of Contents

Part I: Cognitive Approaches in HCI Design Towards Cognitive-Aware Multimodal Presentation: The Modality Effects in High-Load HCI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yujia Cao, Mari¨et Theune, and Anton Nijholt Supporting Situation Awareness in Demanding Operating Environments through Wearable User Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jari Laarni, Juhani Heinil¨ a, Jukka H¨ akkinen, Virpi Kalakoski, Kari Kallinen, Kristian Lukander, Paula L¨ opp¨ onen, Tapio Palom¨ aki, Niklas Ravaja, Paula Savioja, and Antti V¨ a¨ at¨ anen Development of a Technique for Predicting the Human Response to an Emergency Situation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Glyn Lawson, Sarah Sharples, David Clarke, and Sue Cobb A Dynamic Task Representation Method for a Virtual Reality Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maria Chiara Leva, Alison Mragaret Kay, Fabio Mattei, Tom Kontogiannis, Massimiliano De Ambroggi, and Sam Cromie An Investigation of Function Based Design Considering Affordances in Conceptual Design of Mechanical Movement . . . . . . . . . . . . . . . . . . . . . . . . . Ying-Chieh Liu and Su-Ju Lu CWE: Assistance Environment for the Evaluation Operating a Set of Variations of the Cognitive Walkthrough Ergonomic Inspection Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thomas Mahatody, Christophe Kolski, and Mouldi Sagar

3

13

22

32

43

52

The Use of Multimodal Representation in Icon Interpretation . . . . . . . . . . Sin´e McDougall, Alexandra Forsythe, Sarah Isherwood, Agnes Petocz, Irene Reppa, and Catherine Stevens

62

Beyond Emoticons: Combining Affect and Cognition in Icon Design . . . . Sin´e McDougall, Irene Reppa, Gary Smith, and David Playfoot

71

Agency Attribution in Human-Computer Interaction . . . . . . . . . . . . . . . . . John E. McEneaney

81

Human-UAV Co-operation Based on Artificial Cognition . . . . . . . . . . . . . . Claudia Meitinger and Axel Schulte

91

XVI

Table of Contents

Development of an Evaluation Method for Office Work Productivity . . . . Kazune Miyagi, Hiroshi Shimoda, Hirotake Ishii, Kenji Enomoto, Mikio Iwakawa, and Masaaki Terano

101

Supporting Cognitive Collage Creation for Pedestrian Navigation . . . . . . Augustinus H.J. Oomes, Miroslav Bojic, and Gideon Bazen

111

Development of a Novel Platform for Greater Situational Awareness in the Urban Military Terrain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Stephen D. Prior, Siu-Tsen Shen, Anthony S. White, Siddharth Odedra, Mehmet Karamanoglu, Mehmet Ali Erbil, and Tom Foran The User Knows: Considering the Cognitive Contribution of the User in the Design of Auditory Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Catherine Stevens and Agnes Petocz

120

126

Part II: Interaction and Cognition The Influence of Gender and Age on the Visual Codes Working Memory and the Display Duration – A Case Study of Fencers . . . . . . . . . . . . . . . . . Chih-Lin Chang, Kai-Way Li, Yung-Tsan Jou, Hsu-Chang Pan, and Tai-Yen Hsu Comparison of Mobile Device Navigation Information Display Alternatives from the Cognitive Load Perspective . . . . . . . . . . . . . . . . . . . . Murat Can Cobanoglu, Ahmet Alp Kindiroglu, and Selim Balcisoy

139

149

Visual Complexity: Is That All There Is? . . . . . . . . . . . . . . . . . . . . . . . . . . . Alexandra Forsythe

158

Operational Decision Making in Aluminium Smelters . . . . . . . . . . . . . . . . . Yashuang Gao, Mark P. Taylor, John J.J. Chen, and Michael J. Hautus

167

Designers of Different Cognitive Styles Editing E-Learning Materials Studied by Monitoring Physiological and Other Data Simultaneously . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ´ Bodn´ K´ aroly Hercegfi, Olga Csillik, Eva ar, Judit Sass, and Lajos Izs´ o Analyzing Control-Display Movement Compatibility: A Neuroimaging Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . S.M. Hadi Hosseini, Maryam Rostami, Makoto Takahashi, Naoki Miura, Motoaki Sugiura, and Ryuta Kawashima Graphics and Semantics: The Relationship between What Is Seen and What Is Meant in Icon Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sarah Isherwood

179

187

197

Table of Contents

The Effect of Object Features on Multiple Object Tracking and Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tianwei Liu, Wenfeng Chen, Yuming Xuan, and Xiaolan Fu Organizing Smart Networks and Humans into Augmented Teams . . . . . . Martijn Neef, Martin van Rijn, Danielle Keus, and Jan-Willem Marck Quantitative Evaluation of Mental Workload by Using Model of Involuntary Eye Movement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Goro Obinata, Satoru Tokuda, Katsuyuki Fukuda, and Hiroto Hamada Spatial Tasks on a Large, High-Resolution Tiled Display: Females Mentally Rotate Large Objects Faster Than Men . . . . . . . . . . . . . . . . . . . . Bernt Ivar Olsen, Bruno Laeng, Kari-Ann Kristiansen, and Gunnar Hartvigsen Neurocognitive Workload Assessment Using the Virtual Reality Cognitive Performance Assessment Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Thomas D. Parsons, Louise Cosand, Christopher Courtney, Arvind Iyer, and Albert A. Rizzo

XVII

206 213

223

233

243

Sensing Directionality in Tangential Haptic Stimulation . . . . . . . . . . . . . . . Greg Placencia, Mansour Rahimi, and Behrokh Khoshnevis

253

Effects of Design Elements in Magazine Advertisements . . . . . . . . . . . . . . Young Sam Ryu, Taewon Suh, and Sean Dozier

262

The Influence of Shared-Representation on Shared Mental Models in Virtual Teams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rose Saikayasit and Sarah Sharples

269

Harnessing the Power of Multiple Tools to Predict and Mitigate Mental Overload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Charneta Samms, David Jones, Kelly Hale, and Diane Mitchell

279

Acceptance of E-Invoicing in SMEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Karl W. Sandberg, Olof Wahlberg, and Yan Pan Mental Models in Process Visualization - Could They Indicate the Effectiveness of an Operator’s Training? . . . . . . . . . . . . . . . . . . . . . . . . . . . . Karin Schweizer, Denise Gramß, Susi M¨ uhlhausen, and Birgit Vogel-Heuser Effects of Report Order on Identification on Multidimensional Stimulus: Color and Shape . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I-Hsuan Shen and Kong-King Shieh

289

297

307

XVIII

Table of Contents

Confidence Bias in Situation Awareness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ketut Sulistyawati and Yoon Ping Chui

317

Tactical Reconnaissance Using Groups of Partly Autonomous UGVs . . . . Peter Svenmarck, Dennis Andersson, Bj¨ orn Lindahl, Johan Hedstr¨ om, and Patrik Lif

326

Part III: Driving Safety and Support Use of High-Fidelity Simulation to Evaluate Driver Performance with Vehicle Automation Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Timothy Brown, Jane Moeckli, and Dawn Marshall

339

Applying the “Team Player” Approach on Car Design . . . . . . . . . . . . . . . . Staffan Davidsson and H˚ akan Alm

349

New HMI Concept for Motorcycles–The Saferider Approach . . . . . . . . . . . J.P. Frederik Diederichs, Marco Fontana, Giacomo Bencini, Stella Nikolaou, Roberto Montanari, Andrea Spadoni, Harald Widlroither, and Niccol` o Baldanzini

358

Night Vision - Reduced Driver Distraction, Improved Safety and Satisfaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Klaus Fuchs, Bettina Abendroth, and Ralph Bruder Measurement of Depth Attention of Driver in Frontal Scene . . . . . . . . . . . Mamiko Fukuoka, Shun’ichi Doi, Takahiko Kimura, and Toshiaki Miura Understanding the Opinion Forming Processes of Experts and Customers During Evaluations of Automotive Sounds . . . . . . . . . . . . . . . . Louise Humphreys, Sebastiano Giudice, Paul Jennings, Rebecca Cain, Garry Dunne, and Mark Allman-Ward HR Changes in Driving Scenes with Danger and Difficulties Using Driving Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yukiyo Kuriyagawa, Mieko Ohsuga, and Ichiro Kageyama Driver Measurement: Methods and Applications . . . . . . . . . . . . . . . . . . . . . Shane McLaughlin, Jonathan Hankey, and Thomas Dingus The Assessment of Driver’s Arousal States from the Classification of Eye-Blink Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yoshihiro Noguchi, Keiji Shimada, Mieko Ohsuga, Yoshiyuki Kamakura, and Yumiko Inoue Guiding a Driver’s Visual Attention Using Graphical and Auditory Animations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tony Poitschke, Florian Laquai, and Gerhard Rigoll

367 376

386

396 404

414

424

Table of Contents

Fundamental Study for Relationship between Cognitive Task and Brain Activity During Car Driving . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shunji Shimizu, Nobuhide Hirai, Fumikazu Miwakeichi, Senichiro Kikuchi, Yasuhito Yoshizawa, Masanao Sato, Hiroshi Murata, Eiju Watanabe, and Satoshi Kato A Study on a Method to Call Drivers’ Attention to Hazard . . . . . . . . . . . . Hiroshi Takahashi

XIX

434

441

An Analysis of Saccadic Eye Movements and Facial Images for Assessing Vigilance Levels During Simulated Driving . . . . . . . . . . . . . . . . . Akinori Ueno, Shoyo Tei, Tomohide Nonomura, and Yuichi Inoue

451

Implementing Human Factors within the Design Process of Advanced Driver Assistance Systems (ADAS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Boris van Waterschoot and Mascha van der Voort

461

A Survey Study of Chinese Drivers’ Inconsistent Risk Perception . . . . . . . Pei Wang, Pei-Luen Patrick Rau, and Gavriel Salvendy

471

Design for Smart Driving: A Tale of Two Interfaces . . . . . . . . . . . . . . . . . . Mark S. Young, Stewart A. Birrell, and Neville A. Stanton

477

Part IV: Aviation and Transport Supervision of Autonomous Vehicles: Mutual Modeling and Interaction Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Gilles Coppin, Fran¸cois Legras, and Sylvie Saget Conflicts in Human Operator – Unmanned Vehicles Interactions . . . . . . . Fr´ed´eric Dehais, Stephane Mercier, and Catherine Tessier Ergonomic Analysis of Different Computer Tools to Support the German Air Traffic Controllers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Muriel Didier, Margeritta von Wilamowitz-Moellendorff, and Ralph Bruder

489 498

508

Behavior Model Based Recognition of Critical Pilot Workload as Trigger for Cognitive Operator Assistance . . . . . . . . . . . . . . . . . . . . . . . . . . . Diana Donath and Axel Schulte

518

A Design and Training Agenda for the Next Generation of Commercial Aircraft Flight Deck . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Don Harris

529

Future Ability Requirements for Human Operators in Aviation . . . . . . . . Catrin Hasse, Carmen Bruder, Dietrich Grasshoff, and Hinnerk Eißfeldt

537

XX

Table of Contents

The Application of Human Error Template (HET) for Redesigning Standard Operational Procedures in Aviation Operations . . . . . . . . . . . . . Wen-Chin Li, Don Harris, Yueh-Ling Hsu, and Lon-Wen Li

547

Effect of Aircraft Datablock Complexity and Exposure Time on Performance of Change Detection Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chen Ling and Lesheng Hua

554

A Regulatory-Based Approach to Safety Analysis of Unmanned Aircraft Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ¨ James T. Luxhøj and Ahmet Oztekin

564

Using Acoustic Sensor Technologies to Create a More Terrain Capable Unmanned Ground Vehicle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Siddharth Odedra, Stephen D. Prior, Mehmet Karamanoglu, Mehmet Ali Erbil, and Siu-Tsen Shen

574

Critical Interaction Analysis in the Flight Deck . . . . . . . . . . . . . . . . . . . . . . Chiara Santamaria Maurizio, Patrizia Marti, and Simone Pozzi

580

Understanding the Impact of Rail Automation . . . . . . . . . . . . . . . . . . . . . . . Sarah Sharples, Nora Balfe, David Golightly, and Laura Millen

590

Cognitive Workload as a Predictor of Student Pilot Performance . . . . . . . Nathan F. Tilton and Ronald Mellado Miller

600

Direct Perception Displays for Military Radar-Based Air Surveillance . . . Oliver Witt, Morten Grandt, and Heinz K¨ uttelwesch

606

A Selection of Human Factors Tools: Measuring HCI Aspects of Flight Deck Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rolf Zon and Henk van Dijk

616

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

625

Towards Cognitive-Aware Multimodal Presentation: The Modality Effects in High-Load HCI Yujia Cao, Mariët Theune, and Anton Nijholt Human Media Interaction Group, University of Twente, P.O. Box 217, 7500AE, Enschede, The Netherlands {y.cao,m.theune,a.nijholt}@utwente.nl

Abstract. In this study, we argue that multimodal presentations should be created in a cognitive-aware manner, especially in a high-load HCI situation where the user task challenges the full capacity of the human cognition. An experiment was conducted to investigate the cognitive effects of modalities, using a high-load task. The performance measurements and subjective reports consistently confirm a significant modality impact on cognitive workload, stress and performance. A relation between modality usage and physiological states was not found, due to the insufficient sensitivity and individual differences of the physiological measurements. The findings of this experiment can be well explained by several modality-related cognitive theories. We further integrate these theories into a suitability prediction model, which can systematically predict how suitable a certain modality usage is for this presentation task. The model demonstrates a possible approach towards cognitive-aware modality planning and can be modified for other applications. Keywords: Cognitive-aware, multimodal presentation, modality planning, cognitive load, stress, performance, high-load HCI.

1 Introduction Advanced human-computer interactions are often accomplished through multiple modalities, such as text, images, speech, and sound. Modality planning in HCI is often accomplished in a context-aware manner, i.e. the modalities to be used are selected according to communication goals, user profiles, environmental conditions and resource limitations [1]. However, as multimodal presentations are created for human users to perceive, process and act upon, computer systems should understand not only how to convey information, but also how human minds take in and process the information. When taking into account the modality-related knowledge of human cognition, multimodal presentations could be created in a cognitive-aware manner, so they can be more efficiently perceived and processed. We believe that the cognitive aspects of modality planning are particularly essential in a high-load HCI situation, where the interaction challenges the full capacity of the human cognition. A huge body of psychology studies provides modality-related cognitive theories and principles that are potentially useful for cognitive-aware modality planning. D. Harris (Ed.): Engin. Psychol. and Cog. Ergonomics, HCII 2009, LNAI 5639, pp. 3–12, 2009. © Springer-Verlag Berlin Heidelberg 2009

4

Y. Cao, M. Theune, and A. Nijholt

According to Baddeley’s working memory model [4], the working memory has separated stores (perception channels) for visual information and auditory information, and each store has a limited capacity. Therefore, the capacity of working memory can be better used when both channels are used to perceive information. This theory is known as the dual-channel theory. Another modality-related finding of Baddeley is that the working memory relies on sub-vocal speech to rehearse information and maintain memory traces [3]. Furthermore, the dual-coding theory of Paivio [13] states that verbal and nonverbal information are represented and processed in separated mental systems. These two systems are interconnected through dynamic associative processes. Studies on multimedia learning have demonstrated that the associative processes between the verbal and nonverbal mental systems play a major role in knowledge comprehension and long-term memorization [8, 12]. In this study, we attempt to apply these modality-related cognitive theories as a foundation of cognitive-aware multimodal presentation. An experiment was conducted to investigate the cognitive effects of modalities, using a high-load HCI task. The results showed a significant modality impact on performance, cognitive workload and stress. Based on the experimental findings, we integrate the relevant theories into a prediction model that can systematically compare the suitability of different modality usages for this specific task.

2 Experiment We created an earthquake rescue scenario, where the locations of wounded and dead people are continuously reported to the crisis response center and displayed on a computer screen. Based on these reports, a crisis manager directs a doctor to reach all wounded people and save their lives. In this experiment, the subject plays the role of the crisis manager and his/her task is to save as many wounded victims as possible. 2.1 Presentation Material For each victim report, two types of information can be provided: basic information and additional aid. The basic information includes the type of the victim (wounded or dead) and its location. The additional aid reduces the searching area by indicating which half of the screen (left or right) contains this victim. To convey these two types of information, we selected four modalities based on their visual/auditory and verbal/nonverbal properties: text (visual, verbal), image (visual, nonverbal), speech (auditory, verbal) and sound (auditory, nonverbal). The basic information can be efficiently conveyed by locating a visual object on a map. We use text or image to present a victim (see Fig. 1), and the cell it occupies on a grid-based map indicates the location of the victim (see Fig. 2). The additional aid can be presented by text (‘left’ or ‘right’), image (a left arrow or a right arrow), speech (‘left’ or ‘right’) or sound (an ambulance sound coming from the left or the right speaker). Previous studies suggest that the categorization and understanding of concrete objects are slower when they are presented by text than by image [2, 7, 14]. Therefore, in order to better observe the benefit of the additional aid, text is used to present the basic information if additional aids are given. Finally, five experimental conditions were selected (see Table 1).

Towards Cognitive-Aware Multimodal Presentation

5

Table 1. Five experimental presentation conditions Index 1 2 3 4 5

Basic Information Text Image Text Text Text

Additional aid None None Image Speech Sound

Fig. 1. Text and image presentations

Modality properties Visual, verbal Visual, nonverbal Visual + visual, verbal + nonverbal Visual + auditory, verbal + verbal Visual + auditory, verbal + nonverbal

Fig. 2. Sample of the grid-based map (partial) of the victim type

2.2 Task The subject played the role of the crisis manager. The task was to send the doctor to each patient by mouse-clicking on the presentation (text or image). A new patient appeared with a random interval between 2 to 5 seconds, usually at the same time as one or more dead victims. A patient had a life time of 10 seconds and would turn into a dead victim without a timely treatment. A number above the presentation of a patient indicated his remaining life time. When timely treated, patients disappeared from the screen. In each trial, 100 patients were presented in about 5 minutes. The difficulty of the task could be regulated by the number of distracters (dead victims). At the beginning of a trial, there was no object on the grid map and the task was relatively easy. As the number of dead victims grew, it became more and more difficult to identify a patient in the crowded surroundings. The task difficulty reached the maximum (about 40% of the cells contained objects) after about 150 seconds and remained unchanged for the rest of the trial (see Fig. 3). 2.3 Measurements Three categories of measurements were applied, namely performance, subjective and physiological measurements. The performance in each trial was evaluated by three variables: 1) the average reaction time to click on a patient (in seconds), 2) the total number of patients that were not treated within 10 seconds and 3) the time stamp when the first patient died in a trial (in seconds). The NASA Task Load Index [9] was used to obtain subjective reports on cognitive workload and stress. A 20-level rating, from very low (1) to very high (20), was performed on cognitive workload and stress, respectively. In order to further assess cognitive workload from the physiological

6

Y. Cao, M. Theune, and A. Nijholt

states, we recorded the electrocardiograms, galvanic skin conductance and respiration during the experiment. Scientific literature suggests that when the cognitive demand increases, the heart rate increases, the heart rate variability decreases, the skin conductivity increases and the respiration rate increases [5, 10, 15, 17]. 2.4 Subjects and Procedure 20 university students (bachelor, master or Ph.D.) volunteered to participate in this experiment. After entering the lab, the experimenter first applied the physiological sensors to the participant, while he/she was listening to soothing music. When the sensors were set, an additional resting period of 5 minutes was given and then the baseline physiological state was recorded for 5 minutes. Afterwards, the participant received an introduction to the experiment and performed a short training session in order to get familiar with the task and presentation conditions. Finally, the participant performed the five experimental trials with a counterbalanced order. A 5 minutes break was placed between each two successive trials. The subjective ratings on cognitive load and stress were conducted during the breaks. The whole experimental procedure lasted for about 80 minutes.

3 Results Due to our experimental design, we applied repeated-measure ANOVAs on the dependent measurements, with modality as a within-subject factor. Results from the three categories of measurements are presented in this section. 3.1 Performance Measurements ANOVA results showed a significant modality effect on all the three performance measurements. First, modality had an effect on the average reaction time, F (2.87, 54.51) = 12.76, p