Funded Research Projects commencing 2000
Principal Supervisor: Dr Paul Vickers
Title:Multi-modal human-computer interaction
Research Environment:
A student bursary of ?6,764 is available to fund a three-year
full-time studentship in "Multi-modal Human-Computer Interaction".
Working in the Distributed Multimedia Systems research group within
the School of Computing & Mathematical Sciences, you will be required
to undertake supervised research for a higher degree (MPhil/PhD).
Full fee remission is available to citizens of EU countries.
Candidates should hold or expect to obtain a good honours degree in a
relevant discipline. Ideally, candidates will also possess computer
programming skills. Candidates for this project should have interests
and/or experience in areas such as multimedia technology, computer
audio, visualisation, programming, and human-computer interaction.
Project Description:
Computers now offer multiple media output facilities that are able to
provide information simultaneously in both the auditory and visual
channels. Whilst the visual channel has been extensively developed
(for example graphical user interfaces (GUI) and Virtual Reality), the
auditory channel is largely unexplored. This is unfortunate for four
reasons:
- the auditory sense is an important one used extensively in everyday
life
- visual interfaces are now very crowded and are often confusing
- auditory output offers parallelism and a “free hands” approach
- for visually impaired users, exploitation of the auditory channel is
essential
The first three reasons encourage combination of output from both
channels to provide richer and more relevant information for users in
complex situations where the visual medium is stretched. The fourth
reason arises from the ubiquitous nature of the Graphical User
Interface which has severely challenged the visually impaired.
Recent research has tended to focus either on visualisations or on
auditory displays. Some attempts have been made to compare visual and
auditory mappings, but these studies have only looked at one or two
interaction tasks each. Furthermore, almost no attempt has been made
to specify guidelines for how best to employ multi-modal displays in
human-computer interaction (HCI) tasks.
We want to investigate the use of visual and auditory representations
and data mappings across a range of interaction tasks (for example GUI
operation). The aim of the research will be to find out what can and
cannot be represented using visual and auditory mappings and then to
determine what modality offers the better mappings where either can be
used. From this it can be determined whether the use of sound and
vision together complement each other (leading to increased
effectiveness/efficiency of the interface), or result in a perceived
loss of information (leading to decreased efficiency and
effectiveness).
The first stage of the work will be a critical review of the existing
techniques and tools together with a review of the cognitive and
psycho-physical aspects of visual and auditory displays. Next,
empirical evaluations will be carried out to answer the main research
questions. Where necessary, new auditory and visual mappings will be
created using existing or bespoke tools. Finally, from the empirical
evidence gathered, we propose to produce guidelines for interface
designers on how to successfully use both visual and auditory
mappings.
Candidates for this project should have interests in areas such as
multimedia technology, computer audio, visualisation, programming, and
human-computer interaction.
Project Team
Dr Paul Vickers (Principal Investigator)
Dr David England
Dr Madjid Merabti
Scientific & Technological Quality
This research is in the area of human-computer interaction (HCI), with
elements in the field of auditory display. It has links with
psychology and cognitive science. The novel scientific issue being
addressed is the determination of guidelines for the effective use of
auditory and visual display techniques for improving the effectiveness
and efficiency of human-computer interaction. This will probably be
limited to a specific domain (such as GUI operation or virtual
immersive environments).
Auditory display research has shown that audio can be used
successfully in various interaction tasks (such as GUI menu navigation
[4], drag-and-drop [3], and even program debugging [8]). The
development of the GUI has shown that graphics can be used as part of
the user interface, and some have shown how graphics can be used to
provide visualisations of programs [2]. However, comparisons of the
modalities are scarce (and too often experimentally flawed or based on
poor mappings, e.g. [6]) and so a good set of guidelines for
multi-modal interaction design is required.
This project will build on previous work done by the project team and
other members of the School, thus further developing the
multimedia/HCI research profile in CMS.
This research could also bring about several wider social benefits:
- Allow problems to be explored in new ways (e.g. using the temporal
aspects of music)
- Reduce crowding on visual displays
- Improved effectiveness & efficiency in HCI
- Creating new market opportunities for UK firms
- Ability to Undertake the Research
The Principal Investigator has a good grounding in using sound in the
interface and in its empirical evalution and so can provide
appropriate support and guidance. Dr England’s work in interface
design and virtual reality will inform the visual aspects of the
project. His supervisory experience brings valuable project management
skills to the work. Dr Merabti has worked extensively in distributed
multimedia systems and has wide supervisory experience.
Viability & Planning
Because this is an empirical project it will draw on established
research techniques and experimental designs that have been used
successfully for many years in related disciplines.
Project Plan
The project has three main activities.
1. Review of existing mapping techniques and the associated cognitive
issues. This will require a thorough literature review of the
technical aspects. A search of the psychology literature will also be
needed to ensure the cognitive and psycho-physical aspects have been
covered (for instance, see Moore [7]).
2. Empirical evaluation and comparison of auditory and visual
mappings. Drawing on our experience of conducting experiments to
evaluate auditory mappings for program debugging [1, 8, 10],
experimental designs will be specified to allow a) evaluation of
visual mappings, b) evaluation of auditory mappings, and c) evaluation
of complementary visual-auditory mappings.
3. Specification of guidelines for good use of auditory and visual
display techniques in HCI applications. Brewster [5] laid down some
guidelines for using earcons in interfaces, whilst we have proposed
some further guidelines for using musical structures in program
auralisation [8, 9]. The HCI literature has many examples of design
principles for GUIs. The task is, for a given domain (perhaps GUI
operation), to set out some design guidelines for the successful
incorporation of both sound and vision into the interface.
The Investigators will provide expertise to the project with the RA
undertaking the bulk of the three main tasks.
Year 1 of the project will involve literature review and detailed
specification of the questions to be addressed. This will form the
basis of an external publication and the MPhil to PhD transfer report.
Project tasks 1 & 2 (above) are expected to dominate Years 1 and 2.
Experimental results will be collected and analysed, and will provide
material for external publication. The first half of Year 3 will
involve evaluation of the work-to-date leading to specification of the
guidelines. This will be sufficient to allow the thesis to be written
up and the outcomes submitted for journal publication.
Relevance to Beneficiaries
This project will directly support the research of the Principal
Investigator and will add to the HCI research currently being carried
out in the School. Much of the work in this area is in the early
stages of development. Therefore, successful completion of this
project will put us in a good position in the field. The project
outcomes can then be exploited in the form of publications and
external funding bids. Furthermore, the project will help to
consolidate the growing reputation that CMS has for multimedia
teaching and research. An additional RA for this project will help the
synergy of our HCI research and will provide a sound basis for further
research into multi-modal interaction.
Resources & Cost Effectiveness
? Student bursary
? Equipment needs are modest as most of the development and evaluation
can be done with a MS-Windows PC. Use could also be made of the School
’s Silicon Graphics machines. Some specialist sound card hardware may
be required if current software synthesis technologies turn out to be
unsuitable. Some extra software may be needed as the project
progresses. Use can be made of existing WWW/Internet media
technologies for maximum impact.
Exploitation of Research
The composition of the next RAE panel indicates that multimedia/HCI
will feature strongly in future research. This is a growing area that
is sure to see increased funding from the research councils. The
principal investigator has already received a University Research
Fellowship to support work in musical program auralisation and is
currently in the process of preparing a joint bid with his ex-PhD
supervisor for EPSRC funds to continue their work.
References
[1] Alty, J. L. and Vickers, P., The CAITLIN Auralization System:
Hierarchical Leitmotif Design as a Clue to Program Comprehension, in
Proc. ICAD '97 Fourth International Conference on Auditory Display
(Palo Alto, November 3-5, 1997), Xerox PARC, Palo Alto, CA 94304, pp.
89-96
[2] Baecker, R., DiGiano, C. and Marcus, A., Software Visualization
for Debugging, Communications of the ACM, 40 (4), 1997, pp. 44-54.
[3] Brewster, S., Sonically Enhanced Drag and Drop, in Proc. ICAD '98
Fifth International Conference on Auditory Display (Glasgow, Nov 1-4,
1998), British Computer Society ISBN 1 902505 05 0
[4] Brewster, S., Using Non-Speech Sounds to Provide Navigation Cues,
ACM Transactions on Computer-Human Interaction, 5 (3), 1998, pp.
224-259.
[5] Brewster, S. A., Wright, P. C. and Edwards, A. D. N.,
Experimentally Derived Guidelines for the Creation of Earcons, in
Proc. HCI '95 (adjunct proceedings) (Huddersfield, 1995)
[6] Lodha, S. K., Whitmore, D., Hansen, M. and Charp, E., Analysis and
User Evaluation of a Musical-Visual System: Does Music Make Any
Difference?, in Proc. ICAD 2000 Sixth International Conference on
Auditory Display (Atlanta, GA, 2-5 April, 2000), International
Community for Auditory Display, pp. 167-172 ISBN 0-9670904-1-5
[7] Moore, B. C. J., An Introduction to the Psychology of Hearing.
London: Academic Press, 1997.
[8] Vickers, P., CAITLIN: Implementation of a Musical Program
Auralisation System to Study the Effects on Debugging Tasks as
Performed by Novice Pascal Programmers, Ph.D. Thesis, Loughborough
University, Loughborough, 1999.
[9] Vickers, P. and Alty, J. L., Towards some Organising Principles
for Musical Program Auralisation, in Proc. ICAD '98 Fifth
International Conference on Auditory Display (Glasgow, Nov 1-4, 1998),
British Computer Society ISBN 1 902505 05 0
[10] Vickers, P. and Alty, J. L., Musical Program Auralisation:
Empirical Studies, in Proc. ICAD 2000 Sixth International Conference
on Auditory Display (Atlanta, GA, 2-5 April, 2000), International
Community for Auditory Display, pp. 157-166 ISBN 0-9670904-1-5
Informal Enquiries
Telephone: +44 (0) 151 231 2283
E-mail: P.Vickers@LIVJM.AC.UK
|
