In my last blog post I presented an overview about my research within the eHealth domain. In this blog post I will do the same thing, but for my other main research field – multimodal interaction in virtual environments.
What have I done related to multimodal interaction?
Even though I have spent the last couple of years focusing mainly on eHealth, I have done a lot of research – especially as a Ph.D. student at the Royal Institute of Technology – related to multimodal interaction. Most of this research has been focused on multimodal learning environments for collaborative task solving between sighted and visually impaired persons. Haptic feedback has played a major part in the collaborative virtual environments that I have designed and evaluated both in lab settings and in the field in e.g. primary schools. Quite a while ago, I wrote a blog series on haptic feedback focusing on the work I performed within the scope of my doctoral studies. Here are the links to those posts:
- Haptics as an interaction modality
- Haptic feedback in medical applications
- Haptic interaction research: one great memory!
- Haptic feedback in games
- Haptic communicative functions
During my time as a postdoc at Uppsala University, I also performed some activities related to multimodal interaction. Most of this time I devoted to research grant applications and I also wrote a few conference papers. You can read a short summary of these activities here.
In total, my research on multimodal interaction has, up until today, resulted in the following five journal publications (some links lead to open access publications or pre-prints):
- Haptic feedback combined with movement sonification using a friction sound improves task performance in a virtual throwing task. Journal of Multimodal User Interfaces (Authors: Emma Frid, Jonas Moll, Roberto Bresin and Eva-Lotta Sallnäs Pysander)
- A haptic tool for group work about geometrical concepts engaging blind and sighted pupils. ACM Transactions on Accessible Computing (Authors: Jonas Moll and Eva-Lotta Sallnäs Pysander)
- The Effects of Audio and Haptic Feedback on Collaborative Scanning and Placing. Interacting with Computers (Authors: Jonas Moll, Eva-Lotta Sallnäs Pysander, Kerstin Severinson Eklundh and Sten-Olof Hellström)
- Auditory feedback in haptic collaborative interfaces. International Journal of Human-Computer Studies (Authors: Yingying Huang, Jonas Moll, Eva-Lotta Sallnäs Pysander and Yngve Sundblad)
- Audio makes a difference in haptic collaborative virtual environments. Interacting with Computers (Authors: Jonas Moll, Yingying Huang and Eva-Lotta Sallnäs Pysander)
and the following 11 conference papers (some links leads to open access publications or pre-prints):
- Haptic communicative functions and their effects on communication in collaborative multimodal virtual environments. Swedish Cognitive Science Society Conference 2017 (Authors: Jonas Moll and Eva-Lotta Sallnäs Pysander)
- Using eye-tracking to study the effect of haptic feedback on visual focus during collaborative object managing in a multimodal virtual interface. Swedish Cognitive Science Society Conference 2017 (Authors: Jonas Moll, Emma Frid)
- An exploratory study on the effect of auditory feedback on gaze behavior in a virtual throwing task with and without haptic feedback. Sound and Music Computing 2017 (Authors: Emma Frid, Roberto Bresin, Eva-Lotta Sallnäs Pysander and Jonas Moll)
- Sonification of haptic interaction in a virtual scene. Sound and Music Computing Sweden 2014 (Authors: Emma Frid, Roberto Bresin, Jonas Moll and Eva-Lotta Sallnäs Pysander)
- Design and evaluation of interaction technology for medical team meetings. INTERACT 2011 (Authors: Alex Olwal ,Oscar Frykholm, Kristina Groth and Jonas Moll)
- Pointing in multi-disciplinary medical meetings. Computer-Based Medical Systems 2011 (Authors: Eva-Lotta Sallnäs Pysander, Jonas Moll, Oscar Frykholm, Kristina Groth and Jonas Forsslund)
- Communicative Functions of Haptic Feedback. Haptic and Audio Interaction Design 2009 (Authors: Jonas Moll, Eva-Lotta Sallnäs Psyander)
- Designing a collaborative learning tool for collaboration between visually impaired and sighted pupils. NordiCHI 2008 (Author: Jonas Moll)
- Integrating Audio and Haptic Feedback in a Collaborative Virtual Environment. International Conference on Human-Computer Interaction 2007 (Authors: Yingying Huang, Jonas Moll, Eva-Lotta Sallnäs Pysander and Yngve Sundblad)
- Group work about geometrical concepts among blind and sighted pupils using haptic interfaces. World Haptics 2007 (Authors: Eva-Lotta Sallnäs Pysander, Jonas Moll, Kerstin Severinson Eklundh)
- Haptic interface for collaborative learning among sighted and visually impaired pupils in primary school. Enactive 2006 (Authors: Eva-Lotta Sallnäs Pysander, Jonas Moll, Kerstin Severinson Eklundh)
My ongoing research within multimodal interaction
Currently, there is not much going on related to this research field (at least not in my own research). The only ongoing activity I’m engaged in is an extensive literature review related to communication in collaborative virtual environments which will lead to a theoretical research article where I will discuss different technical solutions for haptic communication in the light of the research I have performed within the area up until today. I’m collaborating with my former Ph.D. supervisor Eva-Lotta Sallnäs Pysander on this activity. I hope that this research activity will help me in my continued research on collaboration between visually impaired and sighted pupils based on different types of tasks and learning material.
Upcoming research on multimodal interaction
As I wrote in a recent blog post multimodal interaction, with a focus on haptic feedback, seems to be a new research area at the Centre for empirical research on information systems (CERIS) where I just stared my assistant professorship. Thus, this is the research area in which I can contribute with something new to the department. An area that is already represented at the department, however, is “Information Technology and Learning”, which seems to be a perfect fit in this case!
Last year, I also submitted a research grant application focusing on continued work with collaborative multimodal learning environments. Unfortunately, that one was rejected but no one is giving up. I will work somewhat on revising the application during the autumn and submit as soon as a suitable call pops up. Maybe I will also have additional co-applicants from the CERIS department by then.