Category: Multimodality

eHealth · Haptics · Medical Records Online · Multimodality · Pedagogy

Publications from the first half of my assistant professorship at Örebro University

Jonas MollLeave a comment

In my previous three blog posts I have written about my research on 1) eHealth services for patients, 2) accessible digital collaborative learning environments and 3) teaching in higher education. This blog post collects the publications I have worked on in these areas during the last two years. Most of the articles are published with the open access option, meaning that you can read the entire articles by following the links in the lists below.

eHealth services for patients

Most of the publications from my first two years at Örebro University have focused on eHealth services for patients. The patient accessible electronic health record solution in Sweden, Journalen, has been the primary focus. These are the published journal articles that I have worked with in this area:

  • Huvila, I., Cajander, Å., Moll, J., Enwald, H., Eriksson-Backa, K., and Rexhepi, H. (2021). Technological and informational frames: explaining age-related variation in the use of patient accessible electronic health records as technology and information. Information Technology & People. DOI: 10.1108/ITP-08-2020-0566. [Link to open access publication]. This article is one of many articles that is based on a national patient survey where patients were asked questions about their attitudes towards and use of Journalen. See this blog post for more info on this publication.
  • Moll, J., and Cajander, Å. (2020). Oncology health-care professionals’ perceived effects of patient accessible electronic health records 6 years after launch: A survey study at a major university hospital in Sweden. Health Informatics Journal (Vol 26, No 2). pp: 1392-1403. [Link to open access publication]. This article was also based on a survey, but this time around the respondents were physicians and nurses with a specialization in oncology. See this blog post for more info on this publication.
  • Nurgalieva, L., Cajander, Å., Moll, J., Åhlfeldt, R-M., Huvila, I., and Marchese, M. (2020). ‘I do not share it with others. No, it’s for me, it’s my care’: On sharing of patient accessible electronic health records. Health Informatics Journal. DOI: 10.1177/1460458220912559. [Link to open access publication]. This article was also based on the national patient survey, as well as interviews with cancer patients. I realize now that I never wrote any blog post about this specific publication. I will write such a post later on during this autumn.
  • Rexhepi, H., Moll, J., and Huvila, I. (2020). Online electronic healthcare records: Comparing the views of cancer patients and others. Health Informatics Journal. DOI: 1460458220944727. [Link to open access publication]. In this article, which was also based on results from the national patient survey, we looked specifically at differences between answers from cancer patients and the other respondents. See this blog post for more info on this publication.

The following conference papers were also produced during the period:

  • Moll, J., and Cajander, Å. (2020). On Patient Accessible Electronic Health Records and the Experienced Effect on the Work Environment of Nurses. Studies in Health Technology and Inforamtics (Vol. 270). pp. 1021-1025. [Link to open access publication]. This paper is based on preliminary results from a large interview study with oncology healthcare professionals. I have not written any blog post about this publication, but I will do that later on this autumn.
  • Moll, J., and Rexhepi, H. (2020). The Effect of Patient Accessible Electronic Health Records on Communication and Involvement in Care-A National Patient Survey in Sweden. Studies in Health Technology and Informatics (Vol. 270). pp. 1056-1060. [Link to open access publication]. This paper is also based on the results from the national patient survey. Even in this case, I will write a separate blog post later during the autumn.
  • Rexhepi, H., Moll, J., Huvila, I., and Åhlfeldt, RM. (2020). Do you want to receive bad news through your patient accessible electronic health record? A national survey on receiving bad news in an era of digital health. Proceedings of the Eighteenth International Symposium for Health Information Management Research (Kalmar, Sweden, September 17-18). pp. 169-178. This one is also based on results from the national patient survey and was among the highest rated papers after the review rounds. As a result, we were provided the possibility to get our contribution published in a special issue of Health Informatics Journal. I will write a separate blog post about that article, which was published last week.

Accessible digital collaborative learning environments

This heading is listed here mostly for completeness. The Vinnova project I have written about many times before has led to many interesting ideas for publications, and work has started on some of them. We are still on the manuscript stages, so none of these have been published yet. An article based on a project from my last year at the Royal Institute of Technology was however published during autumn 2019 (no learning environment in focus, but combinations of visual, haptic and audio feedback still played important roles):

  • Frid, E., Moll, J., Bresin, R., & Pysander, E. L. S. (2019). Haptic feedback combined with movement sonification using a friction sound improves task performance in a virtual throwing task. Journal on Multimodal User Interfaces, 13(4), 279-290. [Link to open access publication].

Teaching in higher education

I still don’t have any journal articles related to this area, but I do have some conference papers. Quite a few journal articles related to this area of research are however planned for the second half of my assistant professorship. Some of the conference contributions were published during these last two years:

  • Moll, J., and Josefsson, P. (2020). Communication patterns among students and teachers when using Facebook in a university course. Proceedings of the 14th annual International Technology, Education and Development Conference (INTED 2020) (Valencia, Spain, March 2020). This full paper is based on an analysis of the interaction among students and between students and teachers in a teacher-administrated Facebook group that I used in my course in interaction design at Uppsala University. See this blog post for more info on this publication.
  • Wistrand, K., Moll, J., Hatakka, M. and Andersson, A. (2020). Improving Writing Skills Among Information Systems Students: Guidelines for Incorporating Communication Components in Higher Education. Proceedings of the 2020 IEEE Frontiers in Education Conference (FIE 2020), (Uppsala, Sweden, October 21-24), IEEE. This full paper is based on an analysis of two different ways of introducing scientific writing components in higher education. This one will also get a dedicated blog post during the autumn.
Haptics · Human-Computer Interaction · Multimodality · sonification · Thesis defense

Recently attended Emma Frid’s thesis defense at KTH!

Jonas MollLeave a comment

Disputationsbild_Emma

On Friday, January 10, I attended Emma Frid’s thesis defense at KTH. Emma and I collaborated in a research project a few years ago, and one of the major outcomes was this open access article presenting the results of an experiment with a multimodal interface including both haptic feedback and two different sonification models. Emma’s thesis work relates heavily to the research field of Sound and Music Computing (also the name of a sub-group at the department of Media technology and Interaction Design at KTH where I worked for more than a decade), and focuses specifically on (accessible) digital music instruments and interfaces. The main research question is “How can music interfaces be designed for inclusion?”. The thesis “Diverse Sounds – Enabling Inclusive Sonic Interaction” can be found here. The main supervisor was professor Roberto Bresin and co-supervisor was professor Eva-Lotta Sallnäs Pysander. Both of them work at the department of Media technology and Interaction Design at KTH. The opponent was Reader Andrew McPherson from the school of Electronic Engineering and Computer Science at Queen Mary University of London. The examination committee consisted of senior researcher Elaine Chew from the French National Centre for Scientific Research and the Music Representations Team at the Institute for Research and Coordination in Acoustics/Music, professor Rolf Inge Godøy from the Centre for Interdisciplinary Studies in Rhythm, Time and Motion at University of Oslo, associate professor Dan Overholt from the department of Architecture, Design and Media Technology at Aalborg University and professor Henrik Frisk from the department of Composition, Conducting and Music Theory at Royal College of Music in Stockholm.

After the introduction by associate professor Madeline Balaam, who chaired the event, the opponent held a presentation about Emma’s thesis for about 45 minutes. It was a very good presentation and it was interesting to listen to his interpretation of the work performed. He concluded his presentation by discussing the nine properties that according to Emma should be considered when designing accessible digital music instruments; expressiveness, playability, longevity, customizability, pleasure, sonic quality, robustness, multimodality and causality (see the thesis for a thorough coverage of these properties and the work that gave rise to them). After a short break the opponent, and later on the members of the committee, asked question that formed a good foundation for interesting discussions about the thesis work.

I think Emma did a really good job answering the questions and discussing her work. She elaborated a lot on the themes that were brought up to discussion and it was very clear that she knows a lot about this research field. She was also calm during the entire process and even helped out when the opponent and committee members e.g. needed headsets and/or microphones. One thing that was special about this defense is that the opponent, as well as all members of the grading committee, began their round of questions by congratulating Emma on the excellent job that she has performed! I have not seen that during other defenses I have attended. The defense was rounded off by a very long applaud – it was almost as if the audience expected some kind of extra performance on stage.  🙂

communication · Grant application · Group work · Haptics · Multimodality

Got funding from Vinnova!

Jonas Moll6 Comments

Vinnova_funded

A few months ago I wrote in this blog post that a funding application, where I represented Örebro University as one of the co-applicants, had been submitted to Vinnova (Sweden’s government agency for innovation). And guess what, the project got funded! The name of the project is “Virtual environments supporting group work between sighted and visually impaired pupils”, and as the name suggests we will work closely with visually impaired and sighted pupils (as well as teachers) to develop new virtual learning environments that support collaboration a lot better than today’s special equipment used by visually impaired pupils in schools. My former supervisor at the Royal Institute of Technology, Eva-Lotta Sallnäs Pysander is the project leader and Örebro University, Axess Lab (a company division focusing on digital accessibility from numerous perspectives), The Swedish Association of the Visually Impaired and The National Agency for Special Needs Education and Schools (SPSM) are the other project partners. I will introduce the other project members and write more about the project as soon as press releases have been published by the Royal Institute of Technology and Örebro University.

This project is very special to me for several reasons. First and foremost, this is the very first time I have been contributing extensively to a project application that has resulted in external funding. Writing these kinds of applications requires a lot of work and collaboration between researchers and other stakeholders and it feels great when the efforts finally pay off! I’m also very happy about that I’m now able to focus more on the research field “IT and learning” again. It was quite a while ago that I worked on multimodal learning environments. The research area is really important and I’m glad that Vinnova has acknowledged that. Another reason why this project is special to me is that I started to work on the application shortly after I had started working as an assistant professor at Örebro University – one of the very first things I did within the scope of my new position actually resulted in external research funding! I think this is a very good start for me and I’m really excited about this new project.

If you want to read a short summary of the project, you can visit this page where Vinnova has published some short descriptions about the purpose, approach and expected results. I will write more in a later blog post as soon as I have some university press releases to link to. So stay tuned!  🙂

 

Grant application · Group work · Multimodality

New project grant application submitted to Vinnova!

Jonas Moll3 Comments

Vinnova_submitted

During the last couple of weeks I have worked a lot on a research grant application to Vinnova (call: Digital tools) and today it was finally submitted! The application is based on a collaboration between the following partners, who all apply for funding in the application:

The proposed project is based on earlier proof-of-concept work that I performed at KTH regarding multimodal learning environments supporting collaboration between sighted and visually impaired pupils. I will of course be able to say a lot more about the involved partners and the content of the project after Vinnova has reviewed the application (in the middle of November). I really hope that the project will be funded since I really miss working within this research area, and I also think that the research is important for society.

One confirmation of that both our old and planned research in this area is considered important is that representatives from both the Swedish Association of the Visually Impaired and the National Agency of Special Needs Education and Schools decided to take active part in the project (in case it gets funded). I’m sure they would not do that if they didn’t identify a high importance for society, and especially the main target group, as well as an importance on a national level. Even if the project doesn’t get funding we should definitely make sure to continue the collaboration and discussions with the involved partners – there are always new opportunities for research up ahead and as long as clear mutual benefits can be identified the collaboration should move on.

Stay tuned for more!  🙂

 

communication · design · Group work · Haptics · Human-Computer Interaction · Multimodality · sonification

Overview of my research within multimodal interaction

Jonas Moll1 Comment

thesis

In my last blog post I presented an overview about my research within the eHealth domain. In this blog post I will do the same thing, but for my other main research field – multimodal interaction in virtual environments.

 

What have I done related to multimodal interaction?

Even though I have spent the last couple of years focusing mainly on eHealth, I have done a lot of research – especially as a Ph.D. student at the Royal Institute of Technology – related to multimodal interaction. Most of this research has been focused on multimodal learning environments for collaborative task solving between sighted and visually impaired persons. Haptic feedback has played a major part in the collaborative virtual environments that I have designed and evaluated both in lab settings and in the field in e.g. primary schools. Quite a while ago, I wrote a blog series on haptic feedback focusing on the work I performed within the scope of my doctoral studies. Here are the links to those posts:

During my time as a postdoc at Uppsala University, I also performed some activities related to multimodal interaction. Most of this time I devoted to research grant applications and I also wrote a few conference papers. You can read a short summary of these activities here.

In total, my research on multimodal interaction has, up until today, resulted in the following five journal publications (some links lead to open access publications or pre-prints):

and the following 11 conference papers (some links leads to open access publications or pre-prints):

 

My ongoing research within multimodal interaction

Currently, there is not much going on related to this research field (at least not in my own research). The only ongoing activity I’m engaged in is an extensive literature review related to communication in collaborative virtual environments which will lead to a theoretical research article where I will discuss different technical solutions for haptic communication in the light of the research I have performed within the area up until today. I’m collaborating with my former Ph.D. supervisor Eva-Lotta Sallnäs Pysander on this activity. I hope that this research activity will help me in my continued research on collaboration between visually impaired and sighted pupils based on different types of tasks and learning material.

Upcoming research on multimodal interaction

As I wrote in a recent blog post multimodal interaction, with a focus on haptic feedback, seems to be a new research area at the Centre for empirical research on information systems (CERIS) where I just stared my assistant professorship. Thus, this is the research area in which I can contribute with something new to the department. An area that is already represented at the department, however, is “Information Technology and Learning”, which seems to be a perfect fit in this case!

Last year, I also submitted a research grant application focusing on continued work with collaborative multimodal learning environments. Unfortunately, that one was rejected but no one is giving up. I will work somewhat on revising the application during the autumn and submit as soon as a suitable call pops up. Maybe I will also have additional co-applicants from the CERIS department by then.

communication · eHealth · Group work · Haptics · Human-Computer Interaction · Medical Records Online · Multimodality · Social media in higher education

Today I start working at Örebro University as an assistant professor!

Jonas Moll1 Comment

Fjäll7

Today I can finally take the next step on the academic ladder, since I’m starting up my new job as assistant professor in Informatics at the centre for empirical research on information systems at Örebro University! In November 2018 I applied for the position and in the middle of the spring 2019 I was called to an interview. A few months later I was offered the position. I’m very excited about this great opportunity and I of course intend to make the most out of it. After a very long blog break (mostly due to health issues and the fact that my research efforts during the spring has been rather minor), this also seems like a good opportunity to start posting again.

The assistant professorship is the first step on the so called tenure track. It is an academic position limited to four years, but the intention is often (as in my case) to promote the assistant professor to an associate professor towards the end – a position which is not limited in time. My job includes 70% research and 30% teaching, which is quite common for assistant professorships. I’m not sure yet where e.g. service and communication (like administration, blogging and interviews) fits in.

The job as assistant professor in Informatics is a very good fit for me, since I will be able to continue to work with all my main research interests (the main theme is computer supported communication):

  • eHealth
    – I will continue looking at how patient accessible electronic health records (PAEHR) affect the communication between patients and care professionals. One thing I’m particularly curious about, and that is actually the focus of a project grant application currently in review, is how one can incorporate the PAEHR as a communication mediator curing doctor-patient meetings. Another application in review is about the effects and implementation of psychiatry records online.
  • Multimodal interaction
    – I will also continue looking at how multimodal feedback (especially haptics and sound) can be used to promote collaboration between sighted and visually impaired pupils/students in group work. Most of today’s assistive technologies that are used in school settings are not adapted for collaboration and this is highly problematic when it comes to inclusion of visually impaired pupils/students in group work settings.
  • Social media in higher education
    – My intention is also to continue investigating how social media like Twitter and Facebook can be used as supplementary communication channels in higher education courses.

When it comes to the areas of eHealth and social media in higher education, research is already being conducted by my new colleagues at Örebro University. Multimodal interaction would however be a new research theme for the department. I will elaborate on the different themes listed above in later blog posts as work is progressing. Other research themes from the department (like computer security and ICT for development) could also be added.

I have not heard anything yet regarding the teaching, but given the department’s focus I guess I could be involved in master’s thesis supervision, human-computer interaction project courses and programming courses. I will write more about the teaching part when I know more.

The blog image that I used for this post is one of my own – I took it a few weeks ago during a week I spent in Abisko in northern Sweden.

eHealth · Grant application · Multimodality

Looking forward to a spring of opportunities!

Jonas Moll2 Comments

Spring_2018

This will be a very special year for me, since my postdoc period at Uppsala University will end in September and I currently have no idea what will happen after that. I may be able to find a way to continue my work in Uppsala, but that is far from given. But, as the heading of this post suggests (as well as my picture), there are a bunch of possibilities up ahead. I now have several project ideas and the same goes for my current colleagues at Uppsala University and my former colleagues at KTH. On top of this, there are two interesting assistant professorship and three associate professorship jobs to apply for!

When it comes to open positions, all I have found are in the area of healthcare/eHealth (two of these, from Uppsala University, are technically about information systems in general, but can easily be angled towards health systems). One of the open positions belong to KTH, two to Uppsala University and two to Örebro University. I will write more about these when I have applied for the respective positions. The downside here is that I will probably not be able to work with multimodal interaction if given one of these positions, but on the other hand eHealth is an area that I am really interested in.

Even though it would be great to get a semi-secure assistant professorship, getting research projects would be even more interesting since these often run for four years (same duration as a regular assistant professorship). Research projects which you define yourself, in collaboration with colleagues, are probably also even more in line with your main interests. The current plan is that I will be the main applicant on one project application to Forte (Swedish Research Council for Health, Working Life and Welfare) and co-applicant in one project application to the VR (Swedish Research Council), one project application to Vinnova (the challenge driven innovation call) and two project applications to AFA Försäkringar (AFA Insurance). On top of that I will also be co-applicant on a research program application to Riksbankens Jubileumsfond and one grant application for Interdisciplinary Research Environments to VR. Four of the applications concern collaboration in multimodal environments and three concern eHealth systems. The ideal result here would be to get one project/program in each research area, since that would make it possible for me to continue working with my two favorite topics. One can always hope…

If given the choice I would pick the research program and the interdisciplinary research environment. By the way, fantastic things happen in the world of research applications from time to time – during autumn 2016 the researcher in charge of our HTO group at Uppsala University, Åsa Cajander, got three project grants during one week! One side effect was that our research group moved to a larger office area to make room for new collaborators. If we get that lucky this time around I guess they might have to build a new house for us… 😉

communication · conference · Group work · Haptics · Multimodality

Paper and poster about haptic communicative functions and their effects on communication in collaborative virtual environments

Jonas Moll1 Comment

HaptiCom

Yesterday I blogged about a poster and a conference paper that Emma Frid and I developed for the SweCog conference in Uppsala. In this post I will focus on the second poster and paper that Eva-Lotta Sallnäs Pysander and I developed for the same conference.

The poster shown in the picture above, and even more so the paper, summarizes some of the main points made from my doctoral studies. My main focus during those years was collaboration in multimodal virtual environments with special emphasis on how haptic feedback can be used for communicative purposes. Mediated haptic communication has been studied for quite some time, but my specific contribution here has been to develop and test new functions for two-way haptic communication (see short descriptions of the functions on the poster) and also adapt some already developed ones in order to make them work better in a situation when a sighted person is collaborating with a severely visually impaired one in a collaborative virtual environment. There is a real potential in these kinds of functions when it comes to collaboration between sighted and visually impaired – the haptic feedback does not only enable establishment of a common ground about the interface but also effective two-way communication (see examples of results on the poster above). This is very important for the inclusion of visually impaired persons in group work. The example study is reported in much more depth in this article.

Even though the poster and paper include summaries of work already performed and reported, we are in this case even more explicit about the connection to other kinds of haptic communicative functions. This conclusion also takes the work to the next level:

We argue that for effective collaboration and communication to take place in virtual environments by means of haptic feedback the haptic functions need to be designed as to allow for reciprocal exchange of information. That is, both users need continuous feedback from each other during e.g. a guiding process or joint object handling.

The conference paper, on which the above poster is based, can be found here.

conference · Haptics · Multimodality

Paper and poster about gaze behaviour during collaborative object managing

Jonas Moll2 Comments

Eyetracking

As I wrote in an earlier blog post I got two posters accepted to the SweCog 2017 conference in Uppsala, October 26-27. Unfortunately I got sick right before the conference so I couldn’t attend myself. The posters were, however, shown during the poster session.

The image above shows one of the posters – the one I created together with my KTH colleague Emma Frid. The study presented in the poster is based on the study I wrote about here, where Eva-Lotta Sallnäs Pysander and Roberto Bresin also participated. In the original study we found indications that gaze behaviour could be affected by haptic and audio feedback in a single user setting. In this new collaborative study presented in the poster, where we used a similar interface, we wanted to investigate if gaze behaviour can be affected by haptic feedback during collaborative object managing.

We have not performed the real experiments yet, but results from a pilot study with a few pairs of users (some worked in a non-haptic version of the interface and some in a haptic version) indicated that haptic feedback could have an effect on gaze behaviour (see e.g. the figures presented on the poster above). The results are not significant, but still interesting enough to make it worth running similar experiments with many more participants. A future step to take could also be to investigated how audio feedback (and/or combinations of haptic and audio feedback) affect gaze behaviour during collaborative task solving.

The poster above summarizes the work done. More information can be found in the published conference abstract which you can find here.

conference · Haptics · Multimodality

Got two posters accepted to SweCog 2017!

Jonas Moll1 Comment

SweCog_accept

In an earlier blog post I wrote about my preparations for the Swedish Cognitive Science Society (SweCog) 2017 conference. My plan was to submit at least two papers to that conference and that was exactly what I did. One of the papers, “Using Eye-Tracking to Study the Effect of Haptic Feedback on Visual Focus During Collaborative Object Managing in a Multimodal Virtual Interface” I wrote together with Emma Frid and the other “Haptic communicative functions and their effects on communication in collaborative multimodal virtual environments” I wrote together with Eva-Lotta Sallnäs Pysander. I was first author on both since I led the work and did most of the writing. Earlier this week I got two emails from the conference organizers confirming that both papers had been accepted as posters!

When Eva-Lotta and I submitted the papers (you could only submit one per person) we indicated that we were aiming for oral presentations, but they were both “downgraded” to posters after the reviews. When it comes to first paper, written with Emma, I can understand it, since we were reporting on a pilot study and there were quite a few papers submitted by other researchers which reported on full-scale experiments and evaluations. The other one, on haptic communicative functions, were more theoretical in nature and in that case I think the main problem was the 500 words limit – we couldn’t really elaborate enough on our main findings, when most of the space had to be used to define and explain haptic communicative functions. Anyhow, I’m very happy that the papers were accepted and that we will be able to discuss our work with others during the conference.

The second confirmation email, about that paper on haptic communicative functions, actually included an interesting twist – one of the reviewers of that paper recommended that the paper should be presented by means of a live-demo during the poster session! That really came as a surprise (a positive one) and the organizers were really willing to work with us to make the live-demo happen. Unfortunately, one problem is that the studies referenced in the paper (about an evaluation and an experiment, respectively, during which pairs of users were collaborating by means of haptic and audio communicative functions – see this and this preprint article) used virtual environments based on outdated API:s that no longer work. I’m not sure that I can implement the environments using the newer haptics API Chai3D in time for the conference. But, no matter what, will still have the poster and the possibility to discuss and explain our findings.

So, the only thing remaining now (apart from trying to get a demo working) is to create two informative posters. After the conference I will get back to this topic and elaborate some more on the work presented on the two posters, so expect more posts about SweCog 2017 and my contributions two it!