Category: Human-Computer Interaction

communication · Course · Human-Computer Interaction · Pedagogy · Popular science

Time to start writing another popular science book chapter!

Jonas MollLeave a comment

In the middle of August my colleague Ann-Sofie Hellberg and I submitted a proposal for a popular science book chapter, hoping that it would be considered for inclusion in a book focusing on digitalization of education. A few days ago we got a notification that our proposal had been accepted, so now it’s time to start writing the real chapter!

The book will include several short chapters written by teachers all around the world and the focus will be on experiences with and evaluations of digital learning. As I wrote in this blog post quite a while back, I have contributed to a chapter in the popular science book Digitalisering av högre utbildning [Digitalization of higher education]. That time around I wrote about experiences from using Twitter as a complementary communication channel in a course in communication that I held at KTH. This new book seems to be very similar, since teachers are writing for teachers.

We have not decided on the details yet, but the book chapter will focus on experiences with teaching a course in interaction design online and a special emphasis will be placed on support for communication between course participants. I will of course get back to this as work progresses. Hopefully, the complete version of the chapter will be included in the upcoming book.

Further on, we will also produce at least one scientific publication about the experiences of teaching project courses online, but interviews with some students will be needed as additional input before that work can start. The work with the scientific publication will be a part of the overall research effort around effects of Covid-19 on education at the department of Informatics at Örebro University.

Haptics · Human-Computer Interaction · Multimodality · sonification · Thesis defense

Recently attended Emma Frid’s thesis defense at KTH!

Jonas MollLeave a comment

Disputationsbild_Emma

On Friday, January 10, I attended Emma Frid’s thesis defense at KTH. Emma and I collaborated in a research project a few years ago, and one of the major outcomes was this open access article presenting the results of an experiment with a multimodal interface including both haptic feedback and two different sonification models. Emma’s thesis work relates heavily to the research field of Sound and Music Computing (also the name of a sub-group at the department of Media technology and Interaction Design at KTH where I worked for more than a decade), and focuses specifically on (accessible) digital music instruments and interfaces. The main research question is “How can music interfaces be designed for inclusion?”. The thesis “Diverse Sounds – Enabling Inclusive Sonic Interaction” can be found here. The main supervisor was professor Roberto Bresin and co-supervisor was professor Eva-Lotta Sallnäs Pysander. Both of them work at the department of Media technology and Interaction Design at KTH. The opponent was Reader Andrew McPherson from the school of Electronic Engineering and Computer Science at Queen Mary University of London. The examination committee consisted of senior researcher Elaine Chew from the French National Centre for Scientific Research and the Music Representations Team at the Institute for Research and Coordination in Acoustics/Music, professor Rolf Inge Godøy from the Centre for Interdisciplinary Studies in Rhythm, Time and Motion at University of Oslo, associate professor Dan Overholt from the department of Architecture, Design and Media Technology at Aalborg University and professor Henrik Frisk from the department of Composition, Conducting and Music Theory at Royal College of Music in Stockholm.

After the introduction by associate professor Madeline Balaam, who chaired the event, the opponent held a presentation about Emma’s thesis for about 45 minutes. It was a very good presentation and it was interesting to listen to his interpretation of the work performed. He concluded his presentation by discussing the nine properties that according to Emma should be considered when designing accessible digital music instruments; expressiveness, playability, longevity, customizability, pleasure, sonic quality, robustness, multimodality and causality (see the thesis for a thorough coverage of these properties and the work that gave rise to them). After a short break the opponent, and later on the members of the committee, asked question that formed a good foundation for interesting discussions about the thesis work.

I think Emma did a really good job answering the questions and discussing her work. She elaborated a lot on the themes that were brought up to discussion and it was very clear that she knows a lot about this research field. She was also calm during the entire process and even helped out when the opponent and committee members e.g. needed headsets and/or microphones. One thing that was special about this defense is that the opponent, as well as all members of the grading committee, began their round of questions by congratulating Emma on the excellent job that she has performed! I have not seen that during other defenses I have attended. The defense was rounded off by a very long applaud – it was almost as if the audience expected some kind of extra performance on stage.  🙂

communication · design · Group work · Haptics · Human-Computer Interaction · Multimodality · sonification

Overview of my research within multimodal interaction

Jonas Moll1 Comment

thesis

In my last blog post I presented an overview about my research within the eHealth domain. In this blog post I will do the same thing, but for my other main research field – multimodal interaction in virtual environments.

 

What have I done related to multimodal interaction?

Even though I have spent the last couple of years focusing mainly on eHealth, I have done a lot of research – especially as a Ph.D. student at the Royal Institute of Technology – related to multimodal interaction. Most of this research has been focused on multimodal learning environments for collaborative task solving between sighted and visually impaired persons. Haptic feedback has played a major part in the collaborative virtual environments that I have designed and evaluated both in lab settings and in the field in e.g. primary schools. Quite a while ago, I wrote a blog series on haptic feedback focusing on the work I performed within the scope of my doctoral studies. Here are the links to those posts:

During my time as a postdoc at Uppsala University, I also performed some activities related to multimodal interaction. Most of this time I devoted to research grant applications and I also wrote a few conference papers. You can read a short summary of these activities here.

In total, my research on multimodal interaction has, up until today, resulted in the following five journal publications (some links lead to open access publications or pre-prints):

and the following 11 conference papers (some links leads to open access publications or pre-prints):

 

My ongoing research within multimodal interaction

Currently, there is not much going on related to this research field (at least not in my own research). The only ongoing activity I’m engaged in is an extensive literature review related to communication in collaborative virtual environments which will lead to a theoretical research article where I will discuss different technical solutions for haptic communication in the light of the research I have performed within the area up until today. I’m collaborating with my former Ph.D. supervisor Eva-Lotta Sallnäs Pysander on this activity. I hope that this research activity will help me in my continued research on collaboration between visually impaired and sighted pupils based on different types of tasks and learning material.

Upcoming research on multimodal interaction

As I wrote in a recent blog post multimodal interaction, with a focus on haptic feedback, seems to be a new research area at the Centre for empirical research on information systems (CERIS) where I just stared my assistant professorship. Thus, this is the research area in which I can contribute with something new to the department. An area that is already represented at the department, however, is “Information Technology and Learning”, which seems to be a perfect fit in this case!

Last year, I also submitted a research grant application focusing on continued work with collaborative multimodal learning environments. Unfortunately, that one was rejected but no one is giving up. I will work somewhat on revising the application during the autumn and submit as soon as a suitable call pops up. Maybe I will also have additional co-applicants from the CERIS department by then.

communication · eHealth · Group work · Haptics · Human-Computer Interaction · Medical Records Online · Multimodality · Social media in higher education

Today I start working at Örebro University as an assistant professor!

Jonas Moll1 Comment

Fjäll7

Today I can finally take the next step on the academic ladder, since I’m starting up my new job as assistant professor in Informatics at the centre for empirical research on information systems at Örebro University! In November 2018 I applied for the position and in the middle of the spring 2019 I was called to an interview. A few months later I was offered the position. I’m very excited about this great opportunity and I of course intend to make the most out of it. After a very long blog break (mostly due to health issues and the fact that my research efforts during the spring has been rather minor), this also seems like a good opportunity to start posting again.

The assistant professorship is the first step on the so called tenure track. It is an academic position limited to four years, but the intention is often (as in my case) to promote the assistant professor to an associate professor towards the end – a position which is not limited in time. My job includes 70% research and 30% teaching, which is quite common for assistant professorships. I’m not sure yet where e.g. service and communication (like administration, blogging and interviews) fits in.

The job as assistant professor in Informatics is a very good fit for me, since I will be able to continue to work with all my main research interests (the main theme is computer supported communication):

  • eHealth
    – I will continue looking at how patient accessible electronic health records (PAEHR) affect the communication between patients and care professionals. One thing I’m particularly curious about, and that is actually the focus of a project grant application currently in review, is how one can incorporate the PAEHR as a communication mediator curing doctor-patient meetings. Another application in review is about the effects and implementation of psychiatry records online.
  • Multimodal interaction
    – I will also continue looking at how multimodal feedback (especially haptics and sound) can be used to promote collaboration between sighted and visually impaired pupils/students in group work. Most of today’s assistive technologies that are used in school settings are not adapted for collaboration and this is highly problematic when it comes to inclusion of visually impaired pupils/students in group work settings.
  • Social media in higher education
    – My intention is also to continue investigating how social media like Twitter and Facebook can be used as supplementary communication channels in higher education courses.

When it comes to the areas of eHealth and social media in higher education, research is already being conducted by my new colleagues at Örebro University. Multimodal interaction would however be a new research theme for the department. I will elaborate on the different themes listed above in later blog posts as work is progressing. Other research themes from the department (like computer security and ICT for development) could also be added.

I have not heard anything yet regarding the teaching, but given the department’s focus I guess I could be involved in master’s thesis supervision, human-computer interaction project courses and programming courses. I will write more about the teaching part when I know more.

The blog image that I used for this post is one of my own – I took it a few weeks ago during a week I spent in Abisko in northern Sweden.

Academic writing · conference · DOME · eHealth · Grant application · Human-Computer Interaction · Medical Records Online · National patient survey · Pedagogy · Social media in higher education

Looking back at my two years as a postdoc at Uppsala University, part 1: research

Jonas Moll2 Comments

Gåsflock1

Since my two years as a postdoc at Uppsala University ended September 28, I will try to summarize my results and experiences in a few blog posts before this year ends. I will start today by writing about my research activities. I took the blog picture during my last birdwatching trip to Öland.

Even though most of my research activities during the postdoc period were focused on eHealth, I also did some work related to multimodal communication and pedagogy. Thus, I was active within all of my main research fields. In total, I got one journal article published, two journal articles accepted, one book chapter published and 11 conference papers published. I also attended nine conferences and submitted two research grant applications as main applicant.

Research within eHealth

Within this field I led two major studies related to patient accessible electronic health records (PAEHRs). The interview/observation/survey study at Uppsala University Hospital, which I introduce here, focuses on the effect of PAEHRs on physicians’ and nurses’ work environment. The survey part is completed and a journal manuscript, written by me and Åsa Cajander, was submitted to Health Informatics Journal slightly after my postdoc period had ended. The results are really interesting and I will of course get back to them when the article has been published. Due to some health issues and logistical difficulties, we have only conducted about half of the planned interviews with physicians and nurses, but we will hopefully get the majority of the remaining ones during the first months of the spring term. The analysis of the already performed physician interviews are still ongoing, and it’s very clear that the material that we already have will give many new insights into long-term effect of PAEHR on the work environment of healthcare professionals. This has really been an interesting study and it was also the first study for which I got the opportunity to write an ethical application. It has also been quite a challenge to coordinate the work with nine colleagues from five different universities.

The other major study I was leading was the national patient survey, which I introduce here. I have written about it many times on this blog since it has resulted in several scientific publications as well as presentations. Even in this case, one of the toughest challenges has been to coordinate the work in the distributed project team. This study has already resulted in one journal article and three conference papers and we are currently working with several journal article manuscripts which I will get back to later on. I really enjoy this study and I will keep working with it until everything is published. The study is very important since it’s the first major follow-up study on patients’ attitudes towards and experiences with PAEHRs.

During my time as a postdoc I also took part in the work with two research grant applications, as a co-applicant, within this research area. One of them was an EU grant application led by Meena Daivadanam at Karolinska University Hospital, with the title “The ENGAGE Trial: Improving and health societal outcomes for comorbid mental disorders associated with type 2 diabetes through an integrated support and engagement platform in Uganda and Sweden”. It was an interesting process with many Skype meetings (extremely early in the morning since one participant was in Australia) and a lot of interesting discussions. Unfortunately, we did not get the grant. As I understand it we were one point from getting it. The other research grant application concerned psychiatry records online. My DOME consortium colleague Gunilla Myreteg was the main applicant of this AFA insurances application focusing on implementation and short term effects of psychiatry records online in Region Uppsala. We did not get that grant either, but we are definitely not giving up!

I also attended several conferences related to eHealth during my postdoc period. I really enjoyed participating in and presenting at Vitalis 2017 and Vitalis/MIE (Medical Informatics Europe) 2018. There were so many interesting presentations and taking part in the 1.5 hours DOME arrangements was great! You can see my summary of the Vitalis 2017 version of the DOME session here. At Vitalis/MIE 2018 I was actually active with own presentations and a workshop during each of the three conference days, which was a little tiresome. You can read about that here. My very first conference experience during my postdoc was actually one of the most interesting ones, since I participated as a patient for the very first time! You can read about my contribution here. A few months later I actually got the opportunity to act as a patient once again – this time in a role play at the conference “EHealth in Norway Future Health”! That was a really interesting experience for me both from a patient’s and a researcher’s perspective. You can read about my experiences from this episode here. I really hope that I will get the chance to contribute to research from the patient’s perspective again!

 

Research within multimodal communication

My very few research activities within this subject area were mostly related to research grant applications. During spring 2017 I submitted a VR grant application, with five co-applicants from four universities in Sweden. The application focused on collaboration between deafblind and sighted pupils in a school setting, and more specifically on how multimodal learning environments can support this collaboration. Writing this grant application was a great learning experience! In the end, we did not get this grant but we still got good ratings (“Very good” on all aspects that related to the scientific content). You can read more about my experiences in this blog post. The other grant application was actually a draft which I submitted to Forte during spring 2018. It focused on multimodal learning environments for collaboration between sighted and visually impaired pupils. Unfortunately, the draft was not accepted. After the postdoc ended I wrote another application on this topic as main applicant, but I will cover that in another blog post.

I also got one journal article published in this research field, “Haptic feedback combined with movement sonification using a friction sound improves task performance in a virtual throwing task.” The article presents results from an experiment conducted at KTH right before my postdoc period began. The experiment was extensive and included eye-tracking and different combinations of haptic and audio feedback. I will write more about this study in a later blog post. A few conference papers on results from the eye-tracking analysis have also been published.

 

Research within pedagogy

I also conducted research related to pedagogy and more specifically on the effects of using social media as complementary communication channels in higher education courses. The most important thing I worked on was a short book chapter which was published in the book “Digitalisering av högre utbildning” [Digitalization of higher education] about a month before my postdoc ended. I really enjoyed working on this chapter, together with co-author Pernilla Josefsson, where I contribute with my experiences of using Twitter as one of the communication channels in a university course in engineering communication. This is my very first contribution to teacher education! You can read about the chapter, and find a link to the book, here.

We also contributed with a poster describing a later study on using a teacher-administrated Facebook-group as a complementary communication channel in a course in human-computer interaction. I will write more about this in my next blog post about my postdoc period, since the poster was based on work performed in a pedagogical course I took. Last, I collaborated with several authors when writing a conference paper about a critical incident from the 2017 version of a master course in human-computer interaction. You can read about the paper here.

communication · design · DOME · eHealth · EIT · EIT Health · Grant application · Human-Computer Interaction · Medical Records Online · National patient survey · Social media in higher education

Looking back at 2017!

Jonas MollLeave a comment

Färjeläget

A new year has just begun and before I start blogging about current activities, I will take this opportunity to look back at some of the important things that happened last year. If you have been following this blog regularly, you have probably already read about the last summer’s adventures at the ACM SIGCHI/EIT Health summer school in a number of different posts so I will not get into any detail about that school here. If you want to read about what went on during this eHealth/mHealth design summer school you should read this post and all the posts it links to. The same goes for the very successful DOME session about patient accessible electronic health records at Vitalis, which I describe here.  As in all other of my meta-posts, I have chosen one of the nature pictures I took myself during the year.

Selected research activities

There was quite a lot of research going on last year. The research activities related to all of my focus areas eHealth, multimodal communication and social media in higher education courses. Naturally, most of my activities were connected to eHealth. Primarily, I continued working with the national patient survey on patient accessible electronic health records in Sweden (introduced here) and the interview/observation/survey study with physicians and nurses (introduced here). Last year’s work with the national patient survey resulted in four submitted manuscripts (to one journal and two conferences) and there is a lot more we can do with that study! It became especially interesting when we started comparing responses from different disease groups during the end of 2017. I will tell you more about these results later on when we have some publications. In the other study we conducted several interviews with physicians, about the effects of patients accessible electronic health records on their work environment, during the autumn. Most of the surveys have also been handed in. I will not discuss the results before they get published but I can tell you that both the qualitative and quantitative data gathered this far show that this study was very much needed! During the autumn I also started to, with my DOME consortium colleague Gunilla Myreteg, follow the implementation of psychiatry records online in Region Uppsala. I will write more about that in later posts.

I also started on another track related to eHealth – how we can make use of data from different sources for better diagnoses and treatment of cancer. This topic is also connected to big data and happens to be one of the themes covered in the 2018 version of Uppsala Health Summit. I am one of the organizers behind a workshop there and I also submitted a workshop proposal to a conference on the same theme. It remains to be seen if this will become one of my main research focuses in the future. It’s definitely and interesting and very important topic.

My work on multimodal mediated communication during 2017 focused primarily on revising a journal manuscript (presenting a study I was a part of during my last year at KTH) and submitting two posters to the Swedish Cognitive Science Society conference which was held in Uppsala in October. The accepted submissions are more thoroughly presented here and here.

Teaching

During autumn I was, together with Mohammad Obaid, responsible for a master level introductory course in Human-Computer Interaction. I really enjoyed that and I think the changes made to the course before it started really made a positive difference. One of the new parts we introduced to the course was a very appreciated creative prototyping session which I describe here. The students presented their final results in the middle of December and those results were very good. I describe the final presentation sessions here and the four finalists (which will compete for the winning team award) in these four blog posts:

  1. HCI course finalist 1: enhancing and simplifying the biking experience through augmented reality!
  2. HCI course finalist 2: utilizing haptic feedback for alerts and navigation cues!
  3. HCI course finalist 3: a solution for finding bikes to rent at your destination!
  4. HCI course finalist 4: a device presenting real-time and easily read navigational cues!

My research on the use of social media in higher education courses was also combined with my teaching in this course. The focus of that research is a teacher administrated Facebook-group which has been used as a complementary communication channel during the course. I explain the basic idea here.

Some other activities

Other activities worth bringing up here are my participation in the EIT Health Alumni Connect and the INNOVEIT events in Budapest in October. It was a great experience being there and my blog posts about Connect and INNOVEIT as well as those about the summer school also resulted in me being asked if I wanted to work with the EIT Health Alumni communication manager! Of course I accepted, but it remains to be seen exactly how that work should be carried out and what the tasks will be.

In November, I also took part in the first meeting as a representative in the eHealth council at the National Board of Health and Welfare in Sweden. I wrote about some of my experiences from that meeting here. I think many interesting collaborations can be found here!

In April I also submitted my first ever VR (the Swedish Research Council) application, which focused on support for collaboration between sighted and deafblind pupils in school. I wrote about that here. Unfortunately, we did not get a grant this time, but we got a “Very good” ranking on all criteria related to the content and feasibility so we will definitely move on with our ideas!

Course · Human-Computer Interaction

HCI course finalist 4: a device presenting real-time and easily read navigational cues!

Jonas Moll1 Comment

LeapUp

Today, I’m presenting the fourth finalist in our basic HCI course at Uppsala University. This finalist group is called “LeapUp” and consists of the master students: Shubhesh Aggarwal, Evan James, Nuruddin Kamil, William Sonesson and Anusha Venkatraman.

Like some other finalist groups, LeapUp focused on simplifying navigation, but they used a separate device instead of AR or haptic feedback. The proposed product, which should be attached to the bike handle, as well as an example screen, are shown in the picture above. Most devices for navigation show e.g. different types of status information, a map and indications for when to change direction (e.g. make a turn). The proposed device only shows the directional cue, in the form of a big arrow, the time, battery status and the estimated time to arrival. The focus is on simplicity and highlighting of the most important information needed for navigation. There is also a notification LED light indicating changes, so there is no need to constantly check for e.g. arrow changes. Apart from the hardware device there is also an accompanying app, where the user e.g. can customize colors and indicate a destination on a map. Thus, all necessary inputs are made on a mobile device before the bike ride starts – during the actual biking activity the device only displays the most necessary information and alerts. There are also some other features in the app, like an option for turning off notifications from e.g. Facebook during the bike ride.

The proposed device and accompanying app focuses on simplicity and seems to be a very good alternative for easy and safe navigation. If you want to read more about the device and the app, as well as the work process, you should take a look at the group’s project blog!

Course · design · Human-Computer Interaction

HCI course finalist 3: a solution for finding bikes to rent at your destination!

Jonas Moll1 Comment

Human-Cycling Interaction

So, now time has come to present the third finalist in our basic HCI course. Today’s finalist is the group “Human-Cycling Interaction” and consists of the master students: Jesper Ericsson, Dara Kushnir, Anthony Mathieu, Nam Nguyen, Jules Ruig and Lisanne Wiengarten.

This group did not focus on the actual cycling activity, but rather on the logistics around finding a bike to use while e.g. travelling in other countries. The group designed an app which can be used to locate bikes which are possible to rent (from private persons), at the location you are aiming for. Their solution is similar to Airbnb, but for bikes instead of places to stay. After signing in and choosing location and time interval, a scrollable catalogue of possible bikes to rent is presented. Each bike option in the list contains details such as size, current distance from e.g. city center, status of different equipment and if helmets are included. See the picture above for an example. After having proceeded to order the bike (e.g. rent it during the chosen period), the user can choose to add a number of extra features like insurance, baskets or a helmet. After payment, the order is confirmed and moved to a list of orders made. It is also possible to contact the owner of an ordered bike through a chat feature in the app and there are some other options included as well.

The proposed app adds an important possibility for travelers, since it enables them to find alternatives to often quite expensive public transport solutions like buses and taxis. There are some problems associated with the proposed method and if you want to read more about how the group wants to tackle these and about the work process and prototype development you should definitely take a look at their project blog!

Course · design · Human-Computer Interaction

HCI course finalist 1: enhancing and simplifying the biking experience through augmented reality!

Jonas Moll1 Comment

Ace_that_Interface

In my last blog post I wrote about the final presentation seminar sessions in the master level HCI course that Mohammad Obaid and I have been running this autumn. Based on the scores that were given to each group (from fellow students and teachers) during those sessions, I was able to select the finalists. See the blog post linked above for an explanation of e.g. the voting. In this, and the upcoming three blog posts, I will briefly introduce the finalists. The finalists will be presented in no specific order and pictures from their project blogs will be used only if it’s ok by the respective groups.

Today’s finalist is the group “Ace that Interface”, consisting of the students Joosep Alviste, Imad Collin, Hassan Odimi, Iosif Kakalelis and Mauro José Pappaterra. They came up with the idea shown in the image above – an augmented reality solution for cyclists. The view has different components, providing information to the user, which are placed in a way so that they should not interfere too much with the cyclist’s field of view. General status information, like temperature, time and heartrate, is placed in the upper left part. In the lower left part you find a map showing the surrounding and your current position and beside the map is a presentation of e.g. your speed. If you have chosen a specific route (in an accompanying app) there are also indicators showing where to turn and the distance you need to move on a specific part of the route (compare with usual GPS solutions in cars or Google maps on phones). The navigation arrow blinks when it changes direction, in order to make sure that the cyclist notes when it’s time to e.g. make a turn. In the accompanying app you can choose which information items you want to show (toggle on/off) in the augmented reality interface, select routes, etc.

This is a somewhat futuristic idea, but it is great and has a lot of potential. It presents (customizable) information which is relevant for the cyclist, without forcing him/her to take the hands off the handles or shifting focus to another device. There is a lot more I can say about this idea and the process the students followed, but since the main documentation medium used in this course is a project blog, I think it’s better that you go there for more detailed information. In this blog you can read more about the project members and the group’s whole process from requirements elicitation to the final design.