Grant application · Group work · Multimodality

New project grant application submitted to Vinnova!

Vinnova_submitted

During the last couple of weeks I have worked a lot on a research grant application to Vinnova (call: Digital tools) and today it was finally submitted! The application is based on a collaboration between the following partners, who all apply for funding in the application:

The proposed project is based on earlier proof-of-concept work that I performed at KTH regarding multimodal learning environments supporting collaboration between sighted and visually impaired pupils. I will of course be able to say a lot more about the involved partners and the content of the project after Vinnova has reviewed the application (in the middle of November). I really hope that the project will be funded since I really miss working within this research area, and I also think that the research is important for society.

One confirmation of that both our old and planned research in this area is considered important is that representatives from both the Swedish Association of the Visually Impaired and the National Agency of Special Needs Education and Schools decided to take active part in the project (in case it gets funded). I’m sure they would not do that if they didn’t identify a high importance for society, and especially the main target group, as well as an importance on a national level. Even if the project doesn’t get funding we should definitely make sure to continue the collaboration and discussions with the involved partners – there are always new opportunities for research up ahead and as long as clear mutual benefits can be identified the collaboration should move on.

Stay tuned for more!  🙂

 

communication · design · Group work · Haptics · Human-Computer Interaction · Multimodality · sonification

Overview of my research within multimodal interaction

thesis

In my last blog post I presented an overview about my research within the eHealth domain. In this blog post I will do the same thing, but for my other main research field – multimodal interaction in virtual environments.

 

What have I done related to multimodal interaction?

Even though I have spent the last couple of years focusing mainly on eHealth, I have done a lot of research – especially as a Ph.D. student at the Royal Institute of Technology – related to multimodal interaction. Most of this research has been focused on multimodal learning environments for collaborative task solving between sighted and visually impaired persons. Haptic feedback has played a major part in the collaborative virtual environments that I have designed and evaluated both in lab settings and in the field in e.g. primary schools. Quite a while ago, I wrote a blog series on haptic feedback focusing on the work I performed within the scope of my doctoral studies. Here are the links to those posts:

During my time as a postdoc at Uppsala University, I also performed some activities related to multimodal interaction. Most of this time I devoted to research grant applications and I also wrote a few conference papers. You can read a short summary of these activities here.

In total, my research on multimodal interaction has, up until today, resulted in the following five journal publications (some links lead to open access publications or pre-prints):

and the following 11 conference papers (some links leads to open access publications or pre-prints):

 

My ongoing research within multimodal interaction

Currently, there is not much going on related to this research field (at least not in my own research). The only ongoing activity I’m engaged in is an extensive literature review related to communication in collaborative virtual environments which will lead to a theoretical research article where I will discuss different technical solutions for haptic communication in the light of the research I have performed within the area up until today. I’m collaborating with my former Ph.D. supervisor Eva-Lotta Sallnäs Pysander on this activity. I hope that this research activity will help me in my continued research on collaboration between visually impaired and sighted pupils based on different types of tasks and learning material.

Upcoming research on multimodal interaction

As I wrote in a recent blog post multimodal interaction, with a focus on haptic feedback, seems to be a new research area at the Centre for empirical research on information systems (CERIS) where I just stared my assistant professorship. Thus, this is the research area in which I can contribute with something new to the department. An area that is already represented at the department, however, is “Information Technology and Learning”, which seems to be a perfect fit in this case!

Last year, I also submitted a research grant application focusing on continued work with collaborative multimodal learning environments. Unfortunately, that one was rejected but no one is giving up. I will work somewhat on revising the application during the autumn and submit as soon as a suitable call pops up. Maybe I will also have additional co-applicants from the CERIS department by then.

communication · eHealth · Group work · Haptics · Human-Computer Interaction · Medical Records Online · Multimodality · Social media in higher education

Today I start working at Örebro University as an assistant professor!

Fjäll7

Today I can finally take the next step on the academic ladder, since I’m starting up my new job as assistant professor in Informatics at the centre for empirical research on information systems at Örebro University! In November 2018 I applied for the position and in the middle of the spring 2019 I was called to an interview. A few months later I was offered the position. I’m very excited about this great opportunity and I of course intend to make the most out of it. After a very long blog break (mostly due to health issues and the fact that my research efforts during the spring has been rather minor), this also seems like a good opportunity to start posting again.

The assistant professorship is the first step on the so called tenure track. It is an academic position limited to four years, but the intention is often (as in my case) to promote the assistant professor to an associate professor towards the end – a position which is not limited in time. My job includes 70% research and 30% teaching, which is quite common for assistant professorships. I’m not sure yet where e.g. service and communication (like administration, blogging and interviews) fits in.

The job as assistant professor in Informatics is a very good fit for me, since I will be able to continue to work with all my main research interests (the main theme is computer supported communication):

  • eHealth
    – I will continue looking at how patient accessible electronic health records (PAEHR) affect the communication between patients and care professionals. One thing I’m particularly curious about, and that is actually the focus of a project grant application currently in review, is how one can incorporate the PAEHR as a communication mediator curing doctor-patient meetings. Another application in review is about the effects and implementation of psychiatry records online.
  • Multimodal interaction
    – I will also continue looking at how multimodal feedback (especially haptics and sound) can be used to promote collaboration between sighted and visually impaired pupils/students in group work. Most of today’s assistive technologies that are used in school settings are not adapted for collaboration and this is highly problematic when it comes to inclusion of visually impaired pupils/students in group work settings.
  • Social media in higher education
    – My intention is also to continue investigating how social media like Twitter and Facebook can be used as supplementary communication channels in higher education courses.

When it comes to the areas of eHealth and social media in higher education, research is already being conducted by my new colleagues at Örebro University. Multimodal interaction would however be a new research theme for the department. I will elaborate on the different themes listed above in later blog posts as work is progressing. Other research themes from the department (like computer security and ICT for development) could also be added.

I have not heard anything yet regarding the teaching, but given the department’s focus I guess I could be involved in master’s thesis supervision, human-computer interaction project courses and programming courses. I will write more about the teaching part when I know more.

The blog image that I used for this post is one of my own – I took it a few weeks ago during a week I spent in Abisko in northern Sweden.

Grant application · Group work · Haptics

Recently submitted a project proposal to this year’s Forte junior research grant!

Tordmule

A few weeks ago I wrote a blog post about the spring of opportunities, which basically contained lists of possible research grant and job opportunities that were open or should open during the spring term. Just before lunch today, I submitted the first of my planned project proposals! This one went to Forte and their grants for junior researchers.

The proposed project relates to work that I performed while I was still working at KTH as a Ph.D. student and the content was also inspired by the application that I sent to VR last year (see this blog post). The project is based on the fact that today’s assistive technologies that visually impaired pupils use in the classroom are not really designed for collaborative situations – the technology could sometimes be a hindrance when doing group work with sighted peers. Our hope is that the planned activities will really shed light on the problems this is causing and show how one can make use of modern technology, based on haptic and audio feedback, to find ways ahead. I have already done a few studies in this area which you e.g. can read about in this article (note that this is a pre-print version). I will write more about this when I get an answer from Forte about the draft later on in April.

My co-applicant is Eva-Lotta Sallnäs Pysander from KTH, who was also my main supervisor there. If we get the grant, we will hopefully be able to add a Ph.D. student to the research team as well.

I must say that it’s nice to have this proposal submitted. I really believe in the ideas in it and I also think that the project could make a real difference. Since I’m generally very interested in multimodal interaction and learning, this is also one of the areas I really want to focus on in my research in the future. The other area is eHealth and as I have written before there are quite a few funding opportunities in that area as well during this spring. But right now I’m just enjoying that I have one application out there, just as the razorbill in the image above probably enjoys sitting on a cliff looking out over the ocean…

 

communication · conference · Group work · Haptics · Multimodality

Paper and poster about haptic communicative functions and their effects on communication in collaborative virtual environments

HaptiCom

Yesterday I blogged about a poster and a conference paper that Emma Frid and I developed for the SweCog conference in Uppsala. In this post I will focus on the second poster and paper that Eva-Lotta Sallnäs Pysander and I developed for the same conference.

The poster shown in the picture above, and even more so the paper, summarizes some of the main points made from my doctoral studies. My main focus during those years was collaboration in multimodal virtual environments with special emphasis on how haptic feedback can be used for communicative purposes. Mediated haptic communication has been studied for quite some time, but my specific contribution here has been to develop and test new functions for two-way haptic communication (see short descriptions of the functions on the poster) and also adapt some already developed ones in order to make them work better in a situation when a sighted person is collaborating with a severely visually impaired one in a collaborative virtual environment. There is a real potential in these kinds of functions when it comes to collaboration between sighted and visually impaired – the haptic feedback does not only enable establishment of a common ground about the interface but also effective two-way communication (see examples of results on the poster above). This is very important for the inclusion of visually impaired persons in group work. The example study is reported in much more depth in this article.

Even though the poster and paper include summaries of work already performed and reported, we are in this case even more explicit about the connection to other kinds of haptic communicative functions. This conclusion also takes the work to the next level:

We argue that for effective collaboration and communication to take place in virtual environments by means of haptic feedback the haptic functions need to be designed as to allow for reciprocal exchange of information. That is, both users need continuous feedback from each other during e.g. a guiding process or joint object handling.

The conference paper, on which the above poster is based, can be found here.

conference · eHealth · EIT Health · Group work

Some remarks on EIT Alumni Connect 2017, in Budapest

Connect

As I wrote in my last blog post the EIT Alumni Connect event was hosted in Budapest October 15 –  October 16. I really had a great time during this event and if I should choose one word to describe the overall impression I got from the event it would be “Inspiring”! There were a lot of inspiring and thought provoking talks/keynotes spread over these two days and the hands-on activities provided learning as well as networking opportunities. A lot happened during these days and I cannot cover everything in the blog post, but I will at least make a few important points.

First of all, I really liked the setting in the room we all gathered in during the event. There were around 100 participants and 16 round tables and the activities performed especially during the first day made sure that there were representatives from several alumni networks (EIT Health, EIT Digital, EIT Raw Materials, Climate-KIC and InnoEnergy) by each table. This gave an excellent opportunity to develop an understanding of the different main areas where EIT is involved. One of the main aims of the event was to provide an opportunity to connect with other alumni and the setting ensured that networking could be performed both within and across the represented areas.

One especially interesting group activity was performed during the last part (before dinner) of the first day. The main aim was to work on real problems identified by the participants – problems related to the main areas of EIT. The group work activity started out by a discussion among the participants at the respective tables, about different project ideas that we would like to work with. My idea was, of course, based on patient accessible electronic health records and more specifically means of using these systems as mediators during patient visits. The ideas presented at my table were very different from one another since the participants represented different alumni networks. The next step was a very short pitch, given on stage so that everyone could hear. About 20 participants pitched ideas. Based on the pitches the rest of the participants should decide which idea they wanted to work with for the rest of the day. I never pitched my own idea, since I was really curious about another participant’s idea (about developing a system to enable digital consent for use of health data in research) and hence wanted to work on that instead. During the following 1.5 hours we discussed the respective projects within the newly formed project groups. The focus was to develop the idea and to develop a pitch which should be presented to a jury. This step was really important, since the three top groups would get a grant of 5000 Euros from EIT, which they should use to implement the ideas! Just before lunch the second day the three winners, named EIT Workshop, Impact and EIT Chaos, were announced.

The gamification component introduced in the group work activity was really working. Everyone gave great pitches and there were so many interesting ideas represented in the room. I’m quite sure the jury had a tuff choice to make. I also think it’s great the EIT actually give grants to promising projects. EIT support for projects and most of all start-ups was a theme that was covered in almost all keynotes and activities. The communities really support entrepreneurship and innovation. The different EIT KICs (Knowledge Innovation Communities) have supported several startups and projects through grants, accelerators and incubators. Some of these startups, which have become real success stories, were show cased during one of the presentations.

The keynotes covered EIT in general as well as different opportunities and challenges when it comes to most of all innovation and entrepreneurship. Some lectures also covered successful spin-offs of the kind of group project activities I described above, from earlier years  of EIT Alumni Connect. One of this spin-offs was the new group Women@EIT. The opening keynote showed that many good things have come from the EIT KICs during the recent years. E.g. several companies supported by EIT have great impact in many different areas and more and more students graduate from EIT master programs. A few challenges were, however, also mentioned. One identified need was that end users should be involved to a higher degree (HCI researchers really have an opportunity to contribute here!). EIT also wants to add new KICs so that all major issues we face today are covered (integration, security and water protection are a few areas that are not covered today). The board also wants to create even more collaborations across KICs. I really think the last point is important, since key challenges must be addressed from several different angles.

I really enjoyed being a part of the EIT Alumni Connect activity and I’m really considering applying for one of the open positions at the EIT Health Alumni board. (as I said earlier EIT Health Alumni is fairly new). I think I can make important contributions since I’m both a chronic patient and a researcher – maybe it would be beneficial to have a patient representative on the board? There were quite a few open positions to choose from (they were all presented during a lunch meeting with the current board members earlier today).

I will end this post by some words different participants used to describe EIT Alumni Connect during the first session today. I think they summarize what EIT is about in a good way: networking, explosive, fun, collaboration, innovation, inspiration.

communication · Group work · Haptics

More about my work with haptic communicative functions in collaborative virtual environments

img_1120

I’m currently in the process of writing a job application for an associate professorship and to make sure I don’t miss anything I recently browsed through my old conference folders to find the articles to append. When I came to EuroHaptics 2010 I was reminded about something I had completely forgotten – I was actually one of the organizers behind a workshop at that conference! I spent quite a lot of time preparing for the workshop, which focused on haptic communicative functions, but the weekend before the conference I got very sick and was forced to cancel my participation. I will take this opportunity to briefly introduce the workshop and discuss some of my work prior to it. My earlier blog posts on haptic feedback as an interaction modality were the following:

It was a shame that I could not attend the workshop/conference in Amsterdam, since it was based on my work on collaboration in multimodal environments up to that point. Thus, this was the perfect opportunity to discuss the work performed and get input regarding future work in the area. We described the focus of the workshop in the following way:

In this workshop, concrete examples will be presented and discussed in terms of how the touch modality can support communication and collaboration. Also, the technical challenges of distributed haptic feedback will be addressed. The target audience of the workshop is researchers and practitioners focusing on haptic feedback supporting people in settings where more than one user are involved. We invite other researchers and practitioners to share their research and experience from their different projects focussing specifically on the collaborative perspective. It might be that the collaborative aspects in your project have not yet been addressed. In that case, interesting collaborative aspects can be identified during the discussions in this workshop.

Quite a lot of work was performed by me and my “multimodal colleagues” 🙂 prior to the workshop. First of all, I had performed my master’s thesis work back in 2006 which focused on collaboration between visually impaired and sighted pupils in elementary school. Evaluations were performed in schools, where visually impaired and sighted pupils collaborated in dynamic collaborative environments where objects could be moved. During that work, and especially during a re-analyses performed during my first year as a Ph.D. student (2008) I realized that communicative functions based on haptic feedback had a real potential both when it came to supporting collaborative work and supporting inclusion of the visually impaired pupils in group work with sighted peers. It became especially clear that haptic functions for guiding (holding on to the same object or holding on to the peer’s proxy) can replace verbal guidance to a large extent.

Imagine a situation where you need to guide a visually impaired pupil to a particular place in a virtual environment. If you only have the visual feedback to rely on when establishing a common frame of reference, you need to talk a lot like  “go down, more down, …no, too much, go back…, now to the right…no, not down, up again… here it is!”. If you have haptic feedback available you can just grab the other person’s proxy and move the visually impaired peer to the right place and just say “here”. Needless to say, haptic feedback affects the dialogue between collaborators in this case. If you want to learn more about this explorative study you can read the journal article we finalized a few years after the study.

One problem that was evident from the evaluations with the visually impaired and sighted pupils was that the visually impaired pupil was not aware about what the sighted pupil did when haptic guiding was not utilized. This is why we performed a follow-up study where we added sound cues to the above mentioned dynamic interface to provide feedback on actions taken in the interface (e.g. grasping and putting down objects). We compared the new visual/haptic/audio version to the original visual/haptic one. We managed to show that the dialogue between the collaborators differed depending on which program version they worked in and it was also clear that the work was more effective (measured as time to complete task) in the visual/haptic/audio version. Once again, we could also clearly see how access to haptic feedback influenced the communication. You can read more about this study in this article.

These two studies resulted in a theoretical conference paper presented at HAID (Haptic and Audio Interaction Design) 2009, where we tried to develop a kind of conceptual model regarding haptic communicative functions’ effects on the dialogue between collaborators. This was my first attempt at a meta-analysis of the insights gained within this research area. The paper summarizes and discusses all the effects on the dialogue I had seen in my studies thus far. The paper made quite an impression – it is still the most referenced of all the papers I have produced up until today! At that point we were still quite unique when it came to haptic collaborative environments and I still think I’m one of the very few researchers who study the effect of haptic and audio feedback on the dialogue between collaborators.

The HAID conference paper laid the ground work for the workshop described in the beginning of this post and during the workshop the idea to study collaboration between two sighted persons was introduced and discussed. Next time I write about my earlier work I will introduce my latest study on collaborative multimodal interfaces, that showed that haptic and audio feedback indeed have effects on the dialogue between sighted persons as well!