Kapow!
Augmenting contacts in mixed reality
This was a side project during my doctoral work. Due to the COVID context, I couldn’t attend the lab to work with my haptic setup. During the lockdown, my thesis advisor and I dived into pseudo-haptics. I had the opportunity to develop an application in mixed reality that explored contact augmentation using visual feedback.
Activities
Designed a set of techniques for contact augmentation in mixed reality (MR)
Developed an MR application for exploring contact augmentation through pseudo-haptics
Role
PhD student
MR developer
UX researcher
UI/UX designer
Tools
Headset
Microsoft HoloLens 1 & 2
Software
Unity
MRTK
Adobe Illustrator
Kapow is quite special for me…
This work not only illustrates my experience with mixed reality, but it also showcases many of the things I was lucky enough to do during my doctoral studies.
My thesis research involved a complex haptic system, but with Kapow, the “tangible” aspect of haptics disappeared. How crazy is that, right?
On top of that, I had the opportunity to use my design skills to create graphics that gave this research project a strong personality. My thesis advisor and I had so much fun working on a project that incorporated graphic elements inspired by comics—a shared passion of ours.
Additionally, I had the chance to collaborate with amazing researchers in the fields of mixed reality and human–machine interaction.
Most importantly, I was lucky to work with Julien Cauquis to explore ways of creating visual feedback to enhance object-contact interactions using a design space.
For me, it was both crazy and wonderful to have a paper featuring graphics like these — almost as if the research paper were a comic book.
Challenge
An important challenge for MR developers is maintaining the user’s sense of immersion when they try to touch objects without a truly opaque rendering.
During this research, we asked ourselves: how can we provide contextualized visual feedback in mixed reality to enhance users’ immersion when touching virtual objects?
In other words:
How can we make touching objects in mixed reality more… realistic?
Augmenting contacts in mixed reality
In this research work, I created a set of stylized visual effects (VFX) in mixed reality for augmenting contact with real and virtual objects.
These effects could be separated into the following categories:
Physics-inspired VFX
Comics-inspired VFX
Haptics without haptics
The visual feedback provided in the mixed reality application used a very deep and interesting principle: pseudo-haptics.
In a few words, pseudo-haptics rely on the dominance of visual stimuli over haptics. Sometimes, visual stimuli can give us the sensation that we are actually pressing or manipulating an object without actually touching it.
We explored the influence of this dominance of visual stimuli over haptic stimuli for provoking the sensation of touching something that wasn’t actually there.
In the case of real objects, we explored the possibilities of adding extra properties to the objects we were touching with the use of visual effects.
During the last months of my PhD studies, Julien and I studied further the creation of novel visual effects for augmenting contact with virtual objects in mixed reality.
Julien came up with a huge design space for exploring the different functional branches of visual feedback for contact augmentation.
I’m so glad that he continued with this great research project that eventually brought a nice VRST paper.
