In this article I'd like to share my experience about creating an innovation at R&D lab: what happens behind the scenes, what is the way an innovation goes through before to be applied in real life. I'll present a case about innovation which is based on not so well-known contactless technology. Since the blog is about the crossroads of unexpected worlds, I will also share how non-tech skills helped me in the innovation process.
The original article was published on DOU and is more focused on technical aspects, while here I somewhat simplified the tech stuff and balanced it with the writing aspect.
Intro
When I joined R&D team as an R&D Product Manager, I was responsible for building the business case aiming to commercialize novel technology called e-textures (digital textures) developed by our partners.
E-textures is based on contactless mid-air haptic technology, which aims to create a sense of touch using ultrasound waves without direct contact with an object or a surface. This sense of touch is being created in the air. Therefore, the name of the tech contains "mid-air".
E-textures allows generating digital textures of objects or surfaces by extracting them from the image or video samples using AI algorithms and creating corresponding tactile effects (patterns) that the user can feel when projected by a special hardware — a mid-air haptic device. In simple words, the main feature of the tech is to translate a visual into a tactile pattern.
E-textures user testing
As a result of cooperation with our partners, we created a demo version of the application using e-textures tech. This solution had an interface displaying various textures, such as mammoth fur, python skin, or silk, and allowed to virtually touch & feel them.
Below is an example of interaction: during the experience a user looks at the object vizualized on the screen and feels the texture with the haptic device.
The team and I have tested the demo with users and gathered feedback. It turned out, that users while seeing on the screen a specific texture, for example, a piece of silk textile, expected to feel the exact same sensation as if they touch a true silk. However, mid-air haptic provides more abstract feel of e-texture. Unfortunately, ultrasound is not capable of projecting the digital texture precisely due to technical limitations.
Our brain creates very strong link between what we see (some specific object) and what we expect to feel touching it.
This finding made me assume, that applying e-textures tech this way would make it hard to successfully commercialize it. I was willing to do additional validations to make the final conclusion.
The EuroHaptics conference was approaching, team and I were planning to take part in it in order to demonstrate e-textures app and test it on a wider audience.
Innovation idea
Preparing for the EuroHaptics, I came up with the idea. I thought, what if au contraire to visual-tactile experience which was e-textures, create an audio-tactile one? I believed that it would be meaningful to test both solutions on the same group of users and simultaneously verify whether the perception of the digital texture would be different when not showing the object, but narrating about it.
Hypothesis
An audio narration, normally, makes our brain visualizing the context of the story, however, the "image" is quite blurred, and not very precise, or abstract. And if augmenting it with an abstract tactile sensation, it creates a great synergy between audio and the sense of touch empowering each other. As a result, the brain gets a holistic image and users feel more precisely what they hear in the audio.
Fortunately, the manager has supported the idea, and my team and I have started working on it.
How it was made?
Step 1 Find a right short story
To create audio-tactile experience, it is required to have a short piece of nice audio narration supplemented with sound design. An audio recording must fulfill the following requirements:
the experience should be 1-3 minutes long;
a short piece of narration should have enough keywords & sounds ‘translatable’ into haptic pattern;
the piece should be somehow logical (have a beginning, culmination, and ending) so that the visitors can get the idea.
Crossroads of writing and IT
I checked dozens of guided meditations, audiobooks, songs, ASMR tracks but I was not happy with the search results. Therefore I decided to write a short essay myself, which was supposed to tell a meaningful story. It was a truly amazing moment, because, for the first time, my writing skills were useful at my half-scientific IT work.
The topic came easily. It was late March 2022, the second month of russia full-scale invasion of Ukraine. Being abroad, I was in deep pain and only wishing for Ukraine’s Victory and returning home as soon as possible. So, I wrote an autobiographical emotional story about my future return to the homeland, to Ukraine.
Writing the essay, I was imagining my grandparents’ village home. They passed away a long time ago, but I have a deep connection to that place. This is where the essay’s original name “The Deserted Home” came from.
In the ground of the audio-tactile experience lies my personal emotional story rooting to my ancestors and homeland.
Later on, at Ukrainer I've accidentally seen a photo of a quite similar old village house, typical for my region, with a tree nearby and an old fence. You can see it below to better imagine the story.
While writing the essay, I was tasked to harmoniously inject as much as possible keywords and phrases describing different tactile sensations and sounds which can be translated into a "touchable" and "feelable" by the listener.
You can listen to an audio version of 'The Deserted Home' below:
Step 2 Sound design
In order to immerse the listener deeper into the atmosphere of the narration, I was required to add environmental sounds. This, as well as the voiceover, was done by a professional sound designer and voice actor.
I just prepared the mapping of keywords to sound effects and provided it to the sound designer. Below are some examples:
Step 3 Haptic design
The following step was to create haptic effects. For this purpose, at first, I had to identify all sounds and keywords that could be translated into the tactile pattern. Therefore, I prepared for the engineers another type of mapping along with the description of tactile effects.
In the table below, you can check a few examples:
Step 5 Develop a desktop application
When the specification with the patterns (including voiceover and augmentation with sound design recording) was ready, finally, the engineers could start programming an application.
For the demo version, we developed a simple interface, where except for the audio recording, it displayed the text of the short story, and the keywords for which a haptic effect was applied were highlighted in blue. So that, the user could better connect what they hear with what they feel.
Step 5 Testing
To achieve a great result there was a lot of testing and calibration of haptic patterns, such as adjusting the pattern itself, intensifying of the sensation, and setting exact timing to synchronize the tactile effect with the keyword pronounced or with a sound effect.
Most of the testing happened in my ‘home lab’ as reflected on the photo below.
How does the experience work?
To go through the experience, the visitor wears good acoustic headphones to hear the audio story, looks at the screen with the text and highlighted keywords, and ‘feels’ those keywords via the haptic device.
Since the user can literally touch some of the words or sounds from the narrative and the narrative itself is an emotional story, the experience received the name "The Touching Story".
Watch this video to see how the experience works as well as learn some application cases.
Conference
The team and I managed to achieve what was planned in the initial concept of audio-tactile experience. E-textures demo along with "The Touching Story" went to EuroHaptics 2022.
In a week of a conference, both solutions were tested by hundreds of visitors. "The Touching Story" experience received very warm feedback from the haptic community.
Based on that feedback I managed to confirm the hypothesis: audio narration augmented with tactile effects was much better perceived by the users than visual augmented with tactile effects. The visitors have mentioned better correspondence of tactile sensation to what they hear. And tactile effect indeed makes people deeper immersing and engaging into an audio experience.
Results
So now it was clear, that in the further e-textures technology commercialization efforts we need to rely more on the combination "e-textures + audio". It was a very important conclusion because R&D is quite expensive and we have to drop non-working use cases as soon as possible. As a result of the conference a case of "e-textures + object visualization" was dropped.
The idea of "e-textures + audio" was an innovation, consequently described in the scientific paper and aiming for patenting.
Later on, this experience was traveling around many other conferences, to name a few: Talent Land, Mexico; HAID, UK; IMET, Cyprus; CHI, Germany; IMEX, France. This also proves that it was a successful invention.
Eventually, I wrapped it up into a wider business case, namely, to apply novel tech such as e-textures to creating interesting engaging experiences, which allow users to immerse and live through it on a new level. This is how I launched a new service at the company offering the clients to create immersive experiences.
Outro
In this article I shared my personal experience from generating an idea of innovation to its implementation; from hypothesis testing to transforming the idea into a commercial service. I hope that it shed some light on "how it was made", and what role non-tech skills such as writing played in it.
I'd like to mention, that the key factors of success here were: natural curiosity starting with the phrase "what if?"; desire to innovate, create something new from idea to a product; creativity and readiness to go outside of the written in the job description responsibilities; lack of fear to describe and pitch the idea to stakeholders; and finally, the vital factor — support from the direct manager who approved the initiative, provided the budget and assigned the engineers for this mini-project.
So, my main message in this summary addressed to the creators would be not to be afraid of experimenting, articulating ideas, mixing and merging your experiences from different areas and... innovate, innovate, innovate!
Until you articulate the idea, no one knows about it.