design brief
BABL is a design challenge project for Microsoft Design Expo 2017 with the prompt Intentional Design for Positive Cultural Impact in Mixed Reality. It asks us to consider the context and implications of the technology we’re designing for, while simultaneously demonstrating its clear value and differentiation.
Mixed reality creates a space for new multisensory experiences that span across space and time. Our team explored mixed reality concepts from the Microsoft Hololens website and brainstormed several fields where it could be applied intentionally and responsibly. Among many possibilities, we identified embodied learning as the most promising area for mixed reality.
Over the course of four months, we conducted three main types of design research—exploratory, generative, and evaluative—and presented our findings to the Microsoft UX design lab."
DOMAIN: Language learning
When selecting the target problem space, we considered three main criteria: 1) there should be clear access points to begin research, 2) the topic should present an open and well-defined problem space, and 3) it should be engaging and rich enough for us to explore throughout the semester.
After extensive discussion, we collectively agreed that language learning is the most promising domain for applying embodied learning, given the strong demand and the clear challenges in the learning experience.
Learning a foreign language is challenging because it requires significant time and effort, yet mastering it remains difficult. Moreover, the expressions found in textbooks often don't reflect how native speakers actually communicate. We focused on this gap between textbook language and real-world spoken language and chose to address this problem:
"How might we help language learning experience
easier and more efficient?"
territory map: reframing Language learning
We discussed how we learn our mother tongue and then reconstruct the whole process as a territory map. At the end, we felt that full immersive language learning experience comes from true cultural learning and understanding. We believe that language learning is not just about the four key skills we identified, but that it can lead to a deeper understanding of cultures that are different than our own. From this came the ‘clear need’ of learning a new language and understanding a new culture can be intimidating process and how through our design we could address that.
"Language is a byproduct of social & cultural interaction"
EXPLORATORY RESEARCH
Our exploratory research includes fly-on-the-wall observation, participation, expert interview, student interview and online diary study. We visited a couple of language classes, preschools and language professors and explored how students learn foreign languages and what their pain points are.
Through our research, we confirmed that language and culture are deeply intertwined, as we had assumed. Understanding a culture helps people connect to a new environment or community and become part of it. This understanding, in turn, motivates further learning.
Several interviewees repeatedly shared that the most rewarding moment is when they can deeply connect with others, understand them, and build relationships as part of their community. One of our design implications, therefore, is to facilitate meaningful interactions that help people form relationships more easily.
From conversations with experts and secondary research, we learned about the importance of critical thinking in all pedagogical structures. The ability to actively make connections and view both a new culture and one's own from a different perspective is crucial for successfully engaging with and integrating into a new language and culture.
Finally, one key takeaway is that we want to help learners express, not translate. Conversational fluency is less about thinking in one's native language and translating it into another, and more about constructing thoughts directly in the new language.
After completing the exploration, we summarized our findings using the Rose (Promising Space), Bud (Opportunity Space), and Thorn (Pain Point) method, along with journey mapping, to identify what might work well in a mixed reality learning environment and what might not. The ultimate goal of the mixed reality experience is to address the challenging moments when learners begin to lose motivation and interest in language learning, ensuring they continuously enjoy the learning process. At the end of the exploratory research phase, we defined our design principles as follows
GENERATIVE RESEARCH
Over the course of a few weeks, we conducted three workshops with 12 participants, including language professors and learners from CMU, before moving on to future envisioning. Our goal was to understand each participant's language learning process and the solutions they had in mind. To facilitate smooth and effective prototyping sessions, we provided props to help participants express their ideas more concretely.
The ideation workshop was divided into two parts. First, we asked participants to create a map or timeline of their own learning experience, highlighting their feelings at each stage. Then, we asked everyone to envision the 'magic technology' for an ideal learning experience by sketching scenes or using props.
The first exercise provided valuable insight into the pain points that current language learners face. First, traditional classroom learning often fails to maintain interest and motivation, quickly turning into a chore rather than something enjoyable. Second, many learners feel afraid of judgment and embarrassed when they make mistakes. In certain familial or cultural contexts, the fear of failing to meet language proficiency expectations can prevent learners from feeling comfortable enough to try. Finally, there is often a lack of 'real' practice opportunities outside the classroom. As we found in our exploratory research, practicing language in context is essential for learning and retaining language skills, but such contexts are often hard to find.
In the second exercise, participants' ideas focused on themes like sharing cultural experiences, receiving instant feedback, practicing, prompts, and coaching for non-verbal behaviors. Some creative ideas included a karaoke mic that translates songs and captures pronunciation, a virtual assistant for practicing conversations, a projector that simulates a new environment, and magic goggles that alter the world around us.
We also visited the Duolingo office and conducted the same workshop with the content designers. We asked them to reflect on their happiest moments while learning a second language. Common themes included: 1) becoming more integrated into the new environment, 2) building relationships with locals, and 3) successfully communicating in the new language.
In addition to the workshop, we had an in-depth conversation about creating a more natural and dynamic spoken language database. Given the vast variations in language expressions, it's impossible to build a perfect database. They also advised us to consider regional differences in accents and expressions, even within the same language. For instance, if a learner lives in Ireland for an extended period, speaking with an American English accent may seem less authentic to locals.
While synthesizing key findings, three main concept ideas began to emerge: one, providing passive exposure to the language and culture in everyday life; two, an immersive virtual environment for contextual practice; and three, a companion of sorts that would help learners apply what they’ve learned to a real-world context.
BODYSTORMING & RAPID PROTOTYPING
After narrowing down the concept, we made decisions about how a user could interact with our system and what that interface looks like. We started by playing with the Hololens to get an idea of what interactions exist now and what we’d have to consider.
Because “tour mode” is such a prominent part of our concept, we bodystormed with Hololens and experience how the tour mode look like. One of us gave a “tour” of our building and “introduced” us to local people while we were all on Skype and Facetime. One of the local people said, she thought it was weird for her to say hi to us without seeing us, but when she was able to see who she was talking to, it was a lot smoother.
We also roleplayed the learner/host with our 3 key scenarios and personas and checked how each of them worked well. We found that the system can bring big value when it connects distant learners to local people before landing in a new place. On the other hand, reading prompts and translations while having a conversation didn’t work well especially when the prompt gets longer.
From the bodystorming, we felt that our solution should focus more on connecting people and encourage users to speak it out. We think bodystorming and enactment are very useful activity for mixed reality because it helps expect how the user experience would like.
One another interesting method we used was rapid paper prototyping. We made various paper props and played with them and see what kind of technology is needed for our solution and what kind of features are needed for immersive experience. Lo-fi prototyping definitely helped a lot when we figured out the feasibility of a scenario. And it was also super helpful for our evaluative workshops.
EVALUATIVE RESEARCH
Before jumping to the hi-fi prototyping, we conducted evaluative researches including concept speed dating and evaluative workshop. To test our concepts, we used paper prototypes and sketches to walk our participants through the experience of using the system. We asked for feedback, noted their instant reactions to each scene, any hesitations they had about it, and probed on social boundaries.
What we learnt from the UI evaluation workshop was different learners have very different needs in terms of what helps them learn best. Some participants learnt through grammar tables, others liked to practice in a real social context. Also, People have set expectations for giving and receiving feedback on their language use. Something we observed was people want to be as engaged in the moment as possible with as little intervention from the system. Interventions and helping elements like grammar tables, caption etc.. might be helpful in an individual practice mode, but this would interrupt a conversation which isn’t good for building relationships.
The next thing we did was narrowing down from three different concepts to a final concept and creating 12 storyboards. We asked 6 potential users about which part resonates you the most and which part do we need to reconsider.
From concept speed dating, we learned a couple of key takeaways.
People didn’t think simple passive exposure would help that much. Rather, they wanted to maintain their interest in language learning by exposing themselves into more immersive environment where forcing them to speak the language.
When it comes to immersive language environment, they were curious about the sustainability of the solution in terms of business perspective. For example, How to recruit native speakers? Do learners have to pay for hiring a tutor? etc.
And for the same reason, we should be able to provide strong motivation for both learners and tutors to keep participating in the immersive learning environment.
HI-FI PROTOTYPING
In this making phase, we’ve mostly been making decisions about how a user could interact with our system and what that interface looks like. We started by playing with the Hololens to get an idea of what interactions exist now and what we’d have to consider and then mapped out the system and made a list of all the features we wanted to include.
We referred to the gesture interaction guide from Microsoft Hololens Developer Portal and mix-and-matched interaction combinations for features including doodling, bookmarking scenes and targeting objects, etc.
We then moved into paper prototyping for MR, eventually deciding on a tool “tray” that can be expanded and retracted into the bottom/side of your peripheral vision. The tray would contain tools like doodle, pin, stop sharing view, switch perspective, and end call.
We also considered the screen UX flow. We each created a wireframe flow for the sign-on and matching process and then decided together on a singular flow.
At the end, we designed 2 sets of UI elements for practice mode and tour mode and branded our product as "babl", using the color combination of white and orange. Since we are designing the UIs overlaying in the real space, we focused on improving the readability of each icons.
We also made a simple prototype with Unity and checked how the overall look and feel of our UI elements look like in the real world with Hololens. We learned that the system should be able to adjust the UI colors according to the background color and brightness since the contrast became lower in the light outdoor environment. However, because of its limited feature and view field, we had to made an elaborated prototypes with After Effects.
And finally we shot the concept video that explains how BABL works. We used blue screen to combine the videos of two different spaces. It actually worked really well so I didn't have to use rotoscoping to illustrate the MR moment!
Thoughts & PROVOCATIONS
BABL started from the bold idea of bridging between learners and local hosts at distance. While making the concept video, we found it really awkward to ask people to let us shoot them in public. Some people actually feel intimidated to being seen in the video. Besides technical limitation and social awkwardness, we felt that establishing social norm is very important to apply mixed reality technology in the real world. In the future, we might experience the privacy issue triggered by mixed reality, just like Google Glass became a controversial technology because of its camera recording feature.