homework is a new text vertical of
Google Lens that offers step-by-step helps


Stuck on a problem? Quickly find step-by-step solutions, explainers, videos, and results from web for math, history, chemistry, biology, physics & more.

 
 
 
 

InItiative

Education-related questions comprise a significant portion of Google's global search query stream, accounting for approximately 5%. In the United States, about 0.7% of these queries consist of exact copy-and-paste homework questions. With the advancement of Optical Character Recognition (OCR) technology, there is a substantial opportunity for Lens to facilitate students' lives by digitizing homework questions and enabling direct queries to Search.

 
 
 
 
 

Understanding Users


Our target users are US high school students who seek for online help on both STEM and non-STEM subjects. They have a couple of important characteristics:

  • Mobile youth
    : They are very familiar with asking help online or sharing info with others using social media.

  • Multi-screen users
    : They are also used to use more than one devices at the same time.

  • Actively seeking for online learning
    : Especially after the global pandemic, they rely more on online learning and remote helps.

 

With a pedagogy expert, I had a series of foundational research studies to understand our target audience. The study results indicates that students tend to have a repetitive circle between the problem solving stage and the resource finding stage. As a result, they are often struggled to spending a lot of time on finding the right reference, and digesting materials before solving homework problems.

Students also use a variety of tools and resources that vary in utility depending on the complexity of the question and how confident the student is answering it. One interesting findings from the user studies was students tend to prefer learning tools that provide quick solutions, rather than digging reading materials or watching long videos.

 
 

Cross-product content design


Based on the study result, I set up the content hierarchy for our result panel - Lens put short and exact solution matches to the top priority, and delivers longer educational resources at the bottom section.

Providing useful educational content is essential for the success of the homework vertical. I led the cross-product partnership with Search and Education PA to deliver reliable and trustworthy learning materials, including:

  • Step-by-step math solver: Provide exact solutions on a captured math equation

 
 
 

  • Math editor + keyboard: A fallback editor in case of bad math symbol recognition

 
 
 

  • Organized Search results: A set of relevant education materials that saves user’s time on searching them

 
 
 
 

OCR and camera input


Lens utilizes mathematical optical character recognition (OCR) to identify math symbols; however, its accuracy typically falls short compared to reading standard text. This issue is particularly pronounced in US textbooks, where pages are often densely packed with information. Consequently this leads to lower the performance of our math OCR. I got an indication that the performance of OCR improves when two conditions are met: 1. fewer letters are captured, 2. the size of the captured letters is larger.

I tried to reproduce the failure of recognizing math equations in US textbooks, and when the screenshot takes too many math symbols, it often causes two issues:

  1. Symbols are not precisely recognized

  2. Equations with complicated structured got segmented

 
 
 
 
 

Competitive analysis

To understand how major competitors address the OCR failure issue, I conducted a competitive analysis centered on capturing UX patterns. Most homework solver apps that incorporate camera input utilize a framing technique that confines the screen to a specific target question. This framing not only enhances the visibility of the question but also ensures that it is captured at an appropriate size. That said, I also found that allowing adjustments to the frame in live view significantly decreases usability because manual adjustment makes the camera view shaking. This key finding has led me to two significant design questions.

  1. HMW nudge users to capture fewer characters in the view?

  2. HMW guide users to capture characters with higher resolution?

  3. HMW provide good usability while letting users adjust the framing ?

 
 

Introducing ‘framing UI’ to text input

To enhance OCR quality, I introduced a ‘framing UI’ in both live view and post-capture view. However, frame adjustments are only available to the frozen view so users can capture textbook images quickly and conveniently. The framing UI achieves three key objectives:

  • A clear frame layout can immediately prompt users to frame their target homework questions.

  • Framing area limited helps reduce computational cost of math OCR, therefore the precision of math symbol recognition gets higher.

  • Additionally, it aligns with the overall Lens UX framework.

 
 
 
 

Expanding Lens Design system

During the project, I collaborated with the Lens Visual Design team to enhance the Lens design system. This included developing a UI asset library for homework-solving contexts in both light and dark modes, updating the color system, and creating sample mocks.