Keyflows: Innovating Screen-less Typing for Visually Impaired

Keyflows: Innovating Screen-less Typing for Visually Impaired

Enterprise UI

User Experience

IoT Design

AI/ML

+28%

Improved usability score on the user experience, navigation and gesture usage.

-30%

Reduced task on time required for typing, editing and deleting sentences on-the-go.

Introduction

User Simulation and Experience Research Lab managed by Prof Davide (Dept. Chair at IU) was offered a grant by Google and NSF (National Science Foundation) to explore navigating technology(typing, browsing) for Blind and Visually Impaired (BVI) community as the apps rely more on screens. However BVI users face issues with privacy and safety to use screen-centric apps in public and social settings. We explored efficient ways of auditory navigation by performing exploratory and comparative research studies to identify the context of usage, and compare current apps used, for easy adaption.

MY ROLE

Research, analysis, conceptualization, information architecture, designing high-fidelity prototypes and evaluation.

TEAM

2 Product Designers

2 UX Researchers

1 BVI Researcher

3 Stakeholders

DURATION

2 Years

SOFTWARE

Proof of Concept

Beta prototype aided by Tap-strap and Android phone

Beta prototype aided by Tap-strap and Android phone

We built a beta version of the auditory keyboard- Keyflows using a bluetooth Kit integrated with Android package manager, and coded an algorithm for navigation, three-layer character paradigms, and mapped text entry gestures. This has an ability to move linearly across the characters to select, delete and edit the sentences using gestures then created a video simulations to evaluate its performance. In the following video, I am using Keyflow to type “leaving @ 2 PM” using auto-suggestion feature.

Proposed Screen-less keyboard: keyflows in use, to type on-the-go

Design Process

2 phases of research studies followed by design recommendation

2 phases of research studies followed by design recommendation

We performed initial research followed by two studies: usability and comparative testing. We took the advantage of audio simulations and video prototyping to validate the usability and efficiency of the product. We followed a technology driven approach due constraints with unavailability of reference resources and developers in the team to implement design recommendations.

High-level design thinking process

Research

Decoding user pain-points with screen based typing

Decoding user pain-points with screen based typing

To identify focus points and develop a feature roadmap, we interviewed the Blind individuals
and cleared out initial assumptions.

Literature reviews — We researched related paper works and articles on the topics of accessibility for BVI, partially impaired. This set the base of what to look for in interviews and research studies.

Expert Interview — We interviewed BVI professionals, who are tech savvy and use accessibility product to perform day-to-day activities.

Interaction with the expert during one of the lab sessions

Visually blind user's challenges day-in and day-out

Visually blind user's challenges day-in and day-out

Looking from the lens of “Hierarchy of Needs in Design Terms”, currently, digital interfaces are less shaped to fit-in the contextual usages of the BVI users. According to the World Health Organization (WHO) 285 million people are Blind or Visually Impaired (BVI). They face these challenges on daily basis:

Inhabitant to screens

Replace manual effort & reduce human errors by standards for On-the-Go Inventory Tracking.

Complexity to type-out a text

Help users easily complete their tasks - nozzle manufacturing rather on Miscellaneous tasks.

Cane in one hand and dog-leash in other.

PO Cycles should adapt to variable levels of inventory information density across verticals.

Privacy concerns and social inclusion

PO Cycles should adapt to variable levels of inventory information density across verticals.

Ideation

"Linear and predictive" navigation is usable and useful

"Linear and predictive" navigation is usable and useful

We created two initial low-fidelity prototypes. After thorough expert evaluations and cognitive walkthroughs, we chose Alphabetical layout based on the pros vs cons list. Then prototype was developed to validate the concept.
Alphabetical layout: User controlled, linear scanning of characters and enabled to switch to edit mode.
Probabilistic suggestion of letters until three “grams” after which it switches to suggesting words. Then advanced prototype had been created using android and MYO’s android SDK. The reason behind the programming the prototype was we wanted to present something to the expert users to spark the conversation about the concept and see what different ideas emerge.

The interaction model with defined gestures on the left

Usability Testing

Recruited 20 participants, employees from BOSMA Enterprises

Recruited 20 participants, employees from BOSMA Enterprises

BOSMA, an organization that provides workforce development and training to the blind and visually impaired.


Our mission was to understand the following when a blind individual behaves and experiences during their first exposure to a screen-free, entirely auditory keyboard:

Investigate the user experience and performance.

Gauge their navigation behavior in controlling keyflows

Understand the limits and potential of this approach to enhance the accessibility of typing

Qual results

Qual data to access system performance

Qual data to access system performance

There were major themes: Convenience factor, form factor constraints, system adaptions, user control vs system control.
This qualitative data helped us to learn the user behavior and the quantitative data, to validate the performance of the system.

Convenience factor

Phone out of sight

The user felt convenient to type the sentence without having to hold the phone.

User Control vs system control

Customize the auditory speed

Users wanted to control the speed of the text to speech(system response).

New system adaption

Slow but accomplished the tasks

Most participants were able to type at least one word in screen-less mode but typing took an inordinately long amount of time.

Form factor constraints

Breakpoints in the experience

The Myo armband posed barriers to optimal and fluid interaction.

New system adaption

Convenience factor

Novel technology for an interaction model

The user favor the adaption of this new system mainly the convenience it offer irrespective of the context of usage.

Moreover, the navigation experience keyflows offer seemed easy to use with less learning curve.

Quant Results

Quant Results

"Accurate but slow" typing impacted by form factor and system responses

"Accurate but slow" typing impacted by form factor and system responses

"Accurate but slow" typing impacted by form factor and system responses

High Learnability

Easy to learn system layout and interaction components

Low Recall rate

Difficulty in recalling the gestures and context

Low error rates

Limited gestures improved typing performance

High Task on Time

Typing combination chars led to context switching

Low Cognitive load

Linear nav and simple gestures has less cog load

Ideation

Design iteration to improve "task on time" and recall rate

Design iteration to improve "task on time" and recall rate

The keyflows concept is again modified to incorporate the insights drawn from the usability study as it impeding the task achievement time which is defining the purpose of quick messaging.

We explored nimble ways to model to reduce the cognitive load. To develop the prototype, we performed a few expert evaluations of each layout then we finalized on a hybrid layout of the all the three.

Followed by prototyping with alternative screen-free input device. Along with that we ideated suitable gestures such as taps and mapped the functionalities. To develop the prototype we used android and TapStrap’s android SDK. These are the following features:

The interaction model with defined gestures on the left

Comparative Analysis

Learning Keyflow’s potential over VoiceOver

Learning Keyflow’s potential over VoiceOver

To assess how our new design performance, we sought to do a comparative study against an accessibility industry standard, VoiceOver.  

We strategies the study plan and recruiting procedure aligning to IRB protocols. We conducted the interview sessions with 12 BVI individuals, who have the used Keyflows functionality before, over the span of 4 weeks and interview session for an hour long.

Which device is preferred and why?

Why Voiceover is faster?

Can Keyflows be a substitute to VoiceOver; if so how?

What are the potentials of screen-less approach to enhance the typing usability?

The comparative representation of complex screen-based VS simple auditory/gestured based keyboard

Qual results

Qual data to access system performance

Qual data to access system performance

Similar to the first version, the V2 was also been validated for its design criteria through formulated approaches for screen-less typing such as Degree of using a specific gesture to perform a task (typing or editing), User Burden, System Usability.



We inferred that the high cognitive loads and learnability was due to the increase in the number of gestures. Yet, this predictable layout of the model and landmarks for quick access led to high efficiency, task on time, and accuracy.

Acquired user preference

Typing got faster and better over time

Though voiceover posed complexity to type, user performed the tasks efficiently. Moreover, there users with partially blind were able to see the screen.

Socially acceptable

Social inclusivity is a pay-off

Keyflows can be only a situation substitute( Social inclusivity) of VoiceOver but not always. Even to type the quick messages on-the-go or while engaged in a social setting.

New system adaption

Convenience factor

Gestures of the new-off-the-shelf device was an overkill

Users preferred Keyflows, but it lacked in the performance due to difficulty caused by device and gestures - having to recall and confusion caused while mapping them to their functionality.

Preference for privacy

90% of users prefer Keyflows

The user felt convenient to type the sentence without having to hold the phone vs the voiceover.

System customization

Customize the layouts

Users can build the sense of ownership when they the system has the ability to manipulated the layouts to match their typing patterns.

Quant Results

Keyflows for quick messaging in social environments

Keyflows for quick messaging in social
environ-
ments

Keyflows for quick messaging in social environments

Improved Efficiency

Switching layouts and the typing was efficient

High Accuracy

Predictive layout improved typing with no errors

Low Learnability

Of the system layout and interaction components

Low Task on Time

Typing combination chars led to context switching

High Cognitive load

Three paradigms nav and gestures has high cog load

Solution

Solution

Keyflows use cases to address BVI's pain-points

Keyflows use cases to address BVI's pain-points

DESIGN GOAL: Personalization

With ability to Toggle between user to system controlled

With ability to Toggle between user to system controlled

USER SCENARIO

Manual tap to select a character or to navigate across the layout can be tedious in contexts such as traveling on bus, Switching attention between two tasks (hearing an announcement and system audio feedback).

SOLUTION

Users can scroll through the letters or let the system control it gives the user flexibility when their hands are full.

DESIGN GOAL: SIMPLE

Auto-suggestions mode

Auto-suggestions mode

USER SCENARIO

Drawing similarities from current keyboards features, not having access to auto typed letter can lead to active attention allocation on the typing task.

SOLUTION

System suggests the user with frequently used words or trigram words for improved efficiency.

DESIGN GOAL: Flexible

Customize Layout

Customize Layout

USER SCENARIO

Usually BVI disabled users are trained to use technology and are accustomed to operated them at varying speeds, much higher than a normal abled user. When in a meeting or occupied with other tasks, it can be difficult to type combination character sentences.

SOLUTION

Get the flexibility to modify the default keyboard layer based on their typing patterns.

Learning

Product Innovation

Product Innovation

The project was a innovative solution which pushed my creative abilities. While in the process, I learnt..

Striking a balance between features and complexity. Making it seamless and even thinking about error rectification aspects.

To design better for a Auditory User Interface(AUI) and to improve it we used Amazon Polly, Simulation videos and developed prototypes to gauge the prototypes pros and cons.

Doing pilot tests and documenting results to understand where the system could be improved to get the best out of the actual study.

The project was a innovative solution which pushed my creative abilities. While in the process, I learnt..

Striking a balance between features and complexity. Making it seamless and even thinking about error rectification aspects.

To design better for a Auditory User Interface(AUI) and to improve it we used Amazon Polly, Simulation videos and developed prototypes to gauge the prototypes pros and cons.

Doing pilot tests and documenting results to understand where the system could be improved to get the best out of the actual study.

Do more of…

Do more of…

Objective set out to improve the Keyflows

Intuitive gestures by changing four finger tap to thumb and pinky tap as the latter signifies the start and end point.

Leveraging scrolls over taps to reduce the gestures inputs

Enhance efficiency in terms of audio cues by using affordance and signifiers that have close relation with the functionality and the context of the paradigms

That's me giving demo and briefing on the next steps to the clients

Thank you!

Thank you!

If you have any more questions or want to know more details, please don't hesitate to contact me. For now, please consider checking my other work, my experiments, or learn more about me.

Scroll to top