Work

Play

About

Work

Play

About

Work

Play

About

Work

Play

About

Vision-Enhanced ASL: Promoting Accessibility through VisionOS


Accessibility enhancement for the Vision Pro platform. This initiative is geared towards individuals facing challenges with hearing impairment.


The concepts in this project are entirely hypothetical and are neither implemented by nor affiliated with Apple.

Getting Started

Gestures

I integrated a familiar gesture recognized by the Vision Pro with a subtle modification, transforming it into the emblem for the "sign language mode." The hand gestures prominently symbolize sign language as well.

Clicking

ASL to Text

"Hey" in ASL

Activating ASL Communication with Siri

ASL Siri Integration

The Interactions

Setting up

ASL to text

Users will have the option to configure this implementation on 'settings,' enabling an extra 360-degree hand scan for improved recognition of sign language from various capture angles.

How it works

Using Vision-based sign language recognition (SLR) and training the sign language dataset via cloudKit makes text translation from sign language recognition achievable.

User Flow - Sign Language (ASL) to text

User Flow - Sign Language (ASL) to text


Communication

Be in the moment. Without restrictions.

Live-time ASL to text translation

Apple Vision Pro seamlessly integrates the SLR system into FaceTime and all Apple platforms, offering built-in speech-to-text functionality and now enabling ASL-to-text conversion.

Your emotions
come alive

Utilizing the implemented trained model capable of recognizing sign language and facial expressions, we introduce spatial implementations of the original FaceTime gestures, allowing for reactions to be triggered in real-time.

Productivity

Get on the same page. In the same space.

Accessibility | Siri

Setting up

ASL to text

Users will have the option to configure this implementation on 'settings,' enabling an extra 360-degree hand scan for improved recognition of sign language from various capture angles.

Siri activated through "Hey Siri" trigger motion

Siri comprehending ASL for "What is the weather like today?"

Productivity enhanced with Siri.

Users will have the option to configure this implementation on 'settings,' enabling an extra 360-degree hand scan for improved recognition of sign language from various capture angles.

Design

Living in your space.

Work

Work

Play

Play

About

About

Linkedin

Linkedin

Resume

Resume

Heysu Oh

Last Updated : Nov 2024