Augmented reality transforms any surface into keyboard

Virtual keyboards are a frequent source of frustration for augmented reality (AR) users. The virtual surfaces are slow and error prone, and raising an arm to type on them can cause muscle strain known as “gorilla arm.”

To improve virtual-typing experiences, University of Texas at Dallas researchers have designed a unique interface that allows users to transform everyday objects into typing surfaces within an AR environment.

The patent-pending technology, called PropType, overlays an augmented keyboard surface onto a handheld object and can adapt to curved surfaces.

Student researchers have created a video demonstrating PropType being used on surfaces such as water bottles, coffee cups, books, and soda cans.

“By integrating objects already present in the user’s surroundings, PropType reimagines text input in AR, creating a seamless connection between the physical and virtual worlds,” said Dr. Jin Ryong Kim, Assistant Professor of computer science at the Erik Jonsson School of Engineering and Computer Science.

“PropType capitalises on the tactile feedback provided by the objects themselves, offering better key confirmation and reducing reliance on visual cues.”

The technology provides an alternative to common AR typing solutions such as external physical keyboards, which can disrupt a device-free immersive experience.

“PropType leverages the familiarity of handheld objects to offer a more intuitive and accessible alternative to traditional keyboards, particularly in mobile or hands-free scenarios where conventional input methods are impractical,” Kim said.

Developing PropType involved challenges such as adapting the interface to different shapes and sizes of objects.

The researchers studied how 16 participants interacted with props to understand grab postures and typing gestures.

Then, they developed custom keyboard layouts for different objects. The researchers included an editing tool that allows users to customise keyboard layouts and visual effects.

Kim’s research focuses on haptics, or touch and physical feedback in digital systems, which can include vibrations and heat.

“Touch carries a lot of information; it’s another form of communication that is underexplored in virtual and augmented reality,” he said.