Apple has shared some details concerning accessibility features it is working on, dropping some pretty big hints at how it sees augmented everyday reality. Will we see more on this at WWDC 2022, and how will it be applied?
Making life accessible, making reality data
Two of the upcoming accessibility enhancements seem to suggest Apple’s approach: Door Detection and Live Captions. Here’s what they do:
- Door Detection: Using the iPhone camera, it will detect a door, navigate a user to that door, inform them if the door is open or closed, tell them how to open the door, and it can understand and read things like door numbers.
- Live Captions: Your Apple device will listen to any audio content and give you a real-time transcript of that conversation.
Both are incredible features, but when you consider them a little, they become quite amazing. I see it this way: once an Apple device can create a real-time transcript of what it hears, why should it be unable to translate that transcript into different languages?
What this could mean
We know Apple has the technology to do this — we use it each time we translate a web page. That process is super-fast, so why not simply extend that translation to the transcription delivered by your Apple device?
This could work two ways, also, with your device speaking the language you can’t, enabling you to join complex conversations across multiple tongues.
Door Detection uses technologies Apple has been exploring for some time. You can easily use them yourself — open Photos and search for images of “Lamp Post” and you’ll be able to explore every photo you have that includes a lamp post.
Now, I don’t know about you, but if your device can recognize items in photos, it should be able to recognize them elsewhere using the same machine vision intelligence.
Vision + Intelligence + Context =?
That means that, just as a person who is blind or has low vision can look forward to using Door Detection to find and open a door, it’s reasonable to think they’ll be able to use similar technology to recognize anything else the AI on Apple devices has a name for:
Door Detection tells us this will happen because the technology already exists to enable it. It just needs building out.
So, what’s revolutionary about all of this? It means Apple has already assembled a host of building blocks that enable its technologies to recognize and interact with the world around us. Once technology understands that worls, it can help guide our interactions, augmenting our decisions with information we can use.
A blind or low-vision person about to buy a $1 orange might be told the same fruit is available for half that price further down the street. Or a field service engineer might find their device has already opened the troubleshooting manual for the hardware they happen to be staring at.
[Also read: Apple calls out Meta for hypocrisy]
What we have here are two technologies, ostensibly built for accessibility, that also give the company’s devices interactive understanding around sight and sound. That understanding enables the device to provide contextually useful information concerning what is seen and what is heard to the user.
This can be in response to direct questions, or, reflecting the work Apple has been doing with Siri Suggestions, driven by the device’s knowledge of the kind of help you usually request.
The augmentation of human experience has begun
You don’t need to be an enterprise professional to recognize that this opens a range of opportunities for powerful tools and services for consumer users — along with profoundly powerful enterprise applications around machine vision intelligence and Industry 5.0 across multiple sectors.
One of the great things about those applications is that, because they are based on accessibility technologies, they also enable those who may not yet be as equally represented as they should be in some fields to take more active part.
That’s what I call augmented reality. And that’s what I think we’re about to learn a great deal more about at WWDC 2022.
No wonder Apple has begun to leak information about showcasing these technologies to company directors and the design challenges that went into developing the most logical vehicle for such tech, Apple Glass.
Step-by-step, the building blocks of this multi-year effort are falling into place more rapidly now. I can already hear the critics getting ready to be wrong again.
Please follow me on Twitter, or join me in the AppleHolic’s bar & grill and Apple Discussions groups on MeWe.