With the experimental augmented reality mode for Oculus Quest 2 glasses, Facebook wants to give an idea of future XR glasses and give developers time to experiment with this technology. Quest 2 AR applications should be available to everyone from the end of the year.
Thin AR glasses with a lot of technology are expensive and complex. They also have visual disadvantages, such as transparent graphics or a narrow field of view, which will not be eliminated in the near future. Physics gets in the way.
The highest quality augmented reality is possible today and probably in the long term with the help of virtual reality glasses with built-in cameras. The fact that companies such as Varjo or Lynx, offer in high quality, Facebook can upgrade Quest 2, at least experimentally, into a low-budget version with a software update.
Quest 2: Launch of experimental end-to-end AR transmission technology for developers
Together with the new update 31.0 for Oculus Quest 2 Facebook is launching an end-to-end API in the Oculus Unity development environment. This will be followed by support for other engines, as well as the production version of the end-to-end API.
This update probably won’t make much difference to consumers, as the Quest 2’s low-resolution black-and-white review leaves a lot of room for improvement in terms of quality.
However, developers can experiment with the end-to-end AR technology in Quest 2, implement the initial application concepts and explore Facebook’s next VR glasses. This should certainly provide AR transparency at a higher level and maybe even sell directly as mixed reality glasses if it provides high quality AR and VR with permeability between the two technologies.
Developers will be able to start building and testing mixed reality experiences on Quest 2 with Passthrough API Experimental—an important step in unlocking the next generation of immersive experiences: https://t.co/BrGNTgHKSU (going to miss getting this FAQ at my AMAs =)) pic.twitter.com/y9koPSxiHm
— Boz (@boztank) July 23, 2021
AR apps for consumers too
AR applications based on the productive version of the end-to-end API can then be sold to all users in the Oculus Store. According to Facebook, this is expected to happen by the end of the year. In a blog post, Facebook shows the first demonstrations of possible applications.
Passthrough AR with Oculus Quest 2 is suitable, for example, for remote work with colleagues, when you look at virtual screens and avatars through cameras, but at the same time have access to your real desktop accessories, such as a mouse and keyboard. The permeability between VR and AR can be smoothly adjusted.
Other application scenarios are games that more closely combine the virtual and real world, for example, zombies appearing in the living room, or the ability to draw 3D art in the real world.
Developers can integrate real objects, such as a keyboard, into a virtual environment. End-to-end augmented reality is also designed to better interact with people or pets in close proximity.
Facebook turns to AR Data protection
The device orients itself in the room with the help of security cameras built into Quest 2. In AR mode, it becomes even more obvious that Facebook, of course, can look into the user’s environment. It doesn’t take much imagination to imagine that Facebook could, for example, use fast image analysis algorithms to create a list of what items are in the household in the shortest possible time in order to collect as much marketing data as possible.
At the beginning of the year Facebook has introduced a new AI for image analysis that can quickly and reliably separate several objects in one image, as well as the background and foreground, so that it can then create a suitable image description based on it. For teaching artificial intelligence Facebook used Instagram’s billions of users to tag images on Instagram.
Facebook’s new artificial intelligence for image analysis can identify more objects in an image and determine their spatial location, for example, distinguish between the background and foreground, as well as recognize their size. This creates an additional context for the image description. | Image: Facebook
At least for the current end-to-end technology, Facebook promises that this will not happen: AR applications do not have direct access to video recordings and they are not allowed to save any sequences of them. Direct images from cameras will be processed only on the device. The end-to-end transmission technology was developed “with data protection in mind”.
In any case, low-resolution black-and-white images from Quest 2 are probably not suitable for automatic analysis. This will not always be the case, and, as past experience shows, Facebook’s attitude to data protection issues can also be adjusted later.
