Phone-Powered AR
Phone-Powered AR is augmented reality delivered through an ordinary smartphone rather than through dedicated hardware. The phone's camera captures the real scene, software places virtual objects into that view, and the combined picture appears on the phone's screen. Because almost everyone already owns a capable phone, this is by far the most widely used way people have experienced AR. It is also called smartphone AR or mobile AR, and on this wiki it is treated as one of the main device subtypes of AR, alongside Standalone AR and PC-powered AR.
The two things that made Phone-Powered AR practical at scale are Apple's ARKit framework, introduced in 2017, and Google's ARCore framework, which reached version 1.0 in 2018. Both let a developer track the phone's position in a room and anchor digital content to real surfaces without any extra sensors.
Two modes
Phone-Powered AR shows up in two fairly different forms.
Handheld phone AR
This is the mode most people mean by mobile AR. You hold the phone up, point its rear camera at the world, and look at the screen, where the live camera feed is augmented with virtual content. You move the phone around to look at the scene from different angles, and the software keeps the virtual objects pinned in place. Games like Pokemon Go, furniture preview apps like IKEA Place, and most ARKit and ARCore experiences work this way. The interaction is one-handed or two-handed and the field of view is whatever the screen shows.
Phone-in-a-viewer
In this older approach the phone slots into a head-mounted holder so it sits a few inches in front of your eyes. A pair of lenses sits between your eyes and the screen, and the phone splits its display into a left and right image for stereo depth. This is the arrangement made famous by Google Cardboard, the foldable viewer Google introduced at its I/O conference in 2014 to get VR and lightweight AR onto phones cheaply.[1] The same idea showed up in early and budget AR products that used a phone as both the display and the compute. It freed up your hands, but the experience was limited: a phone screen viewed through cheap lenses is not a real see-through AR display, and the approach has largely fallen out of favor as purpose-built smart glasses and standalone headsets have improved.
How it differs from other AR
The defining trait of Phone-Powered AR is that the device doing the work is a general-purpose phone you bought for other reasons, not hardware designed for AR.
| Subtype | Where the compute lives | Display | Hands-free? |
|---|---|---|---|
| Phone-Powered AR | The smartphone itself | The phone's own screen (handheld), or the phone screen seen through lenses in a viewer | No in handheld mode; yes in a viewer, with caveats |
| Standalone AR | Onboard, inside a self-contained headset or glasses | See-through optics built into the device | Yes |
| PC-powered AR | An external computer the headset is tethered to | Optics in the headset | Usually, the headset is worn, but a cable tethers you to the PC |
Standalone AR devices carry their own processor, battery, and see-through optics, so there is no phone or PC in the loop. PC-powered AR pushes the heavy rendering onto a separate computer and sends the result to a tethered headset. Phone-Powered AR sits at the accessible end of that range: nothing to buy if you already have a recent phone, but you are looking at a flat screen rather than through transparent optics, and in handheld mode you are holding the device the whole time.
Underlying frameworks
ARKit
Apple announced ARKit at WWDC on June 5, 2017, describing it as "a new framework that allows you to easily create unparalleled augmented reality experiences for iPhone and iPad."[2] It shipped to the public with iOS 11 on September 19, 2017, which put AR support on a large installed base of iPhones and iPads at once.[3]
ARCore
Google's ARCore reached version 1.0 on February 23, 2018, announced at Mobile World Congress. At launch Google said it ran on about 100 million Android phones across 13 models from makers including Google, Samsung, LG, ASUS, and OnePlus, and described it as letting developers "build apps that can understand your environment and place objects and information in it."[4] Once it was out of preview, developers could publish ARCore apps to the Play Store.
Well-known examples
- Pokemon Go (2016): Niantic's location game launched on July 6, 2016, in the United States, Australia, and New Zealand. It uses GPS to place Pokemon at real-world spots, and its optional AR mode "uses the camera and gyroscope on the player's mobile device to display a Pokemon as if it were present in the real world." Players can also switch to a plain rendered background instead.[5] It is the example most people point to when explaining mobile AR to someone who has never tried it.
- Snapchat Lenses: Snapchat's AR effects run inside the app's camera, using face tracking on the front camera to attach 3D masks and effects that follow your head and expressions. Snapchat calls these "lenses" rather than filters.[6]
- IKEA Place: IKEA's furniture preview app was one of the first apps built on ARKit and launched in September 2017 alongside iOS 11. It lets you point your camera at a room and drop true-to-scale 3D models of IKEA products into the live view to see how they fit.[7]
Limitations
Phone-Powered AR is convenient, but the convenience comes with real trade-offs in handheld mode. You have to hold the phone up, so it is not hands-free, and holding a phone at eye level gets tiring fast. You see the augmented world only through the rectangle of the screen, which is a narrow, framed window rather than something blended into your natural vision the way see-through optics aim for. And because the virtual content lives on a 2D display you are holding, it never feels as if it shares your physical space the way content in a good Standalone AR headset or pair of smart glasses can. The phone-in-a-viewer setup solves the hands-free problem but introduces its own: you are still staring at a phone screen through lenses, just strapped to your face.
None of this has stopped Phone-Powered AR from being the most common form of AR by a wide margin, simply because the hardware is already in nearly everyone's pocket.
See also
References
- ↑ "Google Cardboard". https://en.wikipedia.org/wiki/Google_Cardboard.
- ↑ "Create Augmented Reality Experiences with ARKit". 2017-06-05. https://developer.apple.com/news/?id=06052017b.
- ↑ "iOS 11". https://en.wikipedia.org/wiki/IOS_11.
- ↑ "Announcing ARCore 1.0 and new updates to Google Lens". 2018-02-23. https://developers.googleblog.com/2018/02/announcing-arcore-10-and-new-updates-to.html.
- ↑ "Pokémon Go". https://en.wikipedia.org/wiki/Pok%C3%A9mon_Go.
- ↑ "Snapchat Lenses: What it is and how it works". https://www.adsmurai.com/en/articles/snapchat-lenses.
- ↑ "IKEA Place, the retailer's first ARKit app, creates lifelike pictures of furniture in your home". 2017-09-12. https://techcrunch.com/2017/09/12/ikea-place-the-retailers-first-arkit-app-creates-lifelike-pictures-of-furniture-in-your-home/.