visionOS 2 better integrates iPhone, Mac and input devices

AirPlay Mirroring on Apple Vision Pro with visionOS 2 dramatically expands the headset’s capabilities, and new features better integrate your iPad, Mac, keyboard, and controllers, too.

The first VR headset to ship after Facebook’s acquisition of Oculus in 2014 was the Samsung Gear VR Innovator Edition, using the Note 4 smartphone as the display, processing center and battery. The Gear VR went through multiple generations before proving that cheap wasn’t cheap enough.

Later, the PC-powered HTC Vive from SteamVR promised to pair with your phone to keep you connected while in VR. Most people who owned a Vive didn’t know this feature existed, and there are few accounts of anyone using it successfully.

Post-iPhone

Previous phone attempts at VR have been lost to history because neither the unholy pairing of Samsung and Facebook-acquired Oculus nor HTC and Valve had enough control of the platform to keep you tethered to your computer anymore. personal while in VR.

In the case of the Gear VR and the cardboard boxes sent by Nintendo, Google, and the New York Times, you could (and still can!) enjoy a simple head-tracked game like Proton Pulse inside an actual cardboard box — just like you can today on a $3500 Apple Vision Pro.

What has changed is that Apple’s 10 years of vertically integrated software and hardware advancements make the iPhone a Swiss army knife for visionOS. With visionOS 2, nearby devices running macOS, iOS, and iPadOS become parallel computing tools to turn to and use when needed.

In visionOS 2, for example, when you connect one of Apple’s keyboards to the headset, it will detect your immersive environment and become another tracked object you can use.

As another example, you can look at the existing Apple Watch, which already includes a remote camera that provides wrist-based viewing and recording buttons for the iPhone. The feature allows photographers and videographers to pan or frame their shots just right, without requiring the iPhone’s screen to be directly visible for framing. Add visionOS to the mix and you can go a step further, with a pair of buttons to press on your wrist to activate the phone’s camera for recording photos or videos.

Taken as a whole, Vision Pro with visionOS 2 becomes a more capable director by unlocking other Apple devices in slightly new ways. One hand can frame the iPhone’s camera photo, wrist buttons are there for controls, and a virtual viewer floats wherever you look to show you what your photo looks like from any angle you choose.

That’s the difference of a decade and tens of billions of dollars in investment to make this shiny-faced computer. Apple’s initial visionOS 1 operating system launched in February alongside the headset and proved that, with next-generation resolution and OLED contrast as a backdrop, a VR system can actually stand on its own as an infinite canvas for your media.

visionOS 1 began to tell this story by offering the Apple TV and Apple Immersive experiences as its flagships, with classic game emulators suddenly added to Apple’s showcases shortly after initial Mac support for mirroring your computer to a headset. visionOS 2 makes things even more interesting by bringing your iPhone and other devices through AirPlay Mirroring.

Starting with visionOS 2, an iPhone connected to charging via magnetic connection can mirror its widgets even when it’s locked. And when is it unlocked? You can view iPhone on a giant screen or on a small screen that hovers right above your keyboard — it’s your choice.

iPhone mirroring in full VR is much more than a minor improvement here. And it works for iPad too.

All in all, AirPlay Mirroring shows why Facebook bought Oculus in 2014, and it’s being shipped by Apple within the first six months of its launch in 2024. I’ve watched a kid play Quest 3 from an Apple Vision Pro while casting from a web browser through the virtual Mac screen. With iPhone mirroring, I might be able to monitor two Quest 3 headsets this way, assuming the high-bandwidth, very-low-latency data stream on my network doesn’t melt my Wi-Fi router. In the other direction, Meta is asking Apple to allow the Quest to be an AirPlay receiver.

The image featured in this post was captured directly from Apple Vision Pro showing five simultaneous windows:

• Stardew Valley purchased from Steam running locally on a Mac Mini via Virtual Mac Display.
• RetroArch runs on headphones.
• Delta emulator running on headphones.
• ALVR connects in the background to a local VR-ready computer.
• The iPhone’s standby window showing the clock and calendar widgets while it’s locked and charging in the dock.

Any iPhone manufactured in the last four years can be mounted for charging this way using MagSafe. A decade ago, you’d have to give up the juice needed to keep your constant companion moving just to enjoy a few minutes of uncomfortable VR in which you can no longer use your phone as a phone.

Roughly a decade after those false starts and, instead of reaching for a PC or using a phone, Apple opted to ship its first VR headset with a phone-sized battery. David Heaney, in his final review of the hardware, rightly criticized the dangling cord that the battery forces on Vision Pro buyers. With visionOS 2 beta running on my headset, I understand why Apple released the device without clock or calendar widgets. That’s because, for visionOS, the iPhone is just another window that takes you away from Bora Bora.

visionOS recognizes that, trusting Apple with your sense of sight, everything in your house is of little use in VR if you can’t use it at least as well as without a headset. This means, for example, that future improvements to the Mac’s virtual display will give you an ultra-wide, high-resolution display wherever you are.

Even the iPhone that built Apple’s spaceship headquarters is little more than a widget for visionOS that informs you of events outside of virtual reality. You can configure visionOS widgets, for example, to quickly show you the air quality outside your home and the temperature outside your climate-controlled environment. Now, you can keep an eye on that DoorDash order on your iPhone without taking off your headphones, just like Wade Watts managed as Gunter in Ernest Cline’s original Ready Player One novel.

And when it comes to gamepads, tilting the analog stick on a gamepad in visionOS 2 now navigates system menus. Having a single Joy-Con in your hand hanging from a wrist strap gives you the basic inputs for many platform games while also more easily navigating the operating system at the same time. You can rearrange apps faster, for example, by using the stick to move through app pages.

visionOS 2 was officially announced for Apple Vision Pro

visionOS 2 introduces a number of new features and improvements across the operating system.

There’s much more to see in visionOS 2, including much-needed improvements to hand tracking, and we’ll continue to cover Apple’s spatial computing operating system as it continues to evolve.

Ethics Disclosure: Ian is married to an Apple employee.