ARCore has an excellent feature – light estimation. The ARCore SDK estimates the global lighting, which you can use as input for your own shaders to make the virtual objects fit in better with the captured real world. In this article, I’m taking a closer look at how the light estimation works in the current ARCore preview SDK.
Note: this article is based on the ARCore developer preview 1. Some details changed in the developer preview 2 – although the generic process is still similar.
Following the basic project setup of the first part of this article, we now get to the fascinating details of the ARCore SDK. Learn how to find and visualize planes. Additionally, I’ll show how to instantiate objects and how to anchor them to the real world using Unity.
The ARCore example contains a simple script to visualize planes, point clouds and to place the Android mascot. We’ll create a shorter version of the script here.
ARCore by Google is still in preview and only runs on a select few phones including the Google Pixel 2. In this article, I’m creating a demo app for ARCore using the ARCore SDK for Unity (Preview 1).
It’s following up on the blog post series where I segmented a 3D model of the brain from an MRI image. Instead of following these steps, you can download the final model used in this article for free from Google Poly.
Previously, the AR efforts of Google were focused on the Tango platform. It included additional hardware depth sensors for accurate recognition of the environment. Unfortunately, only two phones are commercially available equipped with the necessary hardware to run Tango – the Asus ZenFone AR and the Lenovo Phab 2 Pro.
In the first part of the article, we captured a 360° photo using a Samsung Gear 360 camera. Now, we’ll create a new Unity project for Android. Using the right shader and material, we can assign the cylindric projection to a Skybox. This is the perfect 360° photo viewer for Unity, which can then be easily deployed to a Google Daydream / Cardboard VR headset!
The Skybox in Unity is the easiest way to show a 360° photo in VR. Note that 360° 2D and 3D video will be supported out-of-the-box in the upcoming Unity 2017.3 release, according to the current Unity roadmap.
For setting up a 360° panorama as a Skybox, the following guides are extremely helpful if you need further pointers: SimplyVR, Tales from the Rift. The instructions below outline all the necessary steps you need to create your own 360° photo viewer in Unity!
Capturing a 360° photo / video and viewing it in VR is one of the most immersive use cases. As the user is part of a captured real world, the virtual experience is the most life-like possible.
In this article, I will show how to capture a 360° photo using the new Samsung Gear 360 camera (2017 version), then load the photo into a Unity project for Google Daydream / Cardboard to view it in VR on an Android phone (in this case, the Google Pixel 2).
When dealing with Augmented and Virtual Reality, one of the most important tasks is capturing real objects and creating 3D models out of these. In this guide, I will demonstrate a quick method using the Intel RealSense camera to capture a point cloud. Next, I’ll convert the point cloud to a mesh using MeshLab. This mesh can then be exported to an STL file for 3D printing. Another option is visualization in 3D for AR / VR, where I’ll also cover how to preserve the vertex coloring from transferring the original point cloud to Unity.
Update 20. December 2017: A new release of the Mixed Reality toolkit is now out as an official release. It’s recommended to use this, along with Unity 2017.2.1f1.
Update 13. November 2017: The latest source code of the Mixed Reality toolkit now combines both HoloLens and Mixed Reality headsets into a single toolkit that works with one Unity version: 2017.2.0p1 MRTP 4. It’s a special fork of Unity that is optimized for the “Mixed Reality Toolkit Preview”. A later version of Unity will hopefully combine all environments into a single release again. Read more about the environment setup at the GitHub pull request.
Update 19. October 2017: In the meantime, Unity 2017.2 final has been released, and the dev branch of the Mixed Reality toolkit has been merged back to the master. You should now be fine using the following versions for HoloLens development: Unity 2017.2.0f3+, Mixed Reality Toolkit (master branch), Visual Studio 2017.+4, Windows 10.0.15063.0 SDK.
Original Article: Lately, the tools required for HoloLens / Mixed Reality development have been undergoing profound changes. All three tools involved in building HoloLens apps are being restructured:
With the latest Unity 2017.2.0b11 release, everything should now be coming together. In this blog post, I’m describing how to use the latest versions of the tools for creating and deploying a HoloLens app.
When working on Unity HoloLens-projects in teams, sometimes merge conflicts in Unity scenes are unavoidable. Even though the Unity scene file format is text-based, the automatic merge of a standard GIT merge tool wouldn’t always correctly recognize the changes from different versions.
Luckily, Unity comes with a merging tool that is specialized on scene files: UnityYAMLMerge / Smart Merge. However, it’s not straight-forward to integrate into a workflow.
Did you follow the blog post series on setting up Spectator View for HoloLens? The following issues frequently occurred when testing Spectator View. To make your life easier, make sure you check these troubleshooting tips & tricks:
Between your two HoloLenses, there can be some offset in the placing of the holograms. In one HoloLens, you can for example position your hologram directly on the table; on the second HoloLens, it’s one meter next to the table.
In the final part of the HoloLens Spectator View series, I’ll add the final tweak to my sample project and show you what kind of photos the Compositor saves on your PC. Getting to this point has taken a lot more work than expected (+ a lot more blog posts, too!), but the journey was worth it! You can finally get great photos and videos of the mixed reality experience on HoloLens.
Sending the updates didn’t work right away, so I searched for other changes that the sample does differently from my app that might be responsible. One such case was the Script Execution Order.
In Unity, the order in which scripts are initialized is arbitrary. When debugging, I found an issue that an instance of a class wasn’t available yet when a script wanted to access it.