Getting Started with Google ARCore, Part 2: Visualizing Planes & Placing Objects

Models of Brains (segmented from an an MRI) placed in the real world using Google ARCore

Following the basic project setup of the first part of this article, we now get to the fascinating details of the ARCore SDK. Learn how to find and visualize planes. Additionally, I’ll show how to instantiate objects and how to anchor them to the real world using Unity.

Finding Planes with ARCore

The ARCore example contains a simple script to visualize planes, point clouds and to place the Android mascot. We’ll create a shorter version of the script here. Continue reading “Getting Started with Google ARCore, Part 2: Visualizing Planes & Placing Objects”

Getting Started with Google ARCore, Part 1: Project Setup & ARCore SDK

ARCore - Plane Detection running on the Google Pixel 2

ARCore by Google is still in preview and only runs on a select few phones including the Google Pixel 2. In this article, I’m creating a demo app for ARCore using the ARCore SDK for Unity.

It’s following up on the blog post series where I segmented a 3D model of the brain from an MRI image. Instead of following these steps, you can download the final model used in this article for free from Google Poly.

ARCore vs Tango

Previously, the AR efforts of Google were focused on the Tango platform. It included additional hardware depth sensors for accurate recognition of the environment. Unfortunately, only two phones are commercially available equipped with the necessary hardware to run Tango – the Asus ZenFone AR and the Lenovo Phab 2 Pro. Continue reading “Getting Started with Google ARCore, Part 1: Project Setup & ARCore SDK”

Showing a 360° Photo in Google Daydream VR based on Unity, Part 2

360° Photo in the Google Daydream VR (2017) headset, Unity app running on the Google Pixel 2

In the first part of the article, we captured a 360° photo using a Samsung Gear 360 camera. Now, we’ll create a new Unity project for Android. Using the right shader and material, we can assign the cylindric projection to a Skybox. This is the perfect 360° photo viewer for Unity, which can then be easily deployed to a Google Daydream / Cardboard VR headset!

Loading the 360° Photo in Unity

The Skybox in Unity is the easiest way to show a 360° photo in VR. Note that 360° 2D and 3D video will be supported out-of-the-box in the upcoming Unity 2017.3 release, according to the current Unity roadmap.

For setting up a 360° panorama as a Skybox, the following guides are very helpful if you need further pointers: SimplyVR, Tales from the Rift. The instructions below outline all the necessary steps you need to create your own 360° photo viewer in Unity! Continue reading “Showing a 360° Photo in Google Daydream VR based on Unity, Part 2”

Showing a 360° Photo in Google Daydream VR based on Unity, Part 1

Stitched 360° Photo taken with the Samsung Gear 360 in full resolution

Capturing a 360° photo / video and viewing it in VR is one of the most immersive use cases. As the user is part of a captured real world, the virtual experience is the most life-like possible.

In this article, I will show how to capture a 360° photo using the new Samsung Gear 360 camera (2017 version), then load the photo into a Unity project for Google Daydream / Cardboard to view it in VR on an Android phone (in this case, the Google Pixel 2). Continue reading “Showing a 360° Photo in Google Daydream VR based on Unity, Part 1”

How to Combine the Mixed Reality Toolkit, Unity 2017.2 and Visual Studio 2017

MRI scan visualized through HoloLens and Mixed Reality

Update 13. November 2017: The latest source code of the Mixed Reality toolkit now combines both HoloLens and Mixed Reality headsets into a single toolkit that works with one Unity version: 2017.2.0p1 MRTP 4. It’s a special fork of Unity that is optimized for the “Mixed Reality Toolkit Preview”. A later version of Unity will hopefully combine all environments into a single release again. Read more about the environment setup at the GitHub pull request.

Update 19. October 2017: In the meantime, Unity 2017.2 final has been released, and the dev branch of the Mixerd Reality toolkit has been merged back to the master. You should now be fine using the following versions for HoloLens development: Unity 2017.2.0f3+, Mixed Reality Toolkit (master branch), Visual Studio 2017.+4, Windows 10.0.15063.0 SDK.

Original Article: Lately, the tools required for HoloLens / Mixed Reality development have been undergoing profound changes. All three tools involved in building HoloLens apps are being restructured:

  • Unity 2017 unifies Virtual / Augmented Reality APIs, making them flexible enough to target all platforms (e.g., phones with ARKit / ARCore, VR, AR). This also involves new and renamed APIs.
  • HoloToolkit has been renamed to Mixed Reality Toolkit, as Microsoft expands the scope to include the new VR headsets with inside-out tracking going on sale this fall.
  • Visual Studio 2017.3 also introduced some major changes under the hood. This is combined with the C# engine used in Unity slowly being migrated from the old Mono runtime to more recent versions of C#.

With the latest Unity 2017.2.0b11 release, everything should now be coming together. In this blog post, I’m describing how to use the latest versions of the tools for creating and deploying a HoloLens app. Continue reading “How to Combine the Mixed Reality Toolkit, Unity 2017.2 and Visual Studio 2017”

Windows Developer Day, Oct 10th, Vienna: Mixed Reality, Xamarin & more!

Mobile Developer After-Work

The Windows 10 Fall Creators Update brings an impressive array of new features for developers. At the Windows Developer Day, the Microsoft engineering teams will introduce the latest news for developers.

One of the most exciting is Windows Mixed Reality as a feature of the base operating system, bringing Virtual Reality to the masses with reasonably priced VR headsets. Additional topics include Xamarin, .NET Standard 2.0 and the XBox Live Creators Program.

MVPs (Microsoft Most Valuable Professionals) around the globe organize viewing parties. Through the Austrian mobility.builders mobile developer community and together with fellow MVP Helmut Krämer, I have the honor of hosting the local streaming party in Vienna. Continue reading “Windows Developer Day, Oct 10th, Vienna: Mixed Reality, Xamarin & more!”

Basics of Web Technology: HTML5, CSS, JavaScript, Accessibility & WordPress

Basics of Web Technologies

It’s important to know the basics of how the web works. In one of the introductionary lectures at Digital Healthcare, we take a look at HTML5, CSS, JavaScript & more. Understanding basic web technology is required to ensure that web sites have excellent usability as well as accessibility.

This includes how to make sure navigating the page works with screen readers, as well as that the design works well for people with e.g., (color) vision deficiency. Accessibility is especially important for websites and mobile apps of public sector bodies, based on a new directive of the EU that will come into effect in all member states soon. Continue reading “Basics of Web Technology: HTML5, CSS, JavaScript, Accessibility & WordPress”

Cross-Platform Universal Bluetooth Beacon Library Evolves

Universal Bluetooth Beacon Library

Thanks to the help of the new collaborator Chris Tacke, the Universal Bluetooth Beacon Library has evolved to its next level. Based on a modularized architecture, the library is now based on .NET Standard 1.3. Extensions provide the glue to the Bluetooth APIs of various platforms.

Currently, Windows 10 (UWP) and Android (Xamarin) are supported. The core library also works fine on iOS, Mac and Linux. Continue reading “Cross-Platform Universal Bluetooth Beacon Library Evolves”

Porting the NFC / NDEF Library to .NET Standard

NdefDemo App of the NFC Library, now based on .NET Standard

With around 20,000 downloads, the “NFC / NDEF Library for Proximity APIs” is the most popular NFC library for C#. The library has initially been released in 2012 and has since been extended to cover many standardized as well as common NDEF use cases – also thanks to the support of the netidee open innovation platform.

Lately, the underlying platforms have evolved, and now the library has adapted. The main NFC library has now been ported to .NET Standard, which is a common baseline that makes the library compatible to even more different platforms. The new version 4.1.0 is now available under the open source LGPL license on GitHub, as well as on NuGet.

In addition to running on Windows, the library now fully supports all platforms that support .NET Core (like Linux and Mac), as well as Xamarin for Android and iOS. This makes it significantly easier to provide NFC functionality across different platforms – especially now that the iPhone also finally supports the open NDEF standard within NFC through the Apple Core NFC framework. Continue reading “Porting the NFC / NDEF Library to .NET Standard”