Mixed Reality @ Microsoft Insider Dev Tour

Microsoft Insider Dev Tour

On June 20th, the Microsoft Insider Dev Tour will come to Vienna, Austria. It’s a world-wide event series for developers, organized by Microsoft together with Microsoft Developer MVPs.

You’ll learn about the latest trends for developers – including artificial intelligence, progressive web apps and more. Of course, Mixed Reality is also on the agenda!

As a Microsoft MVP for Windows Development, I’ll take over the Mixed Reality session. You’ll see live demos of getting started with both VR headsets, as well as the Microsoft HoloLens. 150 attendees have signed up – so it’ll certainly be a great event!

In addition, Microsoft has released amazing hands-on labs for everyone to follow up and dive deeper into the content presented at the sessions. The Mixed Reality Lab includes controllers, spatial sound and spatial mapping. It’s a great way to get started with some of the most exciting features of MR. Check it out!

 

How to Combine the Mixed Reality Toolkit, Unity 2017 and Visual Studio 2017

MRI scan visualized through HoloLens and Mixed Reality

Update 20. December 2017: A new release of the Mixed Reality toolkit is now out as an official release. It’s recommended to use this, along with Unity 2017.2.1f1.

Update 13. November 2017: The latest source code of the Mixed Reality toolkit now combines both HoloLens and Mixed Reality headsets into a single toolkit that works with one Unity version: 2017.2.0p1 MRTP 4. It’s a special fork of Unity that is optimized for the “Mixed Reality Toolkit Preview”. A later version of Unity will hopefully combine all environments into a single release again. Read more about the environment setup at the GitHub pull request.

Update 19. October 2017: In the meantime, Unity 2017.2 final has been released, and the dev branch of the Mixerd Reality toolkit has been merged back to the master. You should now be fine using the following versions for HoloLens development: Unity 2017.2.0f3+, Mixed Reality Toolkit (master branch), Visual Studio 2017.+4, Windows 10.0.15063.0 SDK.

Original Article: Lately, the tools required for HoloLens / Mixed Reality development have been undergoing profound changes. All three tools involved in building HoloLens apps are being restructured:

  • Unity 2017 unifies Virtual / Augmented Reality APIs, making them flexible enough to target all platforms (e.g., phones with ARKit / ARCore, VR, AR). This also involves new and renamed APIs.
  • HoloToolkit has been renamed to Mixed Reality Toolkit, as Microsoft expands the scope to include the new VR headsets with inside-out tracking going on sale this fall.
  • Visual Studio 2017.3 also introduced some major changes under the hood. This is combined with the C# engine used in Unity slowly being migrated from the old Mono runtime to more recent versions of C#.

With the latest Unity 2017.2.0b11 release, everything should now be coming together. In this blog post, I’m describing how to use the latest versions of the tools for creating and deploying a HoloLens app. Continue reading “How to Combine the Mixed Reality Toolkit, Unity 2017 and Visual Studio 2017”

More Realistic HoloLens Spectator View Photos

HoloLens Spectator View - Photoshop - Screen - Darkened

When capturing a Mixed Reality scene through HoloLens Spectator View, the resulting photos and videos look amazing. But how to make them more realistic? The right blending mode in Photoshop helps.

Are Spectator View Photos Real?

Of course, the captured Spectator View photo is in many ways different to what you see in HoloLens. First off, it’s only 2D; in the HoloLens, you’d see a real 3D hologram. In this area, the live HoloLens view is much more impressive.

On the other hand, the actual resolution and field of view of the HoloLens is less than a (potentially) 20 Megapixel+ Unity rendering that shows holograms all around you. But the field of view is difficult and somehow unnecessary to simulate as we’re looking on the scene from a 3rd person perspective. Continue reading “More Realistic HoloLens Spectator View Photos”

How to set up HoloLens Spectator View, Bonus – Troubleshooting

HoloLens Delete Space

Did you follow the blog post series on setting up Spectator View for HoloLens? The following issues frequently occured when testing Spectator View. To make your life easier, make sure you check these troubleshooting tips & tricks:

HoloLens Sharing Offset

Between your two HoloLenses, there can be some offset in the placing of the holograms. In one HoloLens, you can for example position your hologram directly on the table; on the second HoloLens, it’s one meter next to the table. Continue reading “How to set up HoloLens Spectator View, Bonus – Troubleshooting”

How to set up HoloLens Spectator View, Part 7 – Mixed Reality

HoloLens Spectator View - Sample Photo

In the final part of the HoloLens Spectator View series, I’ll add the final tweak to my sample project and show you what kind of photos the Compositor actually saves on your PC. Getting to this point has taken a lot more work than expected (+ a lot more blog posts, too!), but I still feel that the journey was worth it! You can finally get great photos and videos of the mixed reality experience on HoloLens.

Script Execution Order

Sending the updates didn’t work right away, so I searched for other changes that the sample does differently from my app that might be responsible. One such case was the Script Execution Order.

In Unity, the order in which scripts are initialized is arbitrary. When debugging, I found an issue that an instance of a class wasn’t available yet when a script wanted to access it. Continue reading “How to set up HoloLens Spectator View, Part 7 – Mixed Reality”

How to set up HoloLens Spectator View, Part 6 – Custom Messages for Sharing

Spectator View - Custom Messages class instance

To make Spectator View work in our own HoloLens project, we actually have to understand how it’s working, what it is doing and how it is related to the HoloLens Sharing Experience. Turns out that there is a lot to do in our app (including transmitting custom messages) to prepare it for the full Spectator View experience!

Synchronizing Objects: Anchor & SceneManager

The Spectator View is based on the Sharing experience of the HoloToolkit, but it contains its own “fork” of the code. Instead of us having to manually decide & code which objects and interactions to send via network messages (as in the normal Sharing use case), the fork of the Spectator View transmits some data by default to our DSLR-mounted HoloLens.

However, it’s important to understand that if you use Spectator View, you also need to integrate most of the Sharing code. Essentially, a lot of code is duplicated between the two frameworks, with only slight differences in order to give the DSLR-mounted HoloLens a special role, plus the PC running Unity also needs to connect in order to correctly render the scene. Continue reading “How to set up HoloLens Spectator View, Part 6 – Custom Messages for Sharing”

How to set up HoloLens Spectator View, Part 5 – Sharing your Scene

SpectatorViewManager - Prefabs

In the previous part, we’ve re-compiled Spectator View using the latest HoloToolkit sources. At the time of writing, this resulted in a compile error, as the HoloToolkit for Unity has seen a breaking change since Spectator View was released.

In this part, we’ll first fix the Spectator View code, and then set it up correctly in our own Unity scene. Then, we’ll need to check several other things: how to launch the Sharing Service, adding Internet Connectivity and the Holographic Camera prefab.

Fixing the Spectator View Code

From the two choices at the end of part 4, of course, we go with the adventurous road to fix the code 🙂

Turns out that the AddSurfaceObject()  method from the base class SpatialMappingSource  was changed. Instead of directly creating a Surface Object from a mesh, this process was now split up into two parts. There is one method to create the surface object, and another one to add it. Continue reading “How to set up HoloLens Spectator View, Part 5 – Sharing your Scene”

How to set up HoloLens Spectator View, Part 4 – Preparing SpecatorView Packages

Getting Spectator View to run with your own app is obviously the most important step. How to include the required tools – including Sharing – into your app using the latest versions?

More specifically, I’ll add Spectator View to my playground project, which is mainly for placing a hologram in the room (with a few other useful scripts). Currently, this hologram is a nice, life-size skeleton from the Unity asset store.

HoloToolkit-Unity & Holographic Academy

The Holographic Academy course 240 explains how to set up your project to include Shared Holograms. It’s a good idea to work through the tutorial, as it will show you some of the basics of what is needed for Spectator View.

However, as with most of the Academy tutorials, it’s quite old. The HoloToolkit has evolved since then, and the demos often have little resemblance to what your project would look like if you start from scratch.

Therefore, for your own project, first clone the HoloToolkit-Unity GitHub repository to your local PC. Open the project in Unity and export the Assets folder to a new package according to the instructions.

Once this is finished, import your newly made, fresh HoloToolkit-Unity package into your own Unity project. Alternatively, you can of course use one of the pre-packaged HoloToolkit-Unity releases, which is slightly older, but is usually a good compromise. Continue reading “How to set up HoloLens Spectator View, Part 4 – Preparing SpecatorView Packages”

How to set up HoloLens Spectator View, Part 3 – First Real-Life Test

HoloLens Spectator View allows capturing high quality photos including the mixed reality contents as seen by other HoloLens users. To test the complete system setup, the next step is running the supplied sample app in Unity.

Based on my previous two blog posts, calibration of the HoloLens in relation to the DSLR camera has been successfully completed. If you need to complete these steps, check out Part 1 and Part 2 first.

Spectator View Calibration Data

The calibration app saved its calculated data file to your Documents\CalibrationFiles  folder. Copy the CalibrationData.txt  file to the Assets -folder of the Sample project that is part of the HoloLens Companion Kit / Spectator View GitHub repository.

The following screenshot shows the contents of the CalibrationFiles  directory, where the app saved both the screenshots of the Canon EOS camera, as well as from the HoloLens. The txt file in the directory is the data file you actually need for your app.

Continue reading “How to set up HoloLens Spectator View, Part 3 – First Real-Life Test”

HoloLens: Adding Fallback Input Handlers for Positioning Out-of-Sight Objects

The TapToPlace  script of the HoloToolkit is a great and simple way to place and anchor holograms in the real world. However, it can be problematic if the hologram is placed out of reach / view from a previous session in another room and you can not tap the hologram to place it closer to you once again.

A good solution is to create another “reset” gesture that triggers object placement even if you don’t directly tap the hologram. An easy way to achieve this is the tap-and-hold gesture, which is one of the standard events the Input Manager of the HoloToolkit can send.

To handle interaction events that don’t target a particular hologram, the current HoloToolkit uses Fallback Input Handlers. However, this functionality isn’t very well documented anywhere. This guide explains how to write a custom fallback input handler.

Continue reading “HoloLens: Adding Fallback Input Handlers for Positioning Out-of-Sight Objects”