Category: AR / VR

Categories
AR / VR HoloLens Windows

How to set up HoloLens Spectator View, Part 5 – Sharing your Scene

SpectatorViewManager - Prefabs

In the previous part, we’ve re-compiled Spectator View using the latest HoloToolkit sources. At the time of writing, this resulted in a compile error, as the HoloToolkit for Unity has seen a breaking change since Spectator View was released.

In this part, we’ll first fix the Spectator View code, and then set it up correctly in our own Unity scene. Then, we’ll need to check several other things: how to launch the Sharing Service, adding Internet Connectivity and the Holographic Camera prefab.

Fixing the Spectator View Code

From the two choices at the end of part 4, of course, we go with the adventurous road to fix the code 🙂

Turns out that the AddSurfaceObject()  method from the base class SpatialMappingSource  was changed. Instead of directly creating a Surface Object from a mesh, this process was now split up into two parts. There is one method to create the surface object, and another one to add it.

Continue reading “How to set up HoloLens Spectator View, Part 5 – Sharing your Scene”
Categories
AR / VR HoloLens Windows

How to set up HoloLens Spectator View, Part 4 – Preparing SpecatorView Packages

Getting Spectator View to run with your own app is obviously the most important step. How to include the required tools – including Sharing – into your app using the latest versions?

More specifically, I’ll add Spectator View to my playground project, which is mainly for placing a hologram in the room (with a few other useful scripts). Currently, this hologram is a nice, life-size skeleton from the Unity asset store.

HoloToolkit-Unity & Holographic Academy

The Holographic Academy course 240 explains how to set up your project to include Shared Holograms. It’s a good idea to work through the tutorial, as it will show you some of the basics of what is needed for Spectator View.

However, as with most of the Academy tutorials, it’s quite old. The HoloToolkit has evolved since then, and the demos often have little resemblance to what your project would look like if you start from scratch.

Therefore, for your own project, first clone the HoloToolkit-Unity GitHub repository to your local PC. Open the project in Unity and export the Assets folder to a new package according to the instructions.

Once this is finished, import your newly made, fresh HoloToolkit-Unity package into your own Unity project. Alternatively, you can of course use one of the pre-packaged HoloToolkit-Unity releases, which is slightly older, but is usually a good compromise.

Continue reading “How to set up HoloLens Spectator View, Part 4 – Preparing SpecatorView Packages”
Categories
AR / VR HoloLens Windows

How to set up HoloLens Spectator View, Part 3 – First Real-Life Test

HoloLens Spectator View allows capturing high quality photos including the mixed reality contents as seen by other HoloLens users. To test the complete system setup, the next step is running the supplied sample app in Unity.

Based on my previous two blog posts, calibration of the HoloLens in relation to the DSLR camera has been successfully completed. If you need to complete these steps, check out Part 1 and Part 2 first.

Spectator View Calibration Data

The calibration app saved its calculated data file to your Documents\CalibrationFiles  folder. Copy the CalibrationData.txt  file to the Assets -folder of the Sample project that is part of the HoloLens Companion Kit / Spectator View GitHub repository.

The following screenshot shows the contents of the CalibrationFiles  directory, where the app saved both the screenshots of the Canon EOS camera, as well as from the HoloLens. The txt file in the directory is the data file you actually need for your app.

Continue reading “How to set up HoloLens Spectator View, Part 3 – First Real-Life Test”
Categories
AR / VR HoloLens Windows

HoloLens: Adding Fallback Input Handlers for Positioning Out-of-Sight Objects

The TapToPlace  script of the HoloToolkit is a great and simple way to place and anchor holograms in the real world. However, it can be problematic if the hologram is placed out of reach / view from a previous session in another room and you can not tap the hologram to place it closer to you once again.

A good solution is to create another “reset” gesture that triggers object placement even if you don’t directly tap the hologram. An easy way to achieve this is the tap-and-hold gesture, which is one of the standard events the Input Manager of the HoloToolkit can send.

To handle interaction events that don’t target a particular hologram, the current HoloToolkit uses Fallback Input Handlers. However, this functionality isn’t very well documented anywhere. This guide explains how to write a custom fallback input handler.

Continue reading “HoloLens: Adding Fallback Input Handlers for Positioning Out-of-Sight Objects”
Categories
AR / VR HoloLens Windows

How to set up HoloLens Spectator View, Part 2 – Camera Configuration & Calibration

The next step to getting the HoloLens Spectator View to run is the calibration. This needs to be done once after mounting the HoloLens to the DSLR camera. The output is a calibration info file that lists how the position, field of view and other parameters differ between the HoloLens and the DSLR.

Step 4: Camera Setup

By default, the Canon EOS 5D Mark III outputs some information on top of the live HDMI video, including a big white focus rectangle. That needs to be deactivated in order to get a full-screen, clear image.

Switch the camera to movie mode, and in the camera settings menu, page “SHOOT5: Movie”, set “HDMI output + LCD” to “Mirroring”.

Continue reading “How to set up HoloLens Spectator View, Part 2 – Camera Configuration & Calibration”
Categories
AR / VR HoloLens Windows

How to set up HoloLens Spectator View, Part 1 – 3D Printing, Tools and SDKs

We’re developing several HoloLens apps at the St. Pölten University of Applied Sciences, for example in the area of Healthcare and Radiology, as well as for Smart Engineering. For better demonstrations, the HoloLens Spectator View is crucial: it allows taking 3rd person photos and videos of people using the HoloLens app. That helps in understanding the overall scene setup and the interactions between users and the otherwise invisible holograms.

Step 1: 3D Printing the HoloLens Mount

Microsoft has released 3D printing files for the HoloLens mount, which allows fastening the HoloLens on a digital camera. They recommend using Aluminium for better stability. However, we do not have such a metal-based 3D printer, and using an external company would cost several hundred Euros. Therefore, we decided to go with the standard plastic 3D printing instead.

One issue with the current design is that it’s a bit too large for most 3D printers – the mount is a little bit larger than A4. Luckily, our old 3D printer was able to produce the correct size (3DTouch by Bits from Bytes).

Continue reading “How to set up HoloLens Spectator View, Part 1 – 3D Printing, Tools and SDKs”
Categories
AR / VR HoloLens Windows

How to add Negative Shadows to a HoloLens Scene

Shadows are immensly important for the perception of Augmented Reality scenes. If the holographic 3D object that is placed in the real world has a shadow, it fits better to the world, and users have a better understanding of its placement in the world. This is the result of the study I’ve done some time ago, detailled in the blog posts: Part 1, Part 2, Part 3. How to apply this to Microsoft HoloLens? Negative Shadows are the answer.

HoloLens and Holographic Shadows?

For the HoloLens, rendering shadows is special. The HoloLens displays are light-based – as such, they can add light to a real-world scene. However, they cannot reduce light or darken parts of the real world. If you add a traditional dark shadow to a scene, it simply won’t be visible in the HoloLens.

Continue reading “How to add Negative Shadows to a HoloLens Scene”
Categories
AR / VR HoloLens

Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 3

In the last part of the short blog series, we will take a quick look at the required realism of shadow in Augmented Reality scenarios. Is shadow detail crucial for user acceptance?

Read part 1 for an introduction and how shadow influences the estimation of height, and part 2 for more on estimation of depth and light position.

Required Detail of Shadows

The HoloLens is essentially a head-worn, battery-powered PC. As such, the computing capacity is of course much less compared to most virtual reality systems like the HTC Vive or Oculus Rift where a full-blown PC with a power consumption of around 600 W is rendering the scenes.

Many games use highly simplified shadows that are only a very crude approximation of what a real shadow should look like. In this experiment, the aim was to examine how much simplification of volumentric shadows is accepted by subjects.

The scenario consisted of a high quality torus that floated above the floor. The shadow was generated using different levels of detail; the highest had no visible edges, while the lowest was basically just a pentagon.

Continue reading “Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 3”
Categories
AR / VR HoloLens

Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 2

In the previous blog post of this short series, I described the findings of my study Benefits and Parameters of Shadow in Augmented Reality-Environments on how shadow influences the perception of height (distance to the floor) in Augmented Reality environments. Before we get to a possible solution for Mixed Reality Scenarios with Microsoft HoloLens, let’s take a look at more interesting results of the study.

Estimation of Depth

To correctly percieve a mixed reality scene, the user also has to judge the depth (distance to the camera) of a virtual 3D object in a real scene. If the perception is off, it can destroy the mixed reality effect. In some applications where accuracy is vital, wrong perception of depth can have even more severe implications.

During this experiment, participants of the study were presented with two spheres that varied in size and depth. In random order, every scene was presented twice – once with shadows present, once without shadows.

Continue reading “Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 2”
Categories
AR / VR HoloLens

Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 1

Usually, holograms placed in real environments through Microsoft HoloLens do not have a shadow. This is mainly due to technical reasons of the HoloLens display, as I’ll discuss in a later blog post.

However, for the scene perception and to correctly determine the holographic object position in the real-world 3D space, as well as for the “feeling” that the scene looks real, shadow is of tremendous importance.

Study on Shadow in Augmented Reality

To get exact numbers and to better understand the situation, I’ve written my Bachelor Thesis “Benefits and Parameters of Shadow in Augmented Reality-Environments” in 2004. Even though back then the study was done with marker-based Augmented Reality, the general setting is comparable to scenes created with HoloLens.

Using an experimental approach, 23 participants were tasked with conducting tests and filling out a questionnaire, judging different AR situations. The tool to create the AR scenes was ShadowAReality by Stephan Drab et. al.

Why Shadow is Important for AR

Literature research showed that shadows are of significant importance for realistic perception. They allow a correct estimation of the placement of objects in the virtual direction, as well as the distance to the camera. Furthermore, shadow defines the volume of the object. Overall, shadows contain a big amount of additional information, which obviously lacks if shadows are missing. They play an important role in how realistic a scene looks.

Continue reading “Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 1”