HoloLens: Adding Fallback Input Handlers for Positioning Out-of-Sight Objects

The TapToPlace  script of the HoloToolkit is a great and simple way to place and anchor holograms in the real world. However, it can be problematic if the hologram is placed out of reach / view from a previous session in another room and you can not tap the hologram to place it closer to you once again.

A good solution is to create another “reset” gesture that triggers object placement even if you don’t directly tap the hologram. An easy way to achieve this is the tap-and-hold gesture, which is one of the standard events the Input Manager of the HoloToolkit can send.

To handle interaction events that don’t target a particular hologram, the current HoloToolkit uses Fallback Input Handlers. However, this functionality isn’t very well documented anywhere. This guide explains how to write a custom fallback input handler.

Continue reading “HoloLens: Adding Fallback Input Handlers for Positioning Out-of-Sight Objects”

How to set up HoloLens Spectator View, Part 2 – Camera Configuration & Calibration

The next step to getting the HoloLens Spectator View to run is the calibration. This needs to be done once after mounting the HoloLens to the DSLR camera. The output is a calibration info file that lists how the position, field of view and other parameters differ between the HoloLens and the DSLR.

Step 4: Camera Setup

By default, the Canon EOS 5D Mark III outputs some information on top of the live HDMI video, including a big white focus rectangle. That needs to be deactivated in order to get a full-screen, clear image.

Switch the camera to movie mode, and in the camera settings menu, page “SHOOT5: Movie”, set “HDMI output + LCD” to “Mirroring”.

Continue reading “How to set up HoloLens Spectator View, Part 2 – Camera Configuration & Calibration”

How to set up HoloLens Spectator View, Part 1 – 3D Printing, Tools and SDKs

We’re developing several HoloLens apps at the St. Pölten University of Applied Sciences, for example in the area of Healthcare and Radiology, as well as for Smart Engineering. For better demonstrations, the HoloLens Spectator View is crucial: it allows taking 3rd person photos and videos of people using the HoloLens app. That helps in understanding the overall scene setup and the interactions between users and the otherwise invisible holograms.

Step 1: 3D Printing the HoloLens Mount

Microsoft has released 3D printing files for the HoloLens mount, which allows fastening the HoloLens on a digital camera. They recommend using Aluminium for better stability. However, we do not have such a metal-based 3D printer, and using an external company would cost several hundred Euros. Therefore, we decided to go with the standard plastic 3D printing instead.

One issue with the current design is that it’s a bit too large for most 3D printers – the mount is a little bit larger than A4. Luckily, our old 3D printer was able to produce the correct size (3DTouch by Bits from Bytes). Continue reading “How to set up HoloLens Spectator View, Part 1 – 3D Printing, Tools and SDKs”

How to add Negative Shadows to a HoloLens Scene

Shadows are immensly important for the perception of Augmented Reality scenes. If the holographic 3D object that is placed in the real world has a shadow, it fits better to the world, and users have a better understanding of its placement in the world. This is the result of the study I’ve done some time ago, detailled in the blog posts: Part 1, Part 2, Part 3. How to apply this to Microsoft HoloLens? Negative Shadows are the answer.

HoloLens and Holographic Shadows?

For the HoloLens, rendering shadows is special. The HoloLens displays are light-based – as such, they can add light to a real-world scene. However, they cannot reduce light or darken parts of the real world. If you add a traditional dark shadow to a scene, it simply won’t be visible in the HoloLens. Continue reading “How to add Negative Shadows to a HoloLens Scene”

Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 3

In the last part of the short blog series, we will take a quick look at the required realism of shadow in Augmented Reality scenarios. Is shadow detail crucial for user acceptance?

Read part 1 for an introduction and how shadow influences the estimation of height, and part 2 for more on estimation of depth and light position.

Required Detail of Shadows

The HoloLens is essentially a head-worn, battery-powered PC. As such, the computing capacity is of course much less compared to most virtual reality systems like the HTC Vive or Oculus Rift where a full-blown PC with a power consumption of around 600 W is rendering the scenes.

Many games use highly simplified shadows that are only a very crude approximation of what a real shadow should look like. In this experiment, the aim was to examine how much simplification of volumentric shadows is accepted by subjects.

The scenario consisted of a high quality torus that floated above the floor. The shadow was generated using different levels of detail; the highest had no visible edges, while the lowest was basically just a pentagon. Continue reading “Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 3”

Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 2

In the previous blog post of this short series, I described the findings of my study Benefits and Parameters of Shadow in Augmented Reality-Environments on how shadow influences the perception of height (distance to the floor) in Augmented Reality environments. Before we get to a possible solution for Mixed Reality Scenarios with Microsoft HoloLens, let’s take a look at more interesting results of the study.

Estimation of Depth

To correctly percieve a mixed reality scene, the user also has to judge the depth (distance to the camera) of a virtual 3D object in a real scene. If the perception is off, it can destroy the mixed reality effect. In some applications where accuracy is vital, wrong perception of depth can have even more severe implications.

During this experiment, participants of the study were presented with two spheres that varied in size and depth. In random order, every scene was presented twice – once with shadows present, once without shadows. Continue reading “Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 2”

20 – 30% Better JPEG compression from Google? My Test Results.

The blog post from Google sounds intriguing – 20% to 30% better JPEG image compression at the same visual quality through the Guetzli encoder. That has potential for a huge speed increase of websites.

While there are of course a lot of other better image formats around than JPEG (e.g., WebP or JPEG 2000), time has shown that it’s more or less impossible for them to gain any traction. It’s a pity, but on the other hand ensures that pretty much every device available on the market right now can load and show all websites.

Testing Guetzli!

To check the new Guetzli JPEG encoder out myself, I downloaded the binary executable of v1.0 and put it through some very quick tests. Continue reading “20 – 30% Better JPEG compression from Google? My Test Results.”

Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 1

Usually, holograms placed in real environments through Microsoft HoloLens do not have a shadow. This is mainly due to technical reasons of the HoloLens display, as I’ll discuss in a later blog post.

However, for the scene perception and to correctly determine the holographic object position in the real-world 3D space, as well as for the “feeling” that the scene looks real, shadow is of tremendous importance.

Study on Shadow in Augmented Reality

To get exact numbers and to better understand the situation, I’ve written my Bachelor Thesis “Benefits and Parameters of Shadow in Augmented Reality-Environments” in 2004. Even though back then the study was done with marker-based Augmented Reality, the general setting is comparable to scenes created with HoloLens.

Using an experimental approach, 23 participants were tasked with conducting tests and filling out a questionnaire, judging different AR situations. The tool to create the AR scenes was ShadowAReality by Stephan Drab et. al.

Why Shadow is Important for AR

Literature research showed that shadows are of significant importance for realistic perception. They allow a correct estimation of the placement of objects in the virtual direction, as well as the distance to the camera. Furthermore, shadow defines the volume of the object. Overall, shadows contain a big amount of additional information, which obviously lacks if shadows are missing. They play an important role in how realistic a scene looks. Continue reading “Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 1”

How to create Photos of HoloLens in Use

Screenshots taken with HoloLens are often difficult to comprehend, as they only show the scene as seen by the user, but not the user itself. How is he interacting with the scene, where is he standing in relation to the view?

Therefore, photos of HoloLens apps usually show a 3rd person view. Getting a real live transmission of the 3D objects seen by a HoloLens user works with the new Spectator View by Microsoft. However, setting that up is tricky: you need 2 HoloLenses, some special hardware and in the ideal case a 3D printed mount.

A simpler approach for a still screen is to take a photo with a good quality camera, and then to render the scene with matching perspective.

Unity itself is not directly suited to rendering a scene to an image, and it also lacks some of the advanced tools to align the 3D scene with a photo.

Here, I’ll describe an approach to use 3ds Max to simulate the view of a HoloLens user in 3rd person view. I used this for our PalKranHoloPlan AR8000 project, which we created during the Palfinger Hackathon. Continue reading “How to create Photos of HoloLens in Use”