How to set up HoloLens Spectator View, Part 4 – Preparing SpecatorView Packages

Getting Spectator View to run with your own app is obviously the most important step. How to include the required tools – including Sharing – into your app using the latest versions?

More specifically, I’ll add Spectator View to my playground project, which is mainly for placing a hologram in the room (with a few other useful scripts). Currently, this hologram is a nice, life-size skeleton from the Unity asset store.

HoloToolkit-Unity & Holographic Academy

The Holographic Academy course 240 explains how to set up your project to include Shared Holograms. It’s a good idea to work through the tutorial, as it will show you some of the basics of what is needed for Spectator View.

However, as with most of the Academy tutorials, it’s quite old. The HoloToolkit has evolved since then, and the demos often have little resemblance to what your project would look like if you start from scratch.

Therefore, for your own project, first clone the HoloToolkit-Unity GitHub repository to your local PC. Open the project in Unity and export the Assets folder to a new package according to the instructions.

Once this is finished, import your newly made, fresh HoloToolkit-Unity package into your own Unity project. Alternatively, you can of course use one of the pre-packaged HoloToolkit-Unity releases, which is slightly older, but is usually a good compromise. Continue reading “How to set up HoloLens Spectator View, Part 4 – Preparing SpecatorView Packages”

Bluetooth Beacon Interactor 2 for Windows 10

According to ABI research, 8.2 million Bluetooth Beacons have been shipped. In 2021, the yearly shipments will have increased to an astonishing number of 565 million beacons per year.

The Bluetooth Beacon Interactor for Windows 10 is a free & open source app for scanning your environment for Beacons. It was the first Bluetooth Beacon scanner app available for Windows 10 and works on PCs, smartphones and even the Microsoft HoloLens.

New Features in Version 2.0

The app has just received an update to version 2.0, which adds the two most requested features:

  • URLs in Eddystone URL frames are now clickable. You can directly open the web site linked to by Physical Web beacons.
  • Thanks to a community contribution by kobush, the app now also supports beacons compatible to Apple iBeacons / Proximity Beacons.

Download

You can download the source code of the Bluetooth Beacon Interactor and its base Universal Beacon Library at GitHub. If you would like to scan your surroundings for Bluetooth Beacons and analyze their contents, you can directly download the app from the Windows 10 App Store.

How to set up HoloLens Spectator View, Part 3 – First Real-Life Test

HoloLens Spectator View allows capturing high quality photos including the mixed reality contents as seen by other HoloLens users. To test the complete system setup, the next step is running the supplied sample app in Unity.

Based on my previous two blog posts, calibration of the HoloLens in relation to the DSLR camera has been successfully completed. If you need to complete these steps, check out Part 1 and Part 2 first.

Spectator View Calibration Data

The calibration app saved its calculated data file to your Documents\CalibrationFiles  folder. Copy the CalibrationData.txt  file to the Assets -folder of the Sample project that is part of the HoloLens Companion Kit / Spectator View GitHub repository.

The following screenshot shows the contents of the CalibrationFiles  directory, where the app saved both the screenshots of the Canon EOS camera, as well as from the HoloLens. The txt file in the directory is the data file you actually need for your app.

Continue reading “How to set up HoloLens Spectator View, Part 3 – First Real-Life Test”

How to Parse Bluetooth Beacons (iBeacon and Eddystone) with Arduino 101

Especially when working with Bluetooth Beacons like iBeacon or Eddystone, you will often want to use embedded hardware to create a low-cost, always running system. This short guide explains how to read Bluetooth Beacon messages using the Arduino 101 / Genuino 101 board.

The Arduino / Genuino 101 already includes a Bluetooth chipset and is based on the Intel Curie platform. You can use the same Arduino IDE as for all the other variants of the Arduino Board. Continue reading “How to Parse Bluetooth Beacons (iBeacon and Eddystone) with Arduino 101”

HoloLens: Adding Fallback Input Handlers for Positioning Out-of-Sight Objects

The TapToPlace  script of the HoloToolkit is a great and simple way to place and anchor holograms in the real world. However, it can be problematic if the hologram is placed out of reach / view from a previous session in another room and you can not tap the hologram to place it closer to you once again.

A good solution is to create another “reset” gesture that triggers object placement even if you don’t directly tap the hologram. An easy way to achieve this is the tap-and-hold gesture, which is one of the standard events the Input Manager of the HoloToolkit can send.

To handle interaction events that don’t target a particular hologram, the current HoloToolkit uses Fallback Input Handlers. However, this functionality isn’t very well documented anywhere. This guide explains how to write a custom fallback input handler.

Continue reading “HoloLens: Adding Fallback Input Handlers for Positioning Out-of-Sight Objects”

How to set up HoloLens Spectator View, Part 2 – Camera Configuration & Calibration

The next step to getting the HoloLens Spectator View to run is the calibration. This needs to be done once after mounting the HoloLens to the DSLR camera. The output is a calibration info file that lists how the position, field of view and other parameters differ between the HoloLens and the DSLR.

Step 4: Camera Setup

By default, the Canon EOS 5D Mark III outputs some information on top of the live HDMI video, including a big white focus rectangle. That needs to be deactivated in order to get a full-screen, clear image.

Switch the camera to movie mode, and in the camera settings menu, page “SHOOT5: Movie”, set “HDMI output + LCD” to “Mirroring”.

Continue reading “How to set up HoloLens Spectator View, Part 2 – Camera Configuration & Calibration”

How to set up HoloLens Spectator View, Part 1 – 3D Printing, Tools and SDKs

We’re developing several HoloLens apps at the St. Pölten University of Applied Sciences, for example in the area of Healthcare and Radiology, as well as for Smart Engineering. For better demonstrations, the HoloLens Spectator View is crucial: it allows taking 3rd person photos and videos of people using the HoloLens app. That helps in understanding the overall scene setup and the interactions between users and the otherwise invisible holograms.

Step 1: 3D Printing the HoloLens Mount

Microsoft has released 3D printing files for the HoloLens mount, which allows fastening the HoloLens on a digital camera. They recommend using Aluminium for better stability. However, we do not have such a metal-based 3D printer, and using an external company would cost several hundred Euros. Therefore, we decided to go with the standard plastic 3D printing instead.

One issue with the current design is that it’s a bit too large for most 3D printers – the mount is a little bit larger than A4. Luckily, our old 3D printer was able to produce the correct size (3DTouch by Bits from Bytes). Continue reading “How to set up HoloLens Spectator View, Part 1 – 3D Printing, Tools and SDKs”

How to add Negative Shadows to a HoloLens Scene

Shadows are immensly important for the perception of Augmented Reality scenes. If the holographic 3D object that is placed in the real world has a shadow, it fits better to the world, and users have a better understanding of its placement in the world. This is the result of the study I’ve done some time ago, detailled in the blog posts: Part 1, Part 2, Part 3. How to apply this to Microsoft HoloLens? Negative Shadows are the answer.

HoloLens and Holographic Shadows?

For the HoloLens, rendering shadows is special. The HoloLens displays are light-based – as such, they can add light to a real-world scene. However, they cannot reduce light or darken parts of the real world. If you add a traditional dark shadow to a scene, it simply won’t be visible in the HoloLens. Continue reading “How to add Negative Shadows to a HoloLens Scene”

Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 3

In the last part of the short blog series, we will take a quick look at the required realism of shadow in Augmented Reality scenarios. Is shadow detail crucial for user acceptance?

Read part 1 for an introduction and how shadow influences the estimation of height, and part 2 for more on estimation of depth and light position.

Required Detail of Shadows

The HoloLens is essentially a head-worn, battery-powered PC. As such, the computing capacity is of course much less compared to most virtual reality systems like the HTC Vive or Oculus Rift where a full-blown PC with a power consumption of around 600 W is rendering the scenes.

Many games use highly simplified shadows that are only a very crude approximation of what a real shadow should look like. In this experiment, the aim was to examine how much simplification of volumentric shadows is accepted by subjects.

The scenario consisted of a high quality torus that floated above the floor. The shadow was generated using different levels of detail; the highest had no visible edges, while the lowest was basically just a pentagon. Continue reading “Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 3”

Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 2

In the previous blog post of this short series, I described the findings of my study Benefits and Parameters of Shadow in Augmented Reality-Environments on how shadow influences the perception of height (distance to the floor) in Augmented Reality environments. Before we get to a possible solution for Mixed Reality Scenarios with Microsoft HoloLens, let’s take a look at more interesting results of the study.

Estimation of Depth

To correctly percieve a mixed reality scene, the user also has to judge the depth (distance to the camera) of a virtual 3D object in a real scene. If the perception is off, it can destroy the mixed reality effect. In some applications where accuracy is vital, wrong perception of depth can have even more severe implications.

During this experiment, participants of the study were presented with two spheres that varied in size and depth. In random order, every scene was presented twice – once with shadows present, once without shadows. Continue reading “Benefits and Parameters of Shadow in Augmented Reality-Environments, Part 2”