With the first Bluetooth 5 devices just released (first and foremost the Samsung Galaxy S8), Bluetooth Beacons like Apple iBeacon and Eddystone by Google gain a lot of potential. Increased range means future beacons will cover a much larger area, without increased power consumption. With the payload increase of beacon advertisement frames from 31 bytes to 255 bytes, the amount of information transmitted by beacons can substantially increase.
In my session about Bluetooth Beacons at the Mobile Developer After-Work #14: Creating Immersive Environments, I provided a short update about the latest news. Unfortunately, recording of the live session at the event failed. Therefore, I’ve re-recorded it – you will miss the live experience, but I did have a little more time for more detailed explanations of some important facts, so it’s well worth watching again even if you have been at the #mdaw14 Event!
In the previous part, we’ve re-compiled Spectator View using the latest HoloToolkit sources. At the time of writing, this resulted in a compile error, as the HoloToolkit for Unity has seen a breaking change since Spectator View was released.
In this part, we’ll first fix the Spectator View code, and then set it up correctly in our own Unity scene. Then, we’ll need to check several other things: how to launch the Sharing Service, adding Internet Connectivity and the Holographic Camera prefab.
Fixing the Spectator View Code
From the two choices at the end of part 4, of course, we go with the adventurous road to fix the code 🙂
Getting Spectator View to run with your own app is obviously the most important step. How to include the required tools – including Sharing – into your app using the latest versions?
More specifically, I’ll add Spectator View to my playground project, which is mainly for placing a hologram in the room (with a few other useful scripts). Currently, this hologram is a nice, life-size skeleton from the Unity asset store.
HoloToolkit-Unity & Holographic Academy
The Holographic Academy course 240 explains how to set up your project to include Shared Holograms. It’s a good idea to work through the tutorial, as it will show you some of the basics of what is needed for Spectator View.
However, as with most of the Academy tutorials, it’s quite old. The HoloToolkit has evolved since then, and the demos often have little resemblance to what your project would look like if you start from scratch.
According to ABI research, 8.2 million Bluetooth Beacons have been shipped. In 2021, the yearly shipments will have increased to an astonishing number of 565 million beacons per year.
The Bluetooth Beacon Interactor for Windows 10 is a free & open source app for scanning your environment for Beacons. It was the first Bluetooth Beacon scanner app available for Windows 10 and works on PCs, smartphones and even the Microsoft HoloLens.
New Features in Version 2.0
The app has just received an update to version 2.0, which adds the two most requested features:
URLs in Eddystone URL frames are now clickable. You can directly open the web site linked to by Physical Web beacons.
Thanks to a community contribution by kobush, the app now also supports beacons compatible to Apple iBeacons / Proximity Beacons.
You can download the source code of the Bluetooth Beacon Interactor and its base Universal Beacon Library at GitHub. If you would like to scan your surroundings for Bluetooth Beacons and analyze their contents, you can directly download the app from the Windows 10 App Store.
HoloLens Spectator View allows capturing high quality photos including the mixed reality contents as seen by other HoloLens users. To test the complete system setup, the next step is running the supplied sample app in Unity.
Based on my previous two blog posts, calibration of the HoloLens in relation to the DSLR camera has been successfully completed. If you need to complete these steps, check out Part 1 and Part 2 first.
The following screenshot shows the contents of the
CalibrationFiles directory, where the app saved both the screenshots of the Canon EOS camera, as well as from the HoloLens. The txt file in the directory is the data file you actually need for your app.
Especially when working with Bluetooth Beacons like iBeacon or Eddystone, you will often want to use embedded hardware to create a low-cost, always running system. This short guide explains how to read Bluetooth Beacon messages using the Arduino 101 / Genuino 101 board.
TapToPlace script of the HoloToolkit is a great and simple way to place and anchor holograms in the real world. However, it can be problematic if the hologram is placed out of reach / view from a previous session in another room and you can not tap the hologram to place it closer to you once again.
A good solution is to create another “reset” gesture that triggers object placement even if you don’t directly tap the hologram. An easy way to achieve this is the tap-and-hold gesture, which is one of the standard events the Input Manager of the HoloToolkit can send.
To handle interaction events that don’t target a particular hologram, the current HoloToolkit uses Fallback Input Handlers. However, this functionality isn’t very well documented anywhere. This guide explains how to write a custom fallback input handler.
The next step to getting the HoloLens Spectator View to run is the calibration. This needs to be done once after mounting the HoloLens to the DSLR camera. The output is a calibration info file that lists how the position, field of view and other parameters differ between the HoloLens and the DSLR.
Step 4: Camera Setup
By default, the Canon EOS 5D Mark III outputs some information on top of the live HDMI video, including a big white focus rectangle. That needs to be deactivated in order to get a full-screen, clear image.
Switch the camera to movie mode, and in the camera settings menu, page “SHOOT5: Movie”, set “HDMI output + LCD” to “Mirroring”.
We’re developing several HoloLens apps at the St. Pölten University of Applied Sciences, for example in the area of Healthcare and Radiology, as well as for Smart Engineering. For better demonstrations, the HoloLens Spectator View is crucial: it allows taking 3rd person photos and videos of people using the HoloLens app. That helps in understanding the overall scene setup and the interactions between users and the otherwise invisible holograms.
Step 1: 3D Printing the HoloLens Mount
Microsoft has released 3D printing files for the HoloLens mount, which allows fastening the HoloLens on a digital camera. They recommend using Aluminium for better stability. However, we do not have such a metal-based 3D printer, and using an external company would cost several hundred Euros. Therefore, we decided to go with the standard plastic 3D printing instead.
Shadows are immensly important for the perception of Augmented Reality scenes. If the holographic 3D object that is placed in the real world has a shadow, it fits better to the world, and users have a better understanding of its placement in the world. This is the result of the study I’ve done some time ago, detailled in the blog posts: Part 1, Part 2, Part 3. How to apply this to Microsoft HoloLens? Negative Shadows are the answer.
HoloLens and Holographic Shadows?
For the HoloLens, rendering shadows is special. The HoloLens displays are light-based – as such, they can add light to a real-world scene. However, they cannot reduce light or darken parts of the real world. If you add a traditional dark shadow to a scene, it simply won’t be visible in the HoloLens. Continue reading “How to add Negative Shadows to a HoloLens Scene”