How to set up HoloLens Spectator View, Part 6 – Custom Messages for Sharing

To make Spectator View work in our own HoloLens project, we actually have to understand how it’s working, what it is doing and how it is related to the HoloLens Sharing Experience. Turns out that there is a lot to do in our app (including transmitting custom messages) to prepare it for the full Spectator View experience!

Synchronizing Objects: Anchor & SceneManager

The Spectator View is based on the Sharing experience of the HoloToolkit, but it contains its own “fork” of the code. Instead of us having to manually decide & code which objects and interactions to send via network messages (as in the normal Sharing use case), the fork of the Spectator View transmits some data by default to our DSLR-mounted HoloLens.

However, it’s important to understand that if you use Spectator View, you also need to integrate most of the Sharing code. Essentially, a lot of code is duplicated between the two frameworks, with only slight differences in order to give the DSLR-mounted HoloLens a special role, plus the PC running Unity also needs to connect in order to correctly render the scene.

Object Hierarchy & Scripts

According to the instructions of the SpectatorView sample, you can include and exclude some objects to synchronize (or to hide) in the recording through Anchor\SceneManager. Included objects should be children of the SceneManager.

What is this Anchor\SceneManager and where is it coming from? In the previous step of the blog series, we have assigned the Anchor prefab to the SpectatorViewManager . Turns out this was a dead end.

By looking at the ShardHolograms example, we can see that the  HologramCollection  in the hierarchy panel has all the SV_* scripts as well, plus it’s referenced as  Anchor  object in the SpectatorViewManager . The HologramCollection  actually has a lot more scripts, but most of these don’t concern us as they’re specific to the gameplay of the example.

As I don’t want to change the hierarchy of my entire project, I just add the four scripts to the root GameObject where I’ve placed all the models (in my case called “Objects”):

  • SV_ImportExportAnchorManager
  • SV_CustomMessages
  • SV_RemotePlayerManager
  • SceneManager

Then, I can drag & drop my  Objects  root node with all the scripts applied to the  Anchor  parameter of the SpectatorViewManager . The referenced Objects  game object now has all the required scripts, so that it can serve as Anchor  for the SpectatorViewManager .

SpectatorViewManager: Setting the Anchor

Synchronizing your Scene

Even though there might be some code in Spectator View to automatically synchronize parts of the scene, that didn’t really work (for me). I didn’t dig too deeply into the reason for that; might be that the default scripts only synchronize the scene changes to the Unity Compositor, but not to all involved HoloLens devices?

In any case, a good HoloLens app that is shared between multiple HoloLenses should implement the full Sharing capabilities.

Thus, I went ahead to add custom messages manually to the app, in order to transmit my own events between all involved HoloLens apps so that all HoloLenses connected to the experience (including the PC with Unity that does the recording) are always synchronized.

Sharing Placement Events

To share location updates and other custom scene changes, you need to create, send, receive and handle own custom messages over the network. The SV_CustomMessages  script contains these messages for the Spectator View. Of course, you could go ahead and adapt this to your needs. However, if there is an update to Spectator View, it’s problematic if you change files that belong to original Spectator View Code.

Additionally, also the SharedHolograms sample contains two different implementations of the network messages: the standard Spectator View variant, plus the project-related CustomMessages script. Therefore, it’s probably best to follow this example.

Also, the example is transmitting the placement of an object (in that case the “game base”) and synchronizes that location to all connected clients. To get started in the quickest possible way, I directly re-used the sample code and did not modify it (yet).

Of course, the CustomMessages class also contains a lot of other code that we don’t need (e.g., for shooting projectiles), but we can clean that up at a later stage when everything works. The main method for us of the CustomMessages  class is the SendStageTransform()  method:

I copied the whole CustomMessages  script from the example into the Assets/Scripts  folder of my own project, and added it to my Objects  node in the scene where all the Spectator View scripts are already present:

Spectator View - CustomMessages class instance

That’s a great way to start, as you don’t have to define your own message types for now and the class also handles receiving this message.

Sending Custom Placement on Tap

Now that we have the SendStageTransform  method in our project, we obviously need to actually make use of it. Right now, my project features the functionality to simply place a hologram into the scene. I’ve used the standard TapToPlace script from the HoloToolkit for this functionality.

If we now need to send the location to other HoloLenses whenever a user places the object, we need to modify that script. Again, instead of directly changing code in HoloToolkit’s TapToPlace  script, create your own copy of it in your Scripts folder. I called my script TapToPlaceShared :

I’ve made the following changes to the original TapToPlace script:

  • Initialization: in the Start()  method, the script registers itself as message handler for incoming StageTransform  messages from the CustomMessages  instance. We want to be informed whenever our HoloLens receives a position update from the server.
  • Receiving updates: we wired up the OnStageTransform()  method as callback handler for incoming messages. In that method, our class reads the position and rotation from the network message and applies it to the local transform.
  • Sending updates: in OnSelect() , the code now also shares the new position update with other HoloLenses. Depending on the configuration, the script would either send the position of the game object it’s attached to, or the parent. That follows the configuration options of the original TapToPlace .
  • Anchors: now that’s a tricky thing. World Anchors in Unity connect a game object to a physical position in the world. You can not move the position of an object when it still has a world anchor attached. This valuable piece of info is hidden in the documentation.
    The default TapToPlace  script removes the world anchor, then updates the position when the user taps, and finally adds a new anchor. That works, as there is some time between removing the anchor until the user has placed the object at the new position. In our case, we’d move the object in a single frame. Well, removing the world anchor is asynchronous, as such that wouldn’t work. Therefore, you would need to destroy the anchor immediately as seen here in the documentation.
    In my implementation, I’ve removed the use of world anchors for now – these lines are currently commented out, as the object is placed in the real world nevertheless and it works for the Spectator View scenario. I’ll add the anchors back in later. But to check that everything works, I didn’t want anchors to interfere with any custom placements.

What’s next?

We’re almost there! The next blog post will add the final tweak to our project and show you what you get out of the Spectator View!

HoloLens Spectator View blog post series

This post is part of a short series that guides you through integrating HoloLens Spectator View into your own Mixed Reality app: