To make Spectator View work in our own HoloLens project, we actually have to understand how it’s working, what it is doing and how it is related to the HoloLens Sharing Experience. Turns out that there is a lot to do in our app (including transmitting custom messages) to prepare it for the full Spectator View experience!
Synchronizing Objects: Anchor & SceneManager
The Spectator View is based on the Sharing experience of the HoloToolkit, but it contains its own “fork” of the code. Instead of us having to manually decide & code which objects and interactions to send via network messages (as in the normal Sharing use case), the fork of the Spectator View transmits some data by default to our DSLR-mounted HoloLens.
However, it’s important to understand that if you use Spectator View, you also need to integrate most of the Sharing code. Essentially, a lot of code is duplicated between the two frameworks, with only slight differences in order to give the DSLR-mounted HoloLens a special role, plus the PC running Unity also needs to connect in order to correctly render the scene.
Object Hierarchy & Scripts
According to the instructions of the SpectatorView sample, you can include and exclude some objects to synchronize (or to hide) in the recording through Anchor\SceneManager. Included objects should be children of the SceneManager.
What is this Anchor\SceneManager and where is it coming from? In the previous step of the blog series, we have assigned the Anchor prefab to the SpectatorViewManager . Turns out this was a dead end.
By looking at the ShardHolograms example, we can see that the HologramCollection in the hierarchy panel has all the SV_* scripts as well, plus it’s referenced as Anchor object in the SpectatorViewManager . The HologramCollection actually has a lot more scripts, but most of these don’t concern us as they’re specific to the gameplay of the example.
As I don’t want to change the hierarchy of my entire project, I just add the four scripts to the root GameObject where I’ve placed all the models (in my case called “Objects”):
- SV_ImportExportAnchorManager
- SV_CustomMessages
- SV_RemotePlayerManager
- SceneManager
Then, I can drag & drop my Objects root node with all the scripts applied to the Anchor parameter of the SpectatorViewManager . The referenced Objects game object now has all the required scripts, so that it can serve as Anchor for the SpectatorViewManager .
Synchronizing your Scene
Even though there might be some code in Spectator View to automatically synchronize parts of the scene, that didn’t really work (for me). I didn’t dig too deeply into the reason for that; might be that the default scripts only synchronize the scene changes to the Unity Compositor, but not to all involved HoloLens devices?
In any case, a good HoloLens app that is shared between multiple HoloLenses should implement the full Sharing capabilities.
Thus, I went ahead to add custom messages manually to the app, in order to transmit my own events between all involved HoloLens apps so that all HoloLenses connected to the experience (including the PC with Unity that does the recording) are always synchronized.
Sharing Placement Events
To share location updates and other custom scene changes, you need to create, send, receive and handle own custom messages over the network. The SV_CustomMessages script contains these messages for the Spectator View. Of course, you could go ahead and adapt this to your needs. However, if there is an update to Spectator View, it’s problematic if you change files that belong to original Spectator View Code.
Additionally, also the SharedHolograms sample contains two different implementations of the network messages: the standard Spectator View variant, plus the project-related CustomMessages script. Therefore, it’s probably best to follow this example.
Also, the example is transmitting the placement of an object (in that case the “game base”) and synchronizes that location to all connected clients. To get started in the quickest possible way, I directly re-used the sample code and did not modify it (yet).
Of course, the CustomMessages class also contains a lot of other code that we don’t need (e.g., for shooting projectiles), but we can clean that up at a later stage when everything works. The main method for us of the CustomMessages class is the SendStageTransform() method:
public void SendStageTransform(Vector3 position, Quaternion rotation)
{
// If we are connected to a session, broadcast our head info
if (this.serverConnection != null && this.serverConnection.IsConnected())
{
// Create an outgoing network message to contain all the info we want to send
NetworkOutMessage msg = CreateMessage((byte)TestMessageID.StageTransform);
AppendTransform(msg, position, rotation);
Debug.Log("CustomMessages: SendStageTransform - Broadcasting");
// Send the message as a broadcast, which will cause the server to forward it to all other users in the session.
this.serverConnection.Broadcast(
msg,
MessagePriority.Immediate,
MessageReliability.ReliableOrdered,
MessageChannel.Avatar);
}
}
I copied the whole CustomMessages script from the example into the Assets/Scripts folder of my own project, and added it to my Objects node in the scene where all the Spectator View scripts are already present:
That’s a great way to start, as you don’t have to define your own message types for now and the class also handles receiving this message.
Sending Custom Placement on Tap
Now that we have the SendStageTransform method in our project, we obviously need to actually make use of it. Right now, my project features the functionality to simply place a hologram into the scene. I’ve used the standard TapToPlace script from the HoloToolkit for this functionality.
If we now need to send the location to other HoloLenses whenever a user places the object, we need to modify that script. Again, instead of directly changing code in HoloToolkit’s TapToPlace script, create your own copy of it in your Scripts folder. I called my script TapToPlaceShared :
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See LICENSE in the project root for license information.
using HoloToolkit.Sharing;
using HoloToolkit.Unity.InputModule;
using SpectatorView;
using UnityEngine;
namespace HoloToolkit.Unity.SpatialMapping
{
/// <summary>
/// Modified version of TapToPlace that also sends the updated position
/// of the object through an instance of a CustomMessages class from
/// the Sharing framework.
///
/// The TapToPlace class is a basic way to enable users to move objects
/// and place them on real world surfaces.
/// Put this script on the object you want to be able to move.
/// Users will be able to tap objects, gaze elsewhere, and perform the
/// tap gesture again to place.
/// This script is used in conjunction with GazeManager, GestureManager,
/// and SpatialMappingManager.
/// TapToPlace also adds a WorldAnchor component to enable persistence.
/// </summary>
public class TapToPlaceShared : MonoBehaviour, IInputClickHandler
{
/// <summary>
/// Tracks if we have been sent a transform for the model.
/// The model is rendered relative to the actual anchor.
/// </summary>
public bool GotTransform { get; private set; }
[Tooltip("Supply a friendly name for the anchor as the key name for the WorldAnchorStore.")]
public string SavedAnchorFriendlyName = "SavedAnchorFriendlyName";
[Tooltip("Place parent on tap instead of current game object.")]
public bool PlaceParentOnTap;
[Tooltip("Specify the parent game object to be moved on tap, if the immediate parent is not desired.")]
public GameObject ParentGameObjectToPlace;
/// <summary>
/// Keeps track of if the user is moving the object or not.
/// Setting this to true will enable the user to move and place the object in the scene.
/// Useful when you want to place an object immediately.
/// </summary>
[Tooltip("Setting this to true will enable the user to move and place the object in the scene without needing to tap on the object. Useful when you want to place an object immediately.")]
public bool IsBeingPlaced;
/// <summary>
/// Manages persisted anchors.
/// </summary>
protected WorldAnchorManager anchorManager;
/// <summary>
/// Controls spatial mapping. In this script we access spatialMappingManager
/// to control rendering and to access the physics layer mask.
/// </summary>
protected SpatialMappingManager spatialMappingManager;
protected virtual void Start()
{
// Make sure we have all the components in the scene we need.
anchorManager = WorldAnchorManager.Instance;
if (anchorManager == null)
{
Debug.LogError("This script expects that you have a WorldAnchorManager component in your scene.");
}
spatialMappingManager = SpatialMappingManager.Instance;
if (spatialMappingManager == null)
{
Debug.LogError("This script expects that you have a SpatialMappingManager component in your scene.");
}
//if (anchorManager != null && spatialMappingManager != null)
//{
// anchorManager.AttachAnchor(gameObject, SavedAnchorFriendlyName);
//}
//else
//{
// // If we don't have what we need to proceed, we may as well remove ourselves.
// Destroy(this);
//}
if (PlaceParentOnTap)
{
if (ParentGameObjectToPlace != null && !gameObject.transform.IsChildOf(ParentGameObjectToPlace.transform))
{
Debug.LogError("The specified parent object is not a parent of this object.");
}
DetermineParent();
}
if (CustomMessages.Instance != null)
{
// We care about getting updates for the model transform.
CustomMessages.Instance.MessageHandlers[CustomMessages.TestMessageID.StageTransform] =
this.OnStageTransfrom;
}
else
{
Debug.LogError("Didn't find instance of CustomMessages in TapToPlaceShared Start()");
}
}
protected virtual void Update()
{
// If the user is in placing mode,
// update the placement to match the user's gaze.
if (IsBeingPlaced)
{
// Do a raycast into the world that will only hit the Spatial Mapping mesh.
Vector3 headPosition = Camera.main.transform.position;
Vector3 gazeDirection = Camera.main.transform.forward;
RaycastHit hitInfo;
if (Physics.Raycast(headPosition, gazeDirection, out hitInfo, 30.0f, spatialMappingManager.LayerMask))
{
// Rotate this object to face the user.
Quaternion toQuat = Camera.main.transform.localRotation;
toQuat.x = 0;
toQuat.z = 0;
// Move this object to where the raycast
// hit the Spatial Mapping mesh.
// Here is where you might consider adding intelligence
// to how the object is placed. For example, consider
// placing based on the bottom of the object's
// collider so it sits properly on surfaces.
if (PlaceParentOnTap)
{
// Place the parent object as well but keep the focus on the current game object
Vector3 currentMovement = hitInfo.point - gameObject.transform.position;
ParentGameObjectToPlace.transform.position += currentMovement;
ParentGameObjectToPlace.transform.rotation = toQuat;
}
else
{
gameObject.transform.position = hitInfo.point;
gameObject.transform.rotation = toQuat;
}
}
}
}
public virtual void OnInputClicked(InputClickedEventData eventData)
{
// On each tap gesture, toggle whether the user is in placing mode.
IsBeingPlaced = !IsBeingPlaced;
// If the user is in placing mode, display the spatial mapping mesh.
if (IsBeingPlaced)
{
spatialMappingManager.DrawVisualMeshes = true;
Debug.Log(gameObject.name + " : Removing existing world anchor if any.");
//anchorManager.RemoveAnchor(gameObject);
}
// If the user is not in placing mode, hide the spatial mapping mesh.
else
{
spatialMappingManager.DrawVisualMeshes = false;
// Add world anchor when object placement is done.
//anchorManager.AttachAnchor(gameObject, SavedAnchorFriendlyName);
OnSelect();
}
}
public void OnSelect()
{
// Note that we have a transform.
GotTransform = true;
Debug.Log("TapToPlaceShared > OnSelect (send transform)");
if (CustomMessages.Instance != null)
{
if (PlaceParentOnTap)
{
CustomMessages.Instance.SendStageTransform(ParentGameObjectToPlace.transform.localPosition, transform.localRotation);
}
else
{
CustomMessages.Instance.SendStageTransform(transform.localPosition, transform.localRotation);
}
}
}
/// <summary>
/// When a remote system has a transform for us, we'll get it here.
/// </summary>
/// <param name="msg"></param>
void OnStageTransfrom(NetworkInMessage msg)
{
Debug.Log("TapToPlaceShared > Received transform");
// We read the user ID but we don't use it here.
msg.ReadInt64();
//anchorManager.RemoveAnchor(gameObject);
if (CustomMessages.Instance != null)
{
if (PlaceParentOnTap)
{
ParentGameObjectToPlace.transform.localPosition = CustomMessages.Instance.ReadVector3(msg);
ParentGameObjectToPlace.transform.localRotation = CustomMessages.Instance.ReadQuaternion(msg);
}
else
{
gameObject.transform.localPosition = CustomMessages.Instance.ReadVector3(msg);
gameObject.transform.localRotation = CustomMessages.Instance.ReadQuaternion(msg);
}
}
// Update World Anchor
//anchorManager.AttachAnchor(gameObject, SavedAnchorFriendlyName);
GotTransform = true;
}
private void DetermineParent()
{
if (ParentGameObjectToPlace == null)
{
if (gameObject.transform.parent == null)
{
Debug.LogError("The selected GameObject has no parent.");
PlaceParentOnTap = false;
}
else
{
Debug.LogError("No parent specified. Using immediate parent instead: " + gameObject.transform.parent.gameObject.name);
ParentGameObjectToPlace = gameObject.transform.parent.gameObject;
}
}
}
}
}
I’ve made the following changes to the original TapToPlace script:
- Initialization: in the
Start()method, the script registers itself as message handler for incomingStageTransformmessages from theCustomMessagesinstance. We want to be informed whenever our HoloLens receives a position update from the server. - Receiving updates: we wired up the
OnStageTransform()method as callback handler for incoming messages. In that method, our class reads the position and rotation from the network message and applies it to the local transform. - Sending updates: in
OnSelect(), the code now also shares the new position update with other HoloLenses. Depending on the configuration, the script would either send the position of the game object it’s attached to, or the parent. That follows the configuration options of the originalTapToPlace. - Anchors: now that’s a tricky thing. World Anchors in Unity connect a game object to a physical position in the world. You can not move the position of an object when it still has a world anchor attached. This valuable piece of info is hidden in the documentation.
The defaultTapToPlacescript removes the world anchor, then updates the position when the user taps, and finally adds a new anchor. That works, as there is some time between removing the anchor until the user has placed the object at the new position. In our case, we’d move the object in a single frame. Well, removing the world anchor is asynchronous, as such that wouldn’t work. Therefore, you would need to destroy the anchor immediately.
In my implementation, I’ve removed the use of world anchors for now – these lines are currently commented out, as the object is placed in the real world nevertheless and it works for the Spectator View scenario. I’ll add the anchors back in later. But to check that everything works, I didn’t want anchors to interfere with any custom placements.
What’s next?
We’re almost there! The next blog post will add the final tweak to our project and show you what you get out of the Spectator View!
HoloLens Spectator View blog post series
This post is part of a short series that guides you through integrating HoloLens Spectator View into your own Mixed Reality app: