More Realistic HoloLens Spectator View Photos

When capturing a Mixed Reality scene through HoloLens Spectator View, the resulting photos and videos look amazing. But how to make them more realistic? The right blending mode in Photoshop helps.

Are Spectator View Photos Real?

Of course, the captured Spectator View photo is in many ways different to what you see in HoloLens. First off, it’s only 2D; in the HoloLens, you’d see a real 3D hologram. In this area, the live HoloLens view is much more impressive.

On the other hand, the actual resolution and field of view of the HoloLens is less than a (potentially) 20 Megapixel+ Unity rendering that shows holograms all around you. But the field of view is difficult and somehow unnecessary to simulate as we’re looking on the scene from a 3rd person perspective.

Artificially reducing the rendering resolution and making your holograms blurry wouldn’t really make sense either. When wearing the HoloLens, the resultion is actually less noticable compared to looking at a screenshot. My guess is that this is due to the fact that the displays are really close to your eyes; plus – even more importantly – the HoloLens has a light-based display, which only adds light to the scene. Unless you’re in a really dark environment, the resolution of light pixels is hard to notice.

Light-Based Display & Shadows

In a previous blog post, I’ve already written about the HoloLens display & shadow. The HoloLens only adds light to the scene; it can’t make parts of it darker than they are. This means that you can’t add shadow or darker parts to the scene. For more info on how the HoloLens displays work, read this informative article from Tom’s Hardware.

The HoloLens works around that a bit by having the tinted visor, which makes your whole real-world view a bit darker. It helps the light-display to overlay the world in a way so that you can still see the holograms, even in (moderately) bright environments.

Spectator View Photos & Light

If you take a look at the picture created through the Compositor of Spectator View, you will notice that the rendered hologram actually does have dark or even black parts that completely hide the real world behind them.

This is not possible on the real HoloLens.

HoloLens Spectator View - Sample Photo

Take a look at the eyes of the skeleton. The opacity of the object is set to 90% through the Spectator View Composer window. Therefore, the environment does shine through a tiny little bit. But on the real HoloLens, the eyes wouldn’t be dark; they would be see-through. Everything rendered in black color is not visible on the HoloLens display.

Re-Building the Picture in Photoshop

Photoshop offers more ways for a more realistic blending of the hologram with the real world. To achieve this, you first need to re-build the scene. The Spectator View Compositor capture created 4 images:

Spectator View - Compositor Pictures

Follow these steps to re-create the final scene in Photoshop:

  1. Open pictures (1), (2) and (3) in Photoshop.
  2. Copy (2) as a new layer on top of (1).
  3. For this layer, select Layer > Layer Mask > Reveal All.
  4. Go to picture (3) and copy the contents to the clipboard (Ctrl+A, Ctrl+C).
  5. Back to (1). Hold down Alt and left-click on the layer mask that you created for (2). (by holding down Alt + clicking, you can edit the layer mask)
  6. Paste the alpha channel (Ctrl+V). This sets the layer mask.
  7. Left-click on the main image layer again to view the color image.

If you would like to exactly re-create the compositor image, you’d need to set the opacity of the layer containing (2) to 90%. However, that approach obviously has the same shortcomings.

Spectator View - Photoshop - Normal Blending 90% Opacity

Realistic Blending Mode

The “normal” blending mode is the classic approach, but it doesn’t simulate the light-based display. To more accurately get to this result, change the blending mode to “Screen” and the opacity back to 90%.

As you can see, the hologram is now hardly visible anymore. One issue is that we’re working with 8-bit per channel images. With HDR displays and images, it’d work better to make the hologram really shine on top of the real world.

The other reason for the low visibility is the unfiltered real world at full brightness. The HoloLens has the dark visor, which helps to make the holograms more visible.

Tinted Visor

To simulate this, we can simply add a dark layer between the holograms and the background. Create a new layer, fill it with black and set its opacity to a value that looks good and makes the hologram more visible. In this example, 55% work well:

HoloLens Spectator View - Photoshop - Screen Blending - Darkened

The final result

Of course, the image looks less impressive than the original HoloLens Spectator View image. However, it gives a more realistic impression of how the holograms actually look when they are part of the world.

Especially if you work with darker holograms in bright or complex surroundings, it might be difficult to see the holograms. With this more realistic blending approach, you can better convey the real look to your artists, so that they can optimize the 3D model.

HoloLens Spectator View - More Realistic blending mode of the hologram