Capturing a 3D Point Cloud with Intel RealSense and Converting to a Mesh with MeshLab

Depth and Color images from the RealSense Viewer

When dealing with Augmented and Virtual Reality, one of the most important tasks is capturing real objects and creating 3D models out of these. In this guide, I will demonstrate a quick method using the Intel RealSense camera to capture a point cloud. Next, I’ll convert the point cloud to a mesh using MeshLab. This mesh can then be exported to an STL file for 3D printing. Another option is visualization in 3D for AR / VR, where I’ll also cover how to preserve the vertex coloring from transferring the original point cloud to Unity. Continue reading “Capturing a 3D Point Cloud with Intel RealSense and Converting to a Mesh with MeshLab”

Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 4: Segmenting the Brain

3D Builder: show 3D model of brain segmented from MRI / MRT image

In the previous blog posts, we’ve used a simple grayscale threshold to define the model surface for visualizing a MRI / CT / Ultrasound in 3D. In many cases, you need to have more control over the 3D model generation, e.g., to only visualize the brain, a tumor or a specific part of the scan.

In this blog post, I’ll demonstrate how to segment the brain of an MRT image; but the same method can be used for any segmentation. For example, you can also build a model of the skull based on a CT by following the steps below. Continue reading “Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 4: Segmenting the Brain”

Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 3: 3D Model Maker

MRI 3D Model visualized in Windows 10 3D Builder

So far, we’ve created a volume rendering of a MRI / CT / Ultrasound scan. This is based on Voxels. For 3D printing and highly performant visualization in AR / VR scenarios, we need to create and export a polygon-based model. For the first step, we will use the Grayscale Model Maker and export the 3D Model as .stl to further prepare the model.

To create a 3D model, we have two main options in 3D Slicer:

  • Grayscale Model Maker: directly uses grayscale values from the image data. A threshold defines the surfaces. The model maker also takes care of smoothing the surfaces and reducing the polygon count.
  • Model Maker: this requires labels or discrete data to build a 3D model, meaning you have to segment the image data.

As a first step, we will use the Grayscale Model Maker, and later explore the more advanced options offered by segmentation and the Model Maker. Continue reading “Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 3: 3D Model Maker”

Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 2: 3D Volume Rendering

Region of Interest in a 3D Volume Rendering of an MRI in 3D Slicer

After importing the MRI / CT / Ultrasound data into 3D Slicer in part 1, we’re ready for the first 3D visualization inside the medical software through 3D Volume Rendering. This is an important step to ultimately export the 3D model to Unity for visualization through Google ARCore or Microsoft HoloLens, or for 3D printing.

Slices in 3D View

After optimizing brightness and contrast of the image data, the easiest way of showing the data in 3D is to visualize the three visible slices (planes: axial / top / red; sagittal / side / yellow; coronal / frontal / green view) in the 3D view. This gives a good overview of the position and the relation of the slices to each other. Continue reading “Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 2: 3D Volume Rendering”

Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 1: Importing Data

MRI data with auto W/L in 3D Slicer

Some of the best showcases of Mixed Reality / VR / AR include 3D visualizations of MRI (magnetic resonance imaging), CT (computer tomography) or ultrasound scans. 3D brings tremendous advantages for analyzing the scanned images compared to only viewing 2D slices. Additionally, a good visualization brings value to patients who can gain a better understanding if they can easily explore their own body.

As part of the 3D information visualization lecture at the FH St. Pölten, I’m giving an overview of the process of converting an MRI / CT / ultrasound scan into a hologram that you can view on the Microsoft HoloLens or with Google ARCore. This blog post series explains the hands-on parts, so that you can easily re-create the same results using freely available tools. Continue reading “Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 1: Importing Data”

How to Combine the Mixed Reality Toolkit, Unity 2017.2 and Visual Studio 2017

MRI scan visualized through HoloLens and Mixed Reality

Update 13. November 2017: The latest source code of the Mixed Reality toolkit now combines both HoloLens and Mixed Reality headsets into a single toolkit that works with one Unity version: 2017.2.0p1 MRTP 4. It’s a special fork of Unity that is optimized for the “Mixed Reality Toolkit Preview”. A later version of Unity will hopefully combine all environments into a single release again. Read more about the environment setup at the GitHub pull request.

Update 19. October 2017: In the meantime, Unity 2017.2 final has been released, and the dev branch of the Mixerd Reality toolkit has been merged back to the master. You should now be fine using the following versions for HoloLens development: Unity 2017.2.0f3+, Mixed Reality Toolkit (master branch), Visual Studio 2017.+4, Windows 10.0.15063.0 SDK.

Original Article: Lately, the tools required for HoloLens / Mixed Reality development have been undergoing profound changes. All three tools involved in building HoloLens apps are being restructured:

  • Unity 2017 unifies Virtual / Augmented Reality APIs, making them flexible enough to target all platforms (e.g., phones with ARKit / ARCore, VR, AR). This also involves new and renamed APIs.
  • HoloToolkit has been renamed to Mixed Reality Toolkit, as Microsoft expands the scope to include the new VR headsets with inside-out tracking going on sale this fall.
  • Visual Studio 2017.3 also introduced some major changes under the hood. This is combined with the C# engine used in Unity slowly being migrated from the old Mono runtime to more recent versions of C#.

With the latest Unity 2017.2.0b11 release, everything should now be coming together. In this blog post, I’m describing how to use the latest versions of the tools for creating and deploying a HoloLens app. Continue reading “How to Combine the Mixed Reality Toolkit, Unity 2017.2 and Visual Studio 2017”

Resoving Unity Scene Merge Conflicts with UnityYAMLMerge (Smart Merge) and TortoiseGit

When working on Unity HoloLens-projects in teams, sometimes merge conflicts in Unity scenes are unavoidable. Even though the Unity scene file format is text-based, the automatic merge of a standard GIT merge tool wouldn’t always correctly recognize the changes from different versions.

Luckily, Unity comes with a merging tool that is specialized on scene files: UnityYAMLMerge / Smart Merge. However, it’s not straight-forward to integrate into a workflow. Continue reading “Resoving Unity Scene Merge Conflicts with UnityYAMLMerge (Smart Merge) and TortoiseGit”

More Realistic HoloLens Spectator View Photos

HoloLens Spectator View - Photoshop - Screen - Darkened

When capturing a Mixed Reality scene through HoloLens Spectator View, the resulting photos and videos look amazing. But how to make them more realistic? The right blending mode in Photoshop helps.

Are Spectator View Photos Real?

Of course, the captured Spectator View photo is in many ways different to what you see in HoloLens. First off, it’s only 2D; in the HoloLens, you’d see a real 3D hologram. In this area, the live HoloLens view is much more impressive.

On the other hand, the actual resolution and field of view of the HoloLens is less than a (potentially) 20 Megapixel+ Unity rendering that shows holograms all around you. But the field of view is difficult and somehow unnecessary to simulate as we’re looking on the scene from a 3rd person perspective. Continue reading “More Realistic HoloLens Spectator View Photos”

How to set up HoloLens Spectator View, Bonus – Troubleshooting

HoloLens Delete Space

Did you follow the blog post series on setting up Spectator View for HoloLens? The following issues frequently occured when testing Spectator View. To make your life easier, make sure you check these troubleshooting tips & tricks:

HoloLens Sharing Offset

Between your two HoloLenses, there can be some offset in the placing of the holograms. In one HoloLens, you can for example position your hologram directly on the table; on the second HoloLens, it’s one meter next to the table. Continue reading “How to set up HoloLens Spectator View, Bonus – Troubleshooting”

How to set up HoloLens Spectator View, Part 7 – Mixed Reality

HoloLens Spectator View - Sample Photo

In the final part of the HoloLens Spectator View series, I’ll add the final tweak to my sample project and show you what kind of photos the Compositor actually saves on your PC. Getting to this point has taken a lot more work than expected (+ a lot more blog posts, too!), but I still feel that the journey was worth it! You can finally get great photos and videos of the mixed reality experience on HoloLens.

Script Execution Order

Sending the updates didn’t work right away, so I searched for other changes that the sample does differently from my app that might be responsible. One such case was the Script Execution Order.

In Unity, the order in which scripts are initialized is arbitrary. When debugging, I found an issue that an instance of a class wasn’t available yet when a script wanted to access it. Continue reading “How to set up HoloLens Spectator View, Part 7 – Mixed Reality”