Tag: ARCore

Categories
Android App Development AR / VR

Getting Started with Google ARCore, Part 1: Project Setup & ARCore SDK

ARCore - Plane Detection running on the Google Pixel 2

ARCore by Google is still in preview and only runs on a select few phones including the Google Pixel 2. In this article, I’m creating a demo app for ARCore using the ARCore SDK for Unity (Preview 1).

It’s following up on the blog post series where I segmented a 3D model of the brain from an MRI image. Instead of following these steps, you can download the final model used in this article for free from Google Poly.

ARCore vs Tango

Previously, the AR efforts of Google were focused on the Tango platform. It included additional hardware depth sensors for accurate recognition of the environment. Unfortunately, only two phones are commercially available equipped with the necessary hardware to run Tango – the Asus ZenFone AR and the Lenovo Phab 2 Pro.

Continue reading “Getting Started with Google ARCore, Part 1: Project Setup & ARCore SDK”
Categories
AR / VR HoloLens Image Processing

Capturing a 3D Point Cloud with Intel RealSense and Converting to a Mesh with MeshLab

Depth and Color images from the RealSense Viewer

When dealing with Augmented and Virtual Reality, one of the most important tasks is capturing real objects and creating 3D models out of these. In this guide, I will demonstrate a quick method using the Intel RealSense camera to capture a point cloud. Next, I’ll convert the point cloud to a mesh using MeshLab. This mesh can then be exported to an STL file for 3D printing. Another option is visualization in 3D for AR / VR, where I’ll also cover how to preserve the vertex coloring from transferring the original point cloud to Unity.

Continue reading “Capturing a 3D Point Cloud with Intel RealSense and Converting to a Mesh with MeshLab”
Categories
3D Printing AR / VR Digital Healthcare HoloLens Image Processing Windows

Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 4: Segmenting the Brain

3D Builder: show 3D model of brain segmented from MRI / MRT image

In the previous blog posts, we’ve used a simple grayscale threshold to define the model surface for visualizing an MRI / CT / Ultrasound in 3D. In many cases, you need to have more control over the 3D model generation, e.g., to only visualize the brain, a tumor, or a specific part of the scan.

In this blog post, I’ll demonstrate how to segment the brain of an MRT image; but the same method can be used for any segmentation. For example, you can also build a model of the skull based on a CT by following the steps below.

Continue reading “Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 4: Segmenting the Brain”
Categories
3D Printing AR / VR Digital Healthcare HoloLens Image Processing Windows

Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 3: 3D Model Maker

MRI 3D Model visualized in Windows 10 3D Builder

So far, we’ve created a volume rendering of an MRI / CT / Ultrasound scan. This is based on Voxels. For 3D printing and highly performant visualization in AR / VR scenarios, we need to create and export a polygon-based model. For the first step, we will use the Grayscale Model Maker and export the 3D Model as .stl to further prepare the model.

To create a 3D model, we have two main options in 3D Slicer:

  • Grayscale Model Maker: directly uses grayscale values from the image data. A threshold defines the surfaces. The model maker also takes care of smoothing the surfaces and reducing the polygon count.
  • Model Maker: this requires labels or discrete data to build a 3D model, meaning you have to segment the image data.

As a first step, we will use the Grayscale Model Maker, and later explore the more advanced options offered by segmentation and the Model Maker.

Continue reading “Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 3: 3D Model Maker”
Categories
3D Printing AR / VR Digital Healthcare HoloLens Image Processing Windows

Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 2: 3D Volume Rendering

Region of Interest in a 3D Volume Rendering of an MRI in 3D Slicer

After importing the MRI / CT / Ultrasound data into 3D Slicer in part 1, we’re ready for the first 3D visualization inside the medical software through 3D Volume Rendering. This is a major step to export the 3D model to Unity for visualization through Google ARCore or Microsoft HoloLens, or for 3D printing.

Slices in 3D View

After optimizing brightness and contrast of the image data, the easiest way of showing the data in 3D is to visualize the three visible slices (planes: axial / top / red; sagittal / side / yellow; coronal / frontal / green view) in the 3D view. This gives a good overview of the position and the relation of the slices to each other.

Continue reading “Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 2: 3D Volume Rendering”
Categories
3D Printing AR / VR Digital Healthcare HoloLens Image Processing Windows

Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 1: Importing Data

MRI data with auto W/L in 3D Slicer

Some of the best showcases of Mixed Reality / VR / AR include 3D visualizations of MRI (magnetic resonance imaging), CT (computer tomography) or ultrasound scans. 3D brings tremendous advantages for analyzing the scanned images compared to only viewing 2D slices. Additionally, a good visualization brings value to patients who can gain a better understanding if they can easily explore their own body.

As part of the 3D information visualization lecture at the FH St. Pölten, I’m giving an overview of the process of converting an MRI / CT / ultrasound scan into a hologram that you can view on the Microsoft HoloLens or with Google ARCore. This blog post series explains the hands-on parts, so that you can easily re-create the same results using freely available tools.

Continue reading “Visualizing MRI & CT Scans in Mixed Reality / VR / AR, Part 1: Importing Data”