andreasjakl.com – Digital Healthcare, Augmented Reality, Mobile Apps and more! Andreas Jakl is a lecturer for Digital Healthcare & Smart Engineering @ St. Pölten University of Applied Sciences, Microsoft MVP for Windows Development and Amazon AWS Educate Cloud Ambassador.

June 5, 2020June 5, 2020

Environmental HDR Lighting & Reflections in ARCore: Human Perception (Part 1)

Realistically merging virtual objects with the real world in Augmented Reality has a few challenges. The most important:

Realistic positioning, scale and rotation
Lighting and shadows that match the real-world illumination
Occlusion with real-world objects

The first is working very well in today’s AR systems. Number 3 for occlusion is working OK on the Microsoft HoloLens; and it’s soon also coming to ARCore (a private preview is currently running through the ARCore Depth API – which is probably based on the research by Flynn et al. ).

But what about the second item? Google put a lot of effort into this recently. So, let’s look behind the scenes. How does ARCore estimate HDR (high dynamic range) lighting and reflections from the camera image?

Remember that ARCore needs to scale to a variety of smartphones; thus, a requirement is that it also works on phones that only have a single RGB camera – like the Google Pixel 2.

May 19, 2020May 19, 2020

Alexa for Wellbeing Online Challenge

In the near future, we will primarily interact with technology through voice. Especially for older generations and kids, voice has the lowest entry barrier – compared to the complexity of computers or even smartphones. Simply start talking to speech assistants like Amazon Alexa, and they will help immediately.

To make most use of it, I’ve co-organized the “Alexa for Wellbeing Online Challenge” during the last few weeks. Together with AWS Educate and Hilfswerk Lower Austria, we’ll host a 10-day online hackathon, open to everyone.

August 29, 2019August 30, 2019

Using Amazon Sumerian in Trainings and Classrooms with AWS IAM

In this article, we’ll configure AWS Identity and Access Management (IAM) to easily use Amazon Sumerian with multiple users. This is especially important for classrooms or trainings. You often don’t want to loose time by having attendees set up and activate their own AWS accounts, including their personal credit cards.

Instead, by setting up sub-users in your account beforehand, you have complete control over the experience and can get started right away. Additionally, it helps with troubleshooting for exercises.

Right now, no ready-made AWS Educate classrooms are available that support Amazon Sumerian. If that changes, the classrooms would be a good alternative, as it gives students their own free AWS credits instead of everything billed to a central account.

Securing Your Account

The first step is making sure you own root account is properly secured. A major part is enabling Multi-Factor Authentication (MFA) for your root account. Especially when working in teams and with source control, it’s an easy-to-make mistake to upload your credentials somewhere; you don’t want others to have full control over your whole AWS account, as this can incur major charges to your credit card. Therefore, it’s best to enable MFA before you continue.

August 8, 2019August 9, 2019

Hit Test & Augmented Reality Anchors with Amazon Sumerian (Part 3)

Three call-backs to get from a touch event to a registered AR anchor in Amazon Sumerian.

In an Augmented Reality scene, users looks at the live camera feed. Virtual objects anchor at specific positions of the real world. Our task is to let the user place virtual in the real world. To achieve that, the user simply taps on the smartphone screen. Through a hit test, our script then creates an anchor in the real world and links that to a virtual 3D model entity.

That’s the high level overview. To code this anchoring logic, a few intermediate steps are needed:

Hit Test: converts coordinates of the user’s screen tap and sends the normalized coordinates to the AR system. This checks what’s in the real world at that position.
Register Anchor: next, our script instructs the AR system to create an anchor at that position.
Link Anchor: finally, the ID of the created anchor is linked to our entity. This allows Sumerian to continually update the transform of our 3D entity. Thus, the object stays in place in the real world, even when the user moves around.

Transforming these steps into code, this is what our code architecture looks like. It includes three call-backs, starting with the touch event and ending with the registered and linked AR anchor.

July 30, 2019August 9, 2019

Augmented Reality Anchors and Amazon Sumerian’s ArAnchorComponent (Part 2)

The WebXR standard isn’t finished yet. How does the web-based Amazon Sumerian platform integrate with the real world for Augmented Reality? We’ll take a look at the glue that binds the 3D WebGL contents from the web view to the native AR platform (ARCore / ARKit). To access this, we will also look at Sumerian internal engine classes like ArAnchorComponent, which handle the cross-platform web-to-native mapping.

This article continues from part 1, which covered the scripting basics of Amazon Sumerian and prepared the scene for AR placement.

Anchors in Amazon Sumerian

Let’s start with a bit of background of how Sumerian handles AR.

Ultimately, a 3D model is placed in the user’s real environment using an “Anchor”. This is directly represented in Sumerian. To create an anchor in your scene, your code goes through the following steps:

July 26, 2019August 9, 2019

Augmented Reality Object Placement with Amazon Sumerian (Part 1)

Repositioning your AR objects on the floor with Amazon Sumerian

How to (re)-position the virtual objects in the real world in an Augmented Reality experience – while still having an interactive scene? Elegantly guide your users through the placement process.

The official AR tutorial from Amazon contains a simple script: by tapping anywhere in the scene, it instantly moves the objects to that position. However, for the Digital Healthcare Explained app, I needed a more flexible behavior:

Activate placement mode by tapping on a specific object in the 3D scene. In this case, I decided that tapping the host avatar triggers placement mode.
The host then explains what to do: tapping on another surface moves the host and related objects. Guide users through the process. The Sumerian hosts are ideal to explain the process.
The user taps on a real-world surface in the AR scene.
Next, the scene contents move, the anchor updates and the host confirms.

New ES6 Based Scripting

Additionally, Amazon Sumerian is evolving its scripting language. A major upgrade to ES6 is underway. It’s fully based on classes and fits better into the actions and state machines used in other places of Sumerian. The new APIs are still marked as “Preview”. However, the old APIs are already called “Legacy” or “Old Script Format”.

While documentation for the new Sumerian Engine APIs directly is already available, it’s very brief and doesn’t contain many examples. The official tutorials are still based on the legacy API.

I decided to re-write the script using the new APIs. It involves calling a lot of internal parts of Sumerian. Thus, it’s a lot more complex than all other examples for the new API currently out there. However, it’s interesting to dig more into the internals of how a modern, web-based AR environment works.

June 19, 2019June 19, 2019

Create engaging Healthcare Experiences with Augmented Reality

Download hands-on workshop slides and material for a complete getting-started guide to your first 3D experience – with a background in digital healthcare!

Conference Session

The build.well.being conference is an annual networking event for the doers in Digital Healthcare. The fast-paced event compresses a lot of useful information into a short day: sessions from health professionals (including a keynote from Brian Anthony, associate director at the MIT.nano). Student project pitches. Plus: hands-on workshops.

Amazon Sumerian Workshop @ build.well.being, © FH St. Pölten | Tobias Sautner

Together with Anna Runefelt, I was running a challenging workshop: introducing attendees with a healthcare background to the world of Augmented / Virtual Reality. The aim of the hands-on workshop: creating your first live 3D experience in about 1 hour. This was possible thanks to the easy-to-use interface of Amazon Sumerian. Most of the attendees who followed along indeed managed to get a fully working 3D scene running on their laptops.

March 20, 2019March 20, 2019

Computer Vision & Photo Description: Really Simple HTML / JavaScript Example

Example overview: specify an image URL and your API key, get a complete image description from an Artificial Intelligence service.

Image classification & content description is incredibly powerful. Cloud-based computer vision services instantly return a JSON-based description of what they see in photos.

However, most examples are quite complex. As a beginning developer with your main knowledge in HTML + JavaScript, the following code is for you. You don’t need to worry about Node.js or native apps. The code runs directly in your browser from your computer.

January 25, 2019January 28, 2019

How-To: Retrofit, Moshi, Coroutines & Recycler View for REST Web Service Operations with Kotlin for Android

It might be overwhelming to choose the best way to access a web service from your Android app. Maybe all you want is to parse JSON from a web service and show it in a list in your Kotlin app for Android, while still being future-proof with a library like Retrofit. As a bonus, it’d be great if you could also perform CRUD operations (create, read, update, delete) with the data.

You can choose from basic Java-style HTML requests, or go up to full-scale MVVM design patterns with the new Android Architecture Components. Your source code will look entirely different depending on what approach you chose – so it’s important to make a good choice right at the beginning.

In this article, I’ll show a walk-through using many of the newest components for a modern solution:

Retrofit v2: web service access
Moshi: converting JSON to Kotlin data classes and back
Kotlin Coroutine Adapter for Retrofit: handles threading for asynchronous web access
Object declaration: for singleton patterns in Kotlin
RecyclerView with a click adapter: showing items in a list

December 15, 2018August 30, 2019

Download, Export or Backup Amazon Sumerian Scenes (Part 6)

Amazon Sumerian is a purely cloud-based tool. Its scenes are intended to be run directly from the cloud. As such, one of the most common questions is: how can I download the scene I created in Amazon Sumerian? You might want to do this to have a backup, to send it to a colleague or to move the scene to another region.

Snapshots

The easiest way to backup your scene is to create a snapshot. This is directly integrated into the main Sumerian editor UI. Select the root node of the scene in the Entities panel. Next, navigate to the “Snapshots” section in the inspector panel.

In this panel, you can create snapshots that are easy to return to later. I’d recommend creating a snapshot before major changes in your app, after finishing vital parts of code, and obviously for every publicly released version.

However, the snapshots are internal to the scene. How to get content from one scene to another?