Using Natural Language Understanding, Part 4: Real-World AI Service & Socket.IO

In this last part, we bring the vital sign check list to life. Artificial Intelligence interprets assessments spoken in natural language. It extracts the relevant information and manages an up-to-date, browser-based checklist. Real-time communication is handled through Web Sockets with Socket.IO.

The example scenario focuses on a vital signs checklist in a hospital. The same concept applies to countless other use cases.

In this article, we’ll query the Microsoft LUIS Language Understanding service from a Node.js backend. The results are communicated to the client through Socket.IO.

Connecting LUIS to Node.JS

In the previous article, we verified that our LUIS service works fine. Now, it’s time to connect all components. The aim is to query LUIS from our Node.js backend.

You have multiple options for the service connection. Microsoft provides a Node module on GitHub – however, it hasn’t been updated for quite a while and only offers basic functionality.

As we want to see how queries work, let’s implement the communication between Node.js and LUIS ourselves. This is similar to the Microsoft tutorial for getting started with Node.js and LUIS. But we go a bit farther than this example and send the results to the HTML5 user interface via Socket.IO.

Importing Modules and Setting Constants

Previously, we’ve already installed the “socket.io” and “request” Node modules. Let’s extend the first few lines of “index.js” from our current Node app. The new code loads and initializes everything, including Socket.IO, the “request” module and the built-in “querystring” module:

Add the configuration for accessing your LUIS service directly below. Both keys are longer hexadecimal strings.

You can find the endpoint URL and the application ID from your LUIS portal in the “Settings” tab.

The application key is listed in Azure if you connected it (programmatic key). Alternatively, if you’re using the starter key (endpoint key) and didn’t sign up to Azure, find the key as part of the Endpoint URL in the “Publish” tab.

React to Incoming Socket.io Messages

Whenever a client sends an “assessment” message to our Node.js server, we forward the query to the LUIS service.

Therefore, we create a connection that executes our handler method every time we receive a message with the name “assessment” ( socket.on('assessment', ...)).

In the client-side JavaScript code, we sent the user-entered assessment text through the message parameter. The parameter is available through the function argument called “msg” in the code blow ( function (msg) {}).

Via socket.id, we get the unique ID of the client that sent the message. We’ll need this to later send the results back to that specific user. To keep the code clean, the actual message processing is placed in a new method we’ll create as the next step, called sendToLuis(assessment, socketId) .

Creating the Query URL for Communicating with LUIS

You should always check parameters for validity. I’m omitting this part in the article. For reference, check it out in the extended code on GitHub.

First, we’re building the HTTPS query URL. It’s a combination of the “ENDPOINT” URL, the “APPKEY”, “APPID” and the actual assessment text.

The parameters are processed by the “querystring” module. Its stringify method serializes the variables from “queryParams” into a suitable form for URLs.

The requesting URL is similar to the URL we used in the previous part for testing the LUIS app in the browser. You can easily test it by printing the final “luisRequest” variable contents to the console. This URL is all we need to query LUIS.

Communicating with LUIS over HTTPS

The LUIS service is of course available via an encrypted HTTPS connection instead of HTTP. Node.js contains a built-in module called HTTPS. It allows detailed control, but is very low-level.

For our needs, it’s easier to use the well-known request module. It’s built on top of HTTPS and simplifies common usage scenarios. We’ve already installed it to our Node.js app. We also instantiated it through the variable called “request”.

“request” requires two parameters: first, obviously the query URL. The second parameter is a call-back function containing the results.

We implement the call-back function right there, which expects 3 parameters:

  1. Optional error message (“err”)
  2. Response data, which includes the status code in case the server responded (“response”)
  3. Body text of the HTTPS response (“body”)

If there is a general error, the function directly returns an error message to the client: io.to(socketId).emit('Error', ...)

Otherwise, the function parses the body and checks if the JSON contains a topScoringIntent and an entity.

If everything we expect is present in the response, the function emits a message with the intent name to the client (in our app: “Temperature”, “PupillaryResponse” or “Age”). The (first) entity is the message parameter. As all three intents we’re using only have a single entity each, we don’t need to consider any potential additional entities.

That’s how simple the communication with the LUIS service is. Of course, you should add a bit more error handling. The example source code on GitHub  includes some console outputs. This makes it easier to see what’s happening while running the app.

Finalizing the New Index.Js: Starting the Server

There’s one more change we need in the “index.js” file: a slight adaption of the “listen” statement, compared to the original Hello World code. Change app.listen to http.listen – this ensures that socket.io is looped into the web server’s communication.

Testing the Vital Signs Checklist with Natural Language Understanding

Now we have all components in place to test the overall service. Restart the Node.js server from the console ( node index.js) and open the client-side website in your browser ( http://127.0.0.1:3000/).

Next, enter an assessment in natural language into the text box at the top. After a moment, you’ll see the result in the client UI. Otherwise, a yellow error note appears right below the text box.

Vital Sign Checklist based on Cognitive Services / Natural Languate Understanding

Active Learning: Improve Results

The LUIS service logs all received utterances and its understanding. Especially if LUIS was uncertain about a sentence, you can easily correct the classification. The additional knowledge enhances the re-trained model.

This is called active learning and is extremely important to improve the quality of the language understanding over time.

LUIS: Review endpoint utterances for active learning

Download & Next Steps

You can download a slightly extended version of the source code explained in this article from GitHub. There are many interesting extension areas to make the app even more versatile, for example:

  • Adding speech input for recording verbal assessments
  • Extending the number of recognized scenarios
  • Storing the values in a database and tagging them with patients and timestamps

All these steps would be required to create a fully usable project. However, you can see how easy it is to get started. Within a short timeframe, we implemented a minimum viable prototype. It already supports an amazing core functionality: extracting relevant vital sign data from natural language assessments. With adaptive learning, it’s easy to continually improve the results. And there are countless more scenarios where the LUIS service helps getting things done quicker!

Article Series: Using Natural Language Understanding