App Development Artificial Intelligence Digital Healthcare

Using Natural Language Understanding, Part 4: Real-World AI Service & Socket.IO

In this last part, we bring the vital sign check list to life. Artificial Intelligence interprets assessments spoken in natural language. It extracts the relevant information and manages an up-to-date, browser-based checklist. Real-time communication is handled through Web Sockets with Socket.IO.

The example scenario focuses on a vital signs checklist in a hospital. The same concept applies to countless other use cases.

In this article, we’ll query the Microsoft LUIS Language Understanding service from a Node.js backend. The results are communicated to the client through Socket.IO.

Connecting LUIS to Node.JS

In the previous article, we verified that our LUIS service works fine. Now, it’s time to connect all components. The aim is to query LUIS from our Node.js backend.

You have multiple options for the service connection. Microsoft provides a Node module on GitHub – however, it hasn’t been updated for quite a while and only offers basic functionality.

As we want to see how queries work, let’s implement the communication between Node.js and LUIS ourselves. This is similar to the Microsoft tutorial for getting started with Node.js and LUIS. But we go a bit farther than this example and send the results to the HTML5 user interface via Socket.IO. Also, I’ve replaced the deprecated “request” module with the up-to-date “node-fetch” module.

Importing Modules and Setting Constants

Previously, we’ve already installed the “” and “node-fetch” Node modules. Let’s extend the first few lines of “index.js” from our current Node app. The new code loads and initializes everything, including Socket.IO, the “node-fetch” module and the built-in “querystring” module:

const express = require('express');
const app = express();
var http = require('http').Server(app);
var io = require('')(http);
const fetch = require('node-fetch');
var querystring = require('querystring');

Add the configuration for accessing your LUIS service directly below. Both keys are longer hexadecimal strings.

const APPID = "3ae73467-55db-xxxxxxxxx";
const APPKEY = "e231xxxxxxxx";
const ENDPOINT = `${APPID}/slots/production/predict`;

You can find the endpoint URL and the primary key from your LUIS portal in the “Manage” tab in the “Azure Resources” blade.

The App ID is listed in the “Settings blade” of the same “Manage” tab.

React to Incoming Messages

Whenever a client sends an “assessment” message to our Node.js server, we forward the query to the LUIS service.

Therefore, we create a connection that executes our handler method every time we receive a message with the name “assessment” (socket.on('assessment', ...)).

In the client-side JavaScript code, we sent the user-entered assessment text through the message parameter. The parameter is available through the function argument called “msg” in the code blow (async (msg) => {}).

Via, we get the unique ID of the client that sent the message. We’ll need this to later send the results back to that specific user. To keep the code clean, the actual message processing is placed in a new method we’ll create as the next step, called sendToLuis(assessment, socketId) .

io.on('connection', async (socket) => {
    socket.on('assessment', async (msg) => {
        console.log("Got assessment: " + msg);
        await sendToLuis(msg,;

Creating the Query URL for Communicating with LUIS

You should always check parameters for validity. I’m omitting this part in the article. For reference, check it out in the extended code on GitHub.

First, we’re building the HTTPS query URL. It’s a combination of the “ENDPOINT” URL, the “APPKEY”, “APPID” and the actual assessment text.

The parameters are processed by the “querystring” module. Its stringify method serializes the variables from “queryParams” into a suitable form for URLs.

async function sendToLuis(assessment, socketId) {
    // Send to LUIS
    var queryParams = {
        "subscription-key": APPKEY,
        "verbose": true,    // We need verbose so that we can access the entity data in a generic way
        "show-all-intents": false,
        "log": true,        // Also log query to LUIS console
        "query": assessment

    var luisRequest =
        ENDPOINT +
        '?' + querystring.stringify(queryParams);

The requesting URL is similar to the URL we used in the previous part for testing the LUIS app in the browser. You can easily test it by printing the final “luisRequest” variable contents to the console. This URL is all we need to query LUIS.

Communicating with LUIS over HTTPS

The LUIS service is of course available via an encrypted HTTPS connection instead of HTTP. Node.js contains a built-in module called HTTPS. It allows detailed control but is very low-level.

For our needs, it’s easier to use the well-known node-fetch module. It’s built on top of HTTPS and simplifies common usage scenarios. We’ve already installed it to our Node.js app. We also instantiated it through the variable called “fetch”.

“fetch” requires one parameter: obviously, the query URL. It then returns the response, which we can await. Querying the web service can take some time, so this is an asynchronous operation.

    try {
        const response = await fetch(luisRequest);
        if (!response.ok) {
            console.log('Error with request: ' + response.statusText);
  'Error', 'Sorry, I had problems with the request. ' + response.statusText);
        const data = await response.json();
        console.log("Received data: " + JSON.stringify(data));

        // // Check if the relevant properties exist
        if (!data.prediction.entities.$instance ||
            !Object.keys(data.prediction.entities.$instance) ||
            !Object.keys(data.prediction.entities.$instance)[0] ||
            !data.prediction.entities.$instance[Object.keys(data.prediction.entities.$instance)[0]][0].text) {
            console.log('Missing query, intent or entity: ');
  'Error', 'Sorry, I could not understand that. [No intent or entity identified]');

        const instance = data.prediction.entities.$instance;
        const firstInstance = Object.keys(instance)[0];
        const firstInstanceText = instance[firstInstance][0].text;, firstInstanceText);
    } catch (error) {
        console.log(error);'Error', 'Error understanding your assessment: ' + error);


If there is a general exception with the request, the function directly returns an error message to the client:'Error', ...)

Otherwise, the function parses the body and checks if the JSON contains the data we need.

If everything we expect is present in the response, the function emits a message with the intent name to the client (in our app: “Temperature”, “PupillaryResponse” or “Age”).

LUIS would parse the contents for us, which results in a specific format depending on what we’re listening to (temperature, age, custom list). In our case, we simply want to get the relevant text of the entity instance. There is a bit of JSON parsing going on here to extract that. But simply copy the code and it should work.

As all three intents we’re using only have a single entity each, we don’t need to consider any potential additional entities.

That’s how simple the communication with the LUIS service is. Of course, you should add a bit more error handling. The example source code on GitHub includes some console outputs. This makes it easier to see what’s happening while running the app.

Finalizing the New index.js: Starting the Server

There’s one more change we need in the “index.js” file: a slight adaption of the “listen” statement, compared to the original Hello World code. Change app.listen to http.listen – this ensures that is looped into the web server’s communication.

http.listen(PORT, () => console.log(`Vital Signs Checklist server listening on port ${PORT}!`));

Testing the Vital Signs Checklist with Natural Language Understanding

Now we have all components in place to test the overall service. Restart the Node.js server from the console (node index.js) and open the client-side website in your browser (

Next, enter an assessment in natural language into the text box at the top. After a moment, you’ll see the result in the client UI. Otherwise, a yellow error note appears right below the text box.

Vital Sign Checklist based on Cognitive Services / Natural Languate Understanding

Active Learning: Improve Results

The LUIS service logs all received utterances and its understanding. Especially if LUIS was uncertain about a sentence, you can easily correct the classification. The additional knowledge enhances the re-trained model.

This is called active learning and is extremely important to improve the quality of the language understanding over time.

LUIS: Review endpoint utterances for active learning

Download & Next Steps

You can download a slightly extended version of the source code explained in this article from GitHub. There are many interesting extension areas to make the app even more versatile, for example:

  • Adding speech input for recording verbal assessments
  • Extending the number of recognized scenarios
  • Storing the values in a database and tagging them with patients and timestamps

All these steps would be required to create a fully usable project. However, you can see how easy it is to get started. Within a short timeframe, we implemented a minimum viable prototype. It already supports an amazing core functionality: extracting relevant vital sign data from natural language assessments. With adaptive learning, it’s easy to continually improve the results. And there are countless more scenarios where the LUIS service helps getting things done quicker!

In addition, our app is ready to be published to Azure App Service. Simply install the extension into Visual Studio Code and you can deploy to a live Node.js server running in the cloud with a few clicks!

Article Series: Using Natural Language Understanding