Smarter ideas worth writing about.

J.A.R.V.I.S. - A Virtual Front Desk

Tags: Internet of Things , IoT , Raleigh

Background

On a daily basis the Cardinal office in Raleigh-Durham is visited by clients, food and parcel delivery people, as well as interviewees. Sometimes, a Cardinal employee doesn’t immediately greet these guests. Cardinal sought a solution that would make our office foyer more functional for our guests. With the Internet of Things on the rise, we challenged ourselves to build a system that could detect and interact with guests as well as allow employees to schedule visits and receive notifications that a guest has arrived. Our end result was J.A.R.V.I.S. (Just A Rather Very Intelligent System), an IoT solution that can assist our guests in finding the Cardinal employee they have come to see, acting as a virtual front-desk assistant to the Raleigh office.

What We Used

The Hardware

Device

At the heart of our solution is a single-board computer. Initially, we investigated incorporating a PINE64, a fully open-source “supercomputer” capable of running Android with a base price of $15. Unfortunately, although we were both backers of the Kickstarter project, overwhelming demand of the device resulted in continuous production delays. We decided to switch to using a Raspberry Pi 3B. The Raspberry Pi is arguably one of the most popular choices for small Internet of Things projects using single-board computers. It is also well documented with many example projects to reference for an understanding of how it all works. The 3B is comparable to previous Pi models but has a faster 64bit processor, a faster GPU and built-in WIFI and BLE.

Display

In order for a guest to interact with the system there needed to be a touch-screen attached to the Raspberry Pi. This screen would greet and prompt a visitor when they first entered the office foyer. Although we were able to find many differing sizes and models, we choose to use a 10-inch display made for the Pi and sold by WaveShare. It had a competitive price and was fully documented on the manufacture’s website. The screen is connected to the Pi via HDMI. The touch-screen proved to be nearly plug-n-play. Only minor additions were made to the operating system configuration file to adjust to the 1024x600 screen resolution.

Sensors

J.A.R.V.I.S needed a few different sensors to complete all of its functionality:

  • A PIR (motion) sensor was used to know when someone was walking in the door.
  • An ultrasonic rangefinder was used to determine the proximity of a guest to J.A.R.V.I.S.
  • A 5mp camera module was used to take a picture of the guest.


 


The Software

Operating Systems

The Raspberry Pi proved to be an extremely flexible computer with a number of possible directions in operating systems and frameworks. A new Pi usually comes with a micro SD card that usually has NOOBS (New Out of Box Software) installed. NOOBS is an operating system install manager that offers a number of possible operating systems to install directly on the SD card. To become more familiar with the Pi, we installed Debian, the de facto OS for the Pi. We eventually replaced the traditional Debian operating system with Resin. Resin is a client, server, and device software platform that uses Docker to image applications that are built locally on the device. Via Resin’s dashboard, an operating system can be imaged to be placed on the SD card that can then be placed in the Raspberry Pi. Communication and application is then performed via the Resin dashboard. Going this route insured that there was nothing else running on the Pi to interfere with the application.

Raspberry Pi Application

We first looked at using Python, arguably the Pi’s most popular development language. Knowing that we could easily incorporate certain requirements around the sensors and web services, we focused much of our initial discovery on satisfying our UI and UX needs with Python UI frameworks. To this end, we wanted to have an adequately sized touchscreen with an aesthetically pleasing front-end, something akin to contemporary design and experiences to make the end-user more comfortable with the interactions. In this discovery we experimented with Kivy, a popular yet relatively new open source Python framework intended for multi-touch applications.

In the end, we decided to use a JavaScript based solution, since the Pi supports Node.js. Node.js allowed us to interact with the sensors with a number of available npm packages to explore and utilize. We also discovered Electron, a Node.js framework that uses Chromium to build desktop applications with HTML, CSS and Javascript. With Electron we could employ React for our front-end, a contemporary framework that we both have had experience working with in previous projects.

We were confident that a Resin/Node.js/Electron/React combination would allow us to harness our experience in certain areas and allows to focus on unforeseen challenges and roadblocks that we would likely encounter. A Python / Kivy solution may of, in fact, worked well, but these would be additional technologies that we would have to get familiar with and in the end we decided to minimize the challenges ahead wherever possible but still meet our goals.

Employee Application

Cardinal employees needed a mechanism to schedule guest visits as well as for being notified when a guest arrives. Everyone throughout the office uses Slack as a way to communicate with one another so we created the “J.A.R.V.I.S.” Slackbot for that communication. The Slackbot was built with Node.js and deployed through Azure, saving and retrieving data through a Node.js REST API and a Mongo database hosted in mLab. Socket.io is used for the communication between the Slackbot and the Raspberry Pi kiosk.

How It Works

Raspberry Pi Kiosk

When someone walks through the front door, the motion sensor will be triggered and the touch screen will wake up, displaying the application. The proximity sensor will start sensing and waiting for the person to get close enough to the kiosk. If the person approaches the kiosk, they will be greeted by J.A.R.V.I.S. : “Hello, I am J.A.R.V.I.S. Welcome to Cardinal Solutions. Please check-in and I will find someone to help you.”. A number of Cardinal employees walk through the front foyer throughout the day. With the motion and proximity sensor combination we hoped to minimize J.A.R.V.I.S. intrusion on an employee’s workday while making the system readily able to detect a true guest.

The check-in feature is the main functionality of the application. For safety concerns we did not want to have a list of all of the employees available for anyone who happened to walk into our office. To address this safety concerns, employees can schedule appointments and be notified individually or an office wide notification will be sent out.

For a guest to check-in, a “check-in” button will be visible on the screen (see Image 1 – Check In Screen). When the guest presses the button, the application checks to see if any guests are scheduled within a 45-minute window of the current time. There are two paths the application can take:

  1. If guests are scheduled, a list will be displayed “Are you ______ looking for _____?” as well as a “Notify someone else” (see Image 2 – Prompt Screen)
    1. a. If “Are you _____ looking for _____?” is selected, a photo of the guest is taken and sent to the person expecting them via a Slack direct message from the J.A.R.V.I.S. Slackbot. J.A.R.V.I.S. notifies the user “Thank you, I will let ______ know you are here.”
    2. b. If “Notify someone else” is selected, a photo is taken of the guest and sent to the internal Slack channel for J.A.R.V.I.S. (see Image 3 – Photo Capture) Anyone that is subscribed to the channel will be able to see who is at the front door waiting for someone. J.A.R.V.I.S. replies to the guest “Thank you, I will let someone know you are here.”
  2. If there is not anyone scheduled within a 45-minute window of the current time, a photo is taken of the guest and sent to the internal Slack channel for J.A.R.V.I.S. Anyone that is subscribed to the channel will be able to see who is at the front door waiting for someone. J.A.R.V.I.S. replies to the guest “Thank you, I will let someone know you are here.”

Check-in Screen


Prompt Screen


Photo Capture

J.A.R.V.I.S. Slackbot

If an employee is expecting a guest at a particular time, they can send J.A.R.V.I.S. a direct message saying “I am expecting _____ at _____(am/pm)” and this will allow the kiosk to look out for that guest during a 45-minute window of the time scheduled. (see Image 4 – Schedule Visitor)

Scheduling  a Visitor

If an employee wants to see what guests are scheduled for the day, they can send J.A.R.V.I.S. a direct message saying “list” and they will get a response back with every guest scheduled for that particular day and let’s them know if each guest has checked in or not. (see Image 5 – List Guests)

Listing Today's Scheduled Guests

If an employee wants to see what guests were scheduled prior to the previous day, they can send J.A.R.V.I.S. a direct message saying “list xx/xx/xxxx” (replacing the x’s with a date) and they will get a response back with all the scheduled guests from that date to current date and if each of those have checked in or not. (see Image 6 – Previous Checkins)

Listing Previous Scheduled Guests

If an employee is not sure what J.A.R.V.I.S. is capable of, they can type “help” to see a list of everything it can help with. (see Image 7 – Help Screen)

Viewing Help Menu


Summary

The Internet of Things is growing rapidly as more and more devices become connected to each other. In this ever changing environment, it is important to understand what options are available and how the components can communicate. To get started with a project like this, there are a few things we learned to keep in mind:

  1. Choose a device that has support for what you are wanting to accomplish – there are many devices in the market now. Many of them share the same functionalities but they do have their differences. Make sure the device you start to build with supports everything you are trying to accomplish.
  2. Community support – When exploring new technologies, especially dealing with hardware and software, having good community support is extremely important. There are many little things that can go wrong or not work exactly the way you are hoping for. With good community support, it is easy to find documentation about something you are trying to accomplish or reach out to someone who has done something similar to your project in the past. Using these resources helps the project come together and minimize issues.
  3. Choose a language you are comfortable using – most devices now support a variety of languages. Chances are, there is a language supported that you are already familiar with. Some languages have full support for programming the device, some have their limitations. But at the end of the day, pick a language and framework and stick with it. You will lose a lot of time flip-flopping back and forth between languages and/or frameworks.
  4. Find a quick method of deployment – testing the application on the device is one of the most important parts of development. Most code can be developed locally on your computer and then pushed to the device. Set up continuous integration to speed up this process. That way, whenever a new feature is developed, it can be pushed to the device easily and available for testing quickly.
  5. Read the documentation – not everything is plug and play. There are some features that require special configurations either on the device or with breadboards. Everything used with these systems has a “manual” that gives you all this information. Spend the five extra minutes to read through the documentation to make sure everything works properly and none of the equipment gets damaged.

As software engineers, we were extremely curious about what hardware, software, devices, and systems we would discover throughout the development process. From using Python with the Debian OS to NodeJS with Resin.io, our research proved that there are many technical paths that lead to a solution. As J.A.R.V.I.S. continues to expand and become more intelligent, we hope to advocate the importance of IoT along with exploring many more problems to be solved.


Learn more about the power of a connected world using IoT.



Authors

This post was co-authored by Chris Becom and Eric Richardson, consultants in Cardinal's Raleigh App Dev practice.


Share:

No author identified.