The Need

Robot operators still rely on cumbersome controls to teleoperate unmanned assets. This approach to directing robots is in stark contrast to the ease with which human teammates use speech and gestures to seamlessly interact with each other. 


The Solution

We developed a natural and reliable communication method that enables Commanders to issue instructions to unmanned vehicles (UxVs) using a human-like approach. SHEPARD fuses multiple proven robot control technologies to provide a hands-free human-machine interface (HMI) for operators in diverse environments.


The Benefit

SHEPARD increases soldier effectiveness by reducing the cognitive burden associated with interacting with their new robotic teammates.  Further, it accelerates the adoption of UxVs through the increased trust of their operators.


SHEPARD uses smart devices—like a watch—to make communication with military robots easy


“Our approach to managing UxVs is in stark contrast with how robot operators currently direct unmanned platforms—they use cumbersome, hands-on, head-down controllers. Our goal with SHEPARD is to develop controls that let robots seamlessly integrate into human teams.” - Camille Monnier, SHEPARD Principal Investigator

Unmanned vehicles (UxVs) are vital to military operations, although their widespread integration faces significant challenges. Current control schemes require a trained operator to remotely control an unmanned system to execute a Commander’s orders. This increases the complexity of the command structure, introduing latency and reducing situational awareness.

Developing an Intuitive Interface for the US Army

In response to the US Army’s need for advanced, intuitive interfaces, we designed and prototyped a system for Supervisory HMI Enabling Practical Autonomous Robot Direction (SHEPARD). Our human-machine interface (HMI) consists of a portable software package and a modular hardware kit, including a camera and a lightweight, wearable operator control unit.

SHEPARD uses smart devices—like a watch—for easy communication with military robots. We developed a context-sensitive control module that enables natural, reliable control of multiple platforms by fusing multi-modal inputs, such as voice and gestures, with operational context.

Effortless Human-Robot Communication

By removing the need for a Soldier to teleoperate an unmanned system, SHEPARD allows the robotic platform to become a true support agent. SHEPARD’s hands-free HMI includes a context-sensitive control module based on a model of operator intent. By fusing context cues with inputs provided from multiple control modalities (e.g., voice, gesture), the system infers a model of operator intent that can be used to resolve ambiguous commands and provide situation-appropriate feedback. SHEPARD supports several gesture control configurations to make human-machine communication more natural than ever.


Clearpath Husky platform responding to gesture and voice commands


Related Work

MANTA - a platform-independent, natural control and autonomy robot appliqué that enables a user to easily direct one or more host platforms to perform a range of autonomous behaviors.

Contact Us