NIST can benefit by following IEEE standards such as Distributed Interactive Simulation (DIS) and high-level architecture (HLA) established by the Department of Defense (DoD) and NASA in their development of tools such as the Trick Simulation Toolkit and distributed simulated environments such as SIMNET and the close combat tactical trainer (CCTT). Defining a high-level architecture which establishes a baseline for interaction modalities and biometric feedback datasets provides a path to creating key performance indicators (KPI) which can help measure system performance and define future enhancements. Initially, we recommend using off-the-shelf roomscale virtual reality (VR) hardware like the HTC Vive as well as augmented reality head-mounted displays such as the Microsoft Hololens. HTC has released positional tracking hardware which can be attached to object “blanks” placed in training spaces to represent a wide variety of real-world objects. As KPIs are refined, more exotic hardware such as Intel Project Alloy may be used to augment simulation and after-action review capability.
The key is to establish user and system performance baselines during system design, alpha, and beta testing. Like all large-scale distributed software systems, there are inherent risks which may affect the efficacy of the system upon its initial release. Risks such as creating unbalanced gameplay can be mitigated by leveraging lessons learned from DoD initiatives and the commercial video games industry. Multi-million dollar games regularly roll out game balancing updates developed to tune game rules, missions, and system performance. Regular measurement of user outcomes as well as system performance allows systems administrators to tune gameplay, in this case training curricula, appropriately. Having accurate real-time data which represents the current state of performance is vital. Games such as the widely acclaimed Uncharted series, for example, benefit from real-time analytics created in partnership with middleware providers who develop infrastructure specifically for the purposes of optimizing online multi-user roleplay. We recommend a client-server architecture which mimics modern games. Until a proper safety standard is established and measured in regards to the virtual environment, we recommend that potentially dangerous environments which contain smoke and flames should be simulated with a game engine.
Assuming a high-level architecture has been developed and recommendations for hardware implementation and user experiences have been thoroughly tested, replication would include the standup of the server and network infrastructure, construction of the training facilities, deployment of users and groups within the system, administrative training and documentation deployment, testing of the integrated system, and the creation of a deployment report.
System implementation requires an extensive review of user needs and testing of user interactions. The system would be designed to meet all NIST safety standards as defined by a requirements document and informed during alpha and beta testing. No system is fool-proof, however, with enough data one can make informed decisions to mitigate safety risks and develop mechanisms for expedient reporting of potential hazards.
System design would include photogrammetry-derived environments. Photogrammetry is used in the development of special effects and virtual environments for major motion pictures as well as AAA games. Use of spatial audio further enhances user presence and would be used in conjunction with photo-realistic 3D models throughout the simulation. Multi-user interactions and real-time feedback would include interface elements derived from real-world biometric feedback such as heart rate monitors, voice volume analysis, hand movement, eye tracking, and limb analysis deriving posture and gait.
Position tracking devices, acoustic sensors, pressure sensors, and virtual chaperone elements would increase user immersion and provide a bridge to real-world objects and devices. An API is required to interface and measure the efficacy of these real-world objects as well as inform developers on end-use to improve in-game fidelity. Whenever possible, augmented reality would be used with real-world devices such that “classroom time” is nearly indistinguishable from real-world use.
NIST requires a variety of technologies to be tested. As noted previously, we recommend a high-level architecture whose focus is on interoperability of system components. This includes the development of an API which interfaces with all technologies being tested. For example, NIST requires communication between users and non-playable characters (NPCs). We recommend the inclusion of Microsoft Cognitive Services or similar off-the-shelf libraries for interaction with NPCs. NPCs would have access to the cognitive services component of the API and thus be able to naturally interact with human end-users. Additionally, in-game smart phones, walkie-talkies, and note taking devices should be incorporated through virtualization, known tools would require API hooks and could thus be measured and enhanced by the system. These known tools would be precursors to future interfaces and tools which NIST will be able to identify through emerging behavior in the system.
HTC Vive (per unit) - $799
Vive Tracker (per unit) - $99
Microsoft Hololens (per unit) - $5,000
Backend Server Architecture
Assorted Sensors
Single and multi-user training can occur through curriculum selection in an in-game lobby. This type of system architecture allows for users at assigned locations as well as off-site locations to train cooperatively or singularly in the comfort of their own homes. For example, DARPA’s SIMNET was used in training facilities primarily in Fort Benning, Georgia as well as Fort Rucker, Fort Knox, Fort Leavenworth, and Grafenwoehr, Germany. SIMNET’s successor CCTT uses various-sized locations including mobiles sites created in shipping containers.
An exercise would be preconfigured by an administrator using modular components. Administrator interfaces would be available for sandboxing available components to meet the training needs defined in Example First Responder Situations provided by NIST. All scenarios which NIST seeks training in include a mixture of non-playable characters, vehicles, terrain, residential and commercial architecture, roads, streetlights, and signs. Developing a system around interactive components which feed a wide variety of scenarios allows for a near-unlimited scenario generation. An analogy would be sandbox-style games such as the Grand Theft Auto series which provide best-of-breed examples for living worlds in which scenarios can be applied.
We suggest developing user and system performance baselines during beta testing. Like all large-scale distributed software systems, there are inherent risks of creating an imperfect system upon initial release. Risks such as unbalanced gameplay can be mitigated by leveraging lessons learned from DoD initiatives and the commercial video games industry. Multi-million dollar games regularly roll out game balancing updates developed to tune game rules, missions, and system performance. The key is to regularly measure user outcomes as well as system performance and tune appropriately. Constant analysis with refinement of the baseline for each expected outcome in the system will not only allow rapid correction of system processes but will give administrators accurate representation of the current state of performance. Games such as the widely acclaimed Uncharted series, for example, benefit from real-time analytics created in partnership with middleware providers who develop infrastructure specifically for the purposes of optimizing online multi-user roleplay. We suggest consulting middleware companies whose tools are appropriate to NIST’s needs. Additionally, repurposing game analytics tools for training would reduce the risk inherent in creating new measurement apparatuses. Such custom tools be required, stablished baselines would inform the development of those measurement assets.
