IMMERSE

Overview

IMMERSE is a large DARPA funded project part of the Strategic Social Interaction Modules program. The program consists of three separate but collaborative branches of research related core issues of basic social interaction. These are typically interactions in which at least one party has a desired functional outcome, but may have little to no knowledge of the other party's culture or language.

The first branch is primarily focused on the social science of these types of dynamics trying to answer questions such as, why are some people more successful at this than others, what tends to frustrate efforts to achieve the desired goal, what leads to an escalation (and de-escalation) of force and what types of skills are universal across cultures to achieve success.

The second branch is primarily focused on building a social simulation engine to help teach players how to navigate these types of unfamiliar social situations more effectively. The simulation is embedded in a 3D virtual environment modeled in the Unity Game Engine. The social simulation is implemented by integrating a real-time behavior language with a rule-based artificial intelligence system specifically designed for turn- based social interactions. The player interacts with the environment and other embodied virtual agents through custom automated speech recognition and full body gestures via input from a Microsoft Kinect.

Pedagogical goals are embedded into each scenario, along with specialized pedagogical coaching through specialized virtual agents, to help focus the trainees attention on the most salient points in the simulation and to work on specific problem areas identified by the player model. By changing the parameters of the virtual agents new scenarios can be procedurally generated and require the player to use a different set of skills to achieve their goal.

The third branch is primarily focused on questions of evaluation. For example, what does it mean to improve social interaction skills, how do you know that it has occurred and does the simulation have a positive effect on players who interact with it.

Screenshot

Immerse Virtual Environment

An interaction with one of the virtual characters, while an activity continues to take place in the background.

Relevant Publications

Daniel Shapiro, Michael Mateas, Andrew Stern, Josh McCoy, Reid Swanson, Ben Samuel, and Mike Treanor. 2007. Creating Playable Social Experiences through Whole-body Interaction with Virtual Characters. In Ninth Annual AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, Boston, MA, October.