A Raytheon BBN-led team will develop an artificial intelligence tool, Medical Assistance, Guidance, Instruction and Correction, or MAGIC, to assist combat field medics in 50 medical skills applied during common procedures. The medical assistant is being developed under a DARPA award.

To train the AI, the team will collect and label combat medical data sets comprising:

• more than 2,500 videos recorded in stereo to provide 3D data.
• nearly 50 million images to identify the correct sequence of procedures.

The tool will use DARPA Perceptually-enabled Task Guidance program’s consumer off-the-shelf augmented reality goggles, which will harness the AI to use visual and audible sensors that track tasks and guide field medics in performing procedures correctly and efficiently. The built-in cameras of the AR goggles observe the user's actions, and AI verifies that steps are followed correctly, providing assistance—on request or when intervention is needed—in the form of auditory suggestions or visual overlays in the headset. Those prompts can include next steps, follow-on actions, or critical measurements such as elapsed time or dosing.

“The combat medical environment is challenging and chaotic,” explained Brian VanVoorst, Raytheon BBN scientist. “Our goal for the Raytheon BBN MAGIC AI tool is to help support personnel to provide guidance as needed without disrupting their concentration.”

Raytheon-BBN is teamed with Valkyries Austere Medical Solutions for this work.

The first demonstration of the new tool is planned in 18 months. Training data collected for the program will be shared with the research community.