DARPA’s Mind’s Eye project likely to advance AI

The DARPA “Mind’s Eye” program is another example of an ambitious AI program that is likely to get us closer to human-level AI. This program will be run out of DARPA's TCTO or Transformational Convergence Technology Officee.

The Mind’s Eye program --- to reach its goals --- has to be able to:

-have a fairly large invariant ontology of objects, motions, humans, weapons, military behaviors, scenes, and scenarios it recognizes in many different instantiations, forms, views, scale, and lighting;
-do visual scene recognition and understanding;
-understand behaviors of entities it is seeing;
-map such understandings into a larger higher level representation and understanding of what is taking place around it;
-presumably have to combine audio and visual recognition, since sound is an important source of information in a battlefield;
-have to have complex goal pursuit and attention focusing, to decide what to look at, and track, and spend its optical and computational resources on; and
-have natural language communication capabilities, or some other method of creating concise reports for human consumption and for receiving commands from humans

In sum, this project would require quite an advanced set of AI capabilities to function well.

The following is quoted from a short pdf at https://www.fbo.gov/download/ef9/ef9960d732bf796e6557916b4adf3ea9/DARPA_Minds_Eye_Industry_Day_Announcement_15March2010_(2).pdf , to spark interest for people to attend a meeting at which the project will be discussed in more detail. It does not appear the BAA for this project has been posted yet.

The Mind’s Eye program seeks to develop in machines a capability that currently exists only in animals: visual intelligence. Humans in particular perform a wide range of visual tasks with ease, which no current artificial intelligence can do in a robust way. Humans have inherently strong spatial judgment and are able to learn new spatiotemporal concepts directly from the visual experience. Humans can visualize scenes and objects, as well as the actions involving those objects. Humans possess a powerful ability to manipulate those imagined scenes mentally to solve problems. A machine‐based implementation of such abilities would be broadly applicable to a wide range of applications.

This program pursues the capability to learn generally applicable and generative representations of action between objects in a scene directly from visual inputs, and then reason over those learned representations. A key distinction between this research and the state of the art in machine vision is that the latter has made continual progress in recognizing a wide range of objects and their properties—what might be thought of as the nouns in the description of a scene. The focus of Mind’s Eye is to add the perceptual and cognitive underpinnings for recognizing and reasoning about the verbs in those scenes, enabling a more complete narrative of action in the visual experience.

One of the desired military capabilities resulting from this new form of visual intelligence is a smart camera, with sufficient visual intelligence that it can report on activity in an area of observation. A camera with this kind of visual intelligence could be employed as a payload on a broad range of persistent stare surveillance platforms, from fixed surveillance systems, which would conceivably benefit from abundant computing power, to camera‐equipped perch‐and‐stare micro air vehicles, which would impose extreme limitations on payload size and available computing power. For the purpose of this research, employment of this capability on man‐portable unmanned ground vehicles (UGVs) is assumed. This provides a reasonable yet challenging set of development constraints, along with the potential to transition the technology to an objective ground force capability.

Mind’s Eye strongly emphasizes fundamental research. It is expected that technology development teams will draw equally from the state of the art in cognitive systems, machine vision, and related fields to develop this new visual intelligence. To guide this transformative research toward operational benefits, the program will also feature flexible and opportunistic systems integration. This integration will leverage proven visual intelligence software to develop prototype smart cameras. Integrators will contribute an economical level of effort during the technology development phase, supporting participation in phase I program events (PI meetings, demonstrations, and evaluations) as well as development of detailed systems integration concepts that will be considered by DARPA at appropriate times for increased effort in phase II systems integration.