Project Team:
Hugo Liu, Henry Lieberman


Screen Shot
Video (Macromedia Flash)

Adaptive Linking between Text and Photos Using Common Sense Reasoning (PDF)

Robust Photo Retrieval Using World Semantics (PDF)

Semantic Understanding and Commonsense Reasoning in an Adaptive Photo Agent (PDF)


ARIA (Annotation and Retrieval Integration Agent) is a software agent that acts as an assistant to a user writing email or Web pages. As the user types a story, it does continuous retrieval and ranking on a photo database. It can use descriptions in the story text to semi-automatically annotate pictures based on how they are used.

The focus of this project is threefold: improving ARIA’s automatic annotation capabilities through world-aware semantic understanding of the text; making photo retrieval more robust by using a commonsense knowledge base, Open Mind Commonsense, to make semantic connections between the story text and annotations (e.g. connect "bride" and "wedding"); and learning personal commonsense through the text (e.g. “My sister’s name is Mary.”) that can then be used to improve photo retrieval by enabling personalized semantic connections.

ARIA with Common Sense is one of the first interactive applications being developed that uses broad commonsense knowledge about the world to visibly improve its agent behavior.

For more information email:



  Copyright © MIT Media Lab