Overview

The overall objective of this experiment is to investigate how physically embodied autonomous agents can self-organize perceptually grounded lexicons and gammars for communicating about objects and events in their environment. In order to investigate this question, we conducted a series of experiments using SONY Aibo ERS7 robots. Building on software developed by the German Team for robotic soccer, these robots are equiped with a real-time vision system, probalistic object modeling techniques, object tracking, locomotion and obstacle avoidance.

The robots were programed to perform communicative tasks about objects and events in their immediate environment. They roam around, explore the space around them and try to reach a joint attentional frame, i.e. a situation in which a certain number of objects is in the view of both robots and both robots know the position of the other one. In each interaction, one of the robots is randomly assigned to be the speaker while the other one becomes the hearer. The speaker selects one of the objects that he sees and attempts to draw the attention of the hearer to this object through language. If the hearer understands the utterance correctly, it will point to the object that he understood. If not, or if the hearer points wrong, the speaker signals a failure and points to the intended object, allowing the hearer to learn.

Composite of various aspects of the perception system.

Experiments

Using this basic scenario (or language games), we investigate different types of communication systems:

• Names for Individual Objects: It has been established beyond doubt through computational simulations and mathematical proofs that a group of agents can autonomously self-organise a lexicon for naming individual objects. However all these simulations have assumed that the set of individual objects is known in advance and shared by all agents. We take away this scaffold and show how embodied agents can come up with names for individual objects
• Discriminative Categories: Although it already has been shown how agents can coevolve lexicons and inventories of discriminative categories ("left", "big", "red" etc.), we push this research forward by increasing the complexity of the world.
• Perspective. Since interlocutors can see a scene from arbitrary angles, the agents have to deal with the problem of perspective alignment in order to communicate successfully. We showed which cognitive mechanisms are necessary and sufficient to achieve successful spatial language, why perspective is essential for spatial language, how speaker and hearer could align perspective - even without marking-, and why and how perspective could become explicitly marked as part of spatial dialogue
• Spatial relations and landmarks. With an increasing complexity of the spatial scenes lexical communications systems don't suffice anymore. Using landmarks enhanced with visual patterns for easier recognition, we investigate how agents can come up with grammatical utterances such as "the green ball left of the red box"

Perception of landmarks using the ARToolKitPlus library.