Interaction-based Representation

Scientific Coordinators: Katharina Rohlfing, Frank Joublin

How does the interaction with the physical and social environment shape the buildup of mental representations during development?

Associated projects:

Biological organism have evolved in such a way that their behavior is simultaneously adapted to their needs and to the ecological niche where they live in. Depending on the organism these behaviors are shaped by phylogenetic search, by ontogenetic development and by learning. Each of these dynamic processes reflects an optimization towards a fitness function that is not fixed for ever but may be influenced by the environment or by intrinsic drives within the organism itself. Variations of the environment can be due to ecological changes, to the presence of predators, or to the interaction with conspecifics in the form of social hierarchies and more or less complex communication situations. Of particular interest for this research field is the shaping of the mental representations through various type of interactions during the ontogenetic process. These type of interactions define the range of knowledge an organism can learn about itself, and his environment. Each interaction is oriented from the organism to the outer world:

1) Interaction with oneself:
This type of interaction is the most primal one. It allows the organism to discover its embodiment ( i.e effectors, sensors ) but also some of its internal drives in the form of needs that have to be satisfied for its innate reward system. The representations that should develop during this type of interaction are based on the correlation between motor trial-and-errors production, reward and related perceptions.
Related research questions:

What are the extrinsic and intrinsic basic drives that lead an organism or a robot to start this kind of interaction?
Which sensory-motor mapping can be learned during this phase?
Which sensory-motor predictions can then be used by the organism and for which purpose?
Which innate mechanisms are responsible for selection processes such as attention?

2) Interaction with inanimates:
This type of interaction is restricted to the vicinity of the organism. It allows the discovery of basic aspects of the physics of the world ( i.e. contact, reaction forces, weight, gravity, sound of contact, sources of energy, sources of pain...) but also the discovery of new sensory-motor correlations (i.e. motor production + contact friction = locomotion, transport of objects, ... ). The representations developing within this interaction framework are control loops for the organism locomotion, the intake of energy, the manipulation of objects and representations for the classification of object affordances, quantities, self and object localization, memory of value related sources,...).
Related research questions:

What kind of genetically determined core representation should develop at this stage on the perception side and how do they interact with system's own behavior representations?
How should the system architecture a robot be organized to allow the memorization and the re-use of acquired sensory-motor knowledge for the fullfilment of the system needs?
How can this knowledge be used as a context for high cognitive processes such as the ability to see causal relationships?

3) Interaction with animates:
This type of interaction concerns environmental animate like conspecifics, siblings, but also prey or even predators. It allow the discovery of others autonomy, non-self-produced movements, non-self-produced sounds, new sources of energy (prey), competition, danger. The representations developing in this context are reactive control loops which may include predictive components (like tracking of animated), better perceptual classification, and even planning aspects (like internal strategy selection,...).
Related research questions:

Which principles underlie the transition from reactive to planned behaviors?
How does the organism represent itself versus others?
How are social cues and routine represented in order to provide facilitative environment for learning and communication?

4) Interaction with tutors:
This type of interaction differs from the previous one as it is mainly oriented towards the progress of the tutored organism. This interaction probably relies on specific internal drives and innate process that allow the use of communication first as a mean for remote action, then as a source of non-self experienced knowledge and as a way to share internal states reflecting either external situations or internal processes. It allows via the development of representations for shared attention, imitation, the discovery of the intentionality of others, and of the use of others as reference for the assessment of value for own behaviors.
Related research questions:

Are new drives necessary to develop social skills?
What are the cognitive control structures necessary to use the experience gained in imitation play for the system sake?
Which bootstapping process grounds the first communication behaviors?
How systematic / intelligent is the variability in of human behavior and what does it tell us about underlying action representations?

All these interaction types define new source of knowledge and reward. They all embed an exploration phase in which new knowledge about the environment is extracted to enrich the perceptual & behavioral means used by the organism to satisfy its drives.

The research in this field follows two main streams that strongly interact with each other: for the first stream, behavioral experiments concerning the understanding of the above interactions will be followed by a modeling phase that should allow the generation of new predictions but also deliver hypothesis on which the second stream could build. For the second stream, the construction of artifacts (algorithms implemented on specific embodiment like robot or simulation of robots) is in focus in order to understand the learning mechanism involved in the different interaction types from a computational perspective.