Multimodal Recognition of Manual Interaction

Project Leader: Michael Pardowitz, Robert Haschke, Volker Willert
PhD Student: Alexandra Barchunova

A robot has to learn to carry out and to recognize goal-oriented arm or hand movements. This can be achieved by acquiring essential episodes of manual interaction from a human teacher. Pushing, grasping or lifting an object commonly serve as an example for such episodes, also called interaction patterns. Hereby we address the challenging question of how to perceive manual interaction in a way that generalizes over a wide variety of objects. It is therefore fundamental to establish a representation that is to a high degree invariant w.r.t. the involved object(s). 

The challenging practical goal of this project is to build a system which implements the initial part of a route towards manual action capture. This implies a reliable recognition of a manageable set of basic manual interaction patterns that involve a large set of different objects. There is a number of approaches to be considered. One approach could be based on an implementation of a suitable recognizer for human interaction patterns with a small number of reference objects. If necessary it might as well include some generalization mechanism to achieve object invariance. Another possibility could consist of training a recognizer from video sequences and sensor readings of self-generated interaction patterns of a humanoid robot hand. This approach might require an additional mechanism for a representation of a human hand.

Initially, the recognition will be based upon various sensory channels provided from an Interaction Database: VICON-based geometry information, video sequences, joint angle trajectories from data gloves, and tactile data. Nevertheless the final goal is to build a vision-based system, which can use low-cost and easy-to-operate data acquisition devices.