Almost all languages in the world have a way to formulate commands. Commands specify actions that the body should undertake (such as “stand up”), possibly involving other objects in the scene (such as “pick up the red block”). Action language involves various competences, in particular (i) the ability to perform an action and recognize which action has been performed by others (the so-called mirror problem), and (ii) the ability to identify which objects are to participate in the action (e.g. “the red block” in “pick up the red block”) and understand what role objects play, for example whether it is the agent or undergoer of the action, or the patient or target (as in “put the red block on top of the green one”). This chapter describes evolutionary language game experiments exploring how these competences originate, can be carried out and acquired, by real robots, using evolutionary language games and a whole systems approach.
This chapter introduces the computational infrastructure that is used to bridge the gap between results from sensorimotor processing and language. It consists of a system called Incremental Recruitment Language (IRL) that is able to configure a network of cognitive operations to achieve a particular communicative goal. IRL contains mechanisms for finding such networks, chunking subnetworks for more efficient later reuse, and completing partial networks (as possibly derived from incomplete or only partially understood sentences).
This chapter describes key aspects of a visual perception system as a key component for language game experiments on physical robots. The vision system is responsible for segmenting the continuous flow of incoming visual stimuli into segments and computing a variety of features for each segment. This happens by a combination of bottom-up way processing that work on the incoming signal and top-down processing based on expectations about what was seen before or objects stored in memory. This chapter consists of two parts. The first one is concerned with extracting and maintaining world models about spatial scenes, without any prior knowledge of the possible objects involved. The second part deals with the recognition of gestures and actions which establish the joint attention and pragmatic feedback that is an important aspect of language games.
This chapter explores a semantics-oriented approach to the origins of syntactic structure. It reports on preliminary experiments whereby speakers introduce hierarchical constructions and grammatical markers to express which conceptualization strategy hearers are supposed to invoke. This grammatical information helps hearers to avoid semantic ambiguity or errors in interpretation. A simulation study is performed for spatial grammar using robotic agents that play language games about objects in their shared world. The chapter uses a reconstruction of a fragment of German spatial language to identify the niche of spatial grammar, and then reports on acquisition and formation experiments in which agents seeded with a `pidgin German’ without grammar are made to interact until rudiments of hierarchical structure and grammatical marking emerge.
Grounding language in sensorimotor spaces is an important and difficult task. In order, for robots to be able to interpret and produce utterances about the real world, they have to link symbolic information to continuous perceptual spaces. This requires dealing with inherent vagueness, noise and differences in perspective in the perception of the real world. This paper presents two case studies for spatial language and quantification that show how cognitive operations – the building blocks of grounded procedural semantics – can be efficiently grounded in sensorimotor spaces.
Russian requires speakers of the language to conceptualize events using temporal language devices such as Aktionsarten and aspect, which relate to particular profiles and characteristics of events such as whether the event just started, whether it is ongoing or it is a repeated event. This chapter explores how such temporal features of events can be processed and learned by robots through grounded situated interactions. We use a whole systems approach, tightly integrating perception, conceptualization grammatical processing and learning and demonstrate how a system of Aktionsarten can be acquired.
Basic postures such as sit, stand and lie are ubiquitous in human interaction. In order to build robots that aid and support humans in their daily life, we need to understand how posture categories can be learned and recognized. This paper presents an unsupervised learning approach to posture recognition for a biped humanoid robot. The approach is based on Slow Feature Analysis (SFA), a biologically inspired algorithm for extracting slowly changing signals from signals varying on a fast time scale. Two experiments are carried out: First, we consider the problem of recognizing static postures in a multimodal sensory stream which consists of visual and proprioceptive stimuli. Secondly, we show how to extract a low-dimensional representation of the sensory state space which is suitable for posture recognition in a more complex setting. We point out that the beneficial performance of SFA in this task can be related to the fact that SFA computes manifolds which are used in robotics to model invariants in motion and behavior. Based on this insight, we also propose a method for using SFA components for guided exploration of the state space.
This chapter introduces the modular humanoid robot Myon, covering its mechatronical design, embedded low-level software, distributed processing architecture, and the complementary experimental environment. The Myon humanoid is the descendant of various robotic hardware platforms which have been built over the years and therefore combines the latest research results on the one hand, and the expertise of how a robot has to be built for experiments on embodiment and language evolution on the other hand. In contrast to many other platforms, the Myon humanoid can be used as a whole or in parts. Both the underlying architecture and the supportive application software allow for ad hoc changes in the experimental setup.
This chapter studies how basic spatial categories such as left-right, front-back, far-near or north-south can emerge in a population of robotic agents in co-evolution with terms that express these categories. It introduces various language strategies and tests them first in reconstructions of German spatial terms, then in acquisition experiments to demonstrate the adequacy of the strategy for learning these terms, and finally in language formation experiments showing how a spatial vocabulary and the concepts expressed by it can emerge in a population of embodied agents from scratch.
This chapter investigates how a vocabulary for talking about body actions can emerge in a population of grounded autonomous agents instantiated as humanoid robots. The agents play a Posture Game in which the speaker asks the hearer to take on a certain posture. The speaker either signals success if the hearer indeed performs an action to achieve the posture or he shows the posture himself so that the hearer can acquire the name. The challenge of emergent body language raises not only fundamental issues in how a perceptually grounded lexicon can arise in a population of autonomous agents but also more general questions of human cognition, in particular how agents can develop a body model and a mirror system so that they can recognize actions of others as being the same as their own.
This chapter explores a possible language strategy for verbalizing aspect: the encoding of Aktionsarten by means of morphological markers. Russian tense-aspect system is used as a model. We first operationalize this system and reconstruct the learning operators needed for acquiring it. Then we perform a first language formation experiment in which a novel system of Aktionsarten emerges and gets coordinated between the agents, driven by a need for higher expressivity.
All languages of the world have a way to talk about space and spatial relations of objects. Cross-culturally, immense variation in how people conceptualize space for language has been attested. Different spatial conceptualization strategies such as proximal, projective and absolute have been identified to underlie peoples conception of spatial reality. This paper argues that spatial conceptualization strategies are negotiated in a cultural process of linguistic selection. Conceptualization strategies originate in the cognitive capabilities of agents. The ecological conditions and the structure of the environment influence the conceptualization strategy agents invent and which corresponding system of lexicon and ontology of spatial relations is selected for. The validity of these claims is explored using populations of humanoid robots.
This chapter presents an operational grammar for German spatial language, in particular German locative phrases, as a case study for processing distributed information. It investigates the complex interplay of syntactic phenomena and spatial semantics, with a specific emphasis on efficient processing of syntactic indeterminacy and semantic ambiguity. Since FCG applies constructions in a sequence one after the other, the main challenge lies in mutual dependencies between constructions, that is, some constructions require pieces of information in order to make decisions that are only later on provided by other constructions. We present solutions and design patterns for dealing with these processing issues, which all have in common the strategy of postponing decisions as long as possible in processing until all the necessary information for making the decision is available.
This thesis contributes to our understanding of the origins of spatial language by carrying out language game experiments with artificial agents instantiated as humanoid robots. It tests the theory of language evolution by linguistic selection, which states that language emerges through a cultural process based on the recruitment of various cognitive capacities in the service of language. Agents generate possible paradigmatic choices in their language systems and explore different language strategies. Which ones survive and dominate depends on linguistic selection criteria, such as expressive adequacy with respect to the ecological challenges and conditions in the environment, minimization of cognitive effort, and communicative success. To anchor this case study in empirical phenomena, the thesis reconstructs the syntax and semantics of German spatial language, in particular German locative phrases. Syntactic processing is organized using Fluid Construction Grammar (FCG), a computational formalism for representing linguistic knowledge. For the semantics the thesis focusses in particular on proximal, projective and absolute spatial categories as well as perspective, perspective reversal and frame of reference. The semantic investigations use the perspective of Embodied Cognitive Semantics. The spatial semantics is grounded in the sensorimotor experiences of the robot and made compositional by using the Incremental Recruitment Language (IRL) developed for this purpose. The complete reconstructed system allows humanoid robots to communicate successfully and efficiently using the German locative system and provides a performance base line. The reconstruction shows that the computational formalisms, i.e. FCG and IRL, are sufficient for tackling complex natural language phenomena. Moreover, the reconstruction efforts reveal the tight interaction of syntax and semantics in German locative phrases. The second part of the thesis concentrates on the evolution of spatial language. First the focus is on the formation and acquisition of spatial language by proposing strategies in the form of invention, adoption, and alignment operators. The thesis shows the adequacy of these strategies in acquisition experiments in which some agents act as learners and others as tutors. It shows next in language formation experiments that these strategies are sufficient to allow a population to self-organize a spatial language system from scratch. The thesis continues by studying the origins and competition of language strategies. Different conceptual strategies are considered and studied systematically, particularly in relation to the properties of the environment, for example, whether a global landmark is available. Different linguistic strategies are studied as well, for instance, the problem of choosing a particular reference object on the scene can be solved by the invention of markers, which allows many different reference objects, or by converging to a standard single reference object, such as a global landmark. The thesis demonstrates that the theory of language evolution by linguistic selection leads to operational experiments in which artificial agents self-organize semantically rich and syntactically complex language. Moreover, many issues in cognitive science, ranging from perception and conceptualization to language processing, had to be dealt with to instantiate this theory, so that this thesis contributes not only to the study of language evolution but to the investigation of the cognitive bases of spatial language as well.
Artificial agents trying to achieve communicative goals in situated interactions in the real-world need powerful computational systems for conceptualizing their environment. In order to provide embodied artificial systems with rich semantics reminiscent of human language complexity, agents need ways of both conceptualizing complex compositional semantic structure and actively reconstructing semantic structure, due to uncertainty and ambiguity in transmission. Furthermore, the systems must be open-ended and adaptive and allow agents to adjust their semantic inventories in order to reach their goals. This paper presents recent progress in modeling open-ended, grounded semantics through a unified software system that addresses these problems.
Over the past several decades, psycholinguists have gained countless insights into the process of child language acquisition. Can these findings be used for the development of language competence in autonomous artificial systems? This paper reports on our attempt to apply insights from developmental psychology in order to enable artificial systems to acquire language. We consider a comprehensive chain of computational processes, starting from conceptualization and extending through language generation and interpretation, and show how they can be intertwined to allow for acquisition of complex aspects of grammar.
In this paper we test the dominant paradigm for modeling the semantics of determined noun phrases called Generalized Quantifier Theory in embodied interactions with robots. We contrast the traditional approach with a new approach, called Clustering Determination, which is heavily inspired by research on grounding of sensorimotor categories, and we show that our approach performs better in noisy, real world, referential communication.
This paper presents a software system that integrates different computational paradigms to solve cognitive tasks of different levels. The system has been employed to empower research on very different platforms ranging from simple two-wheeled structures with only a few cheap sensors, to complex two-legged humanoid robots, with many actuators, degrees of freedom and sensors. It is flexible and adjustable enough to be used in part or as a whole, to target different research domains
PROJECTS and questions, including Evolutionary Robotics, RoboCup and Artificial Language Evolution on Autonomous Robots (ALEAR, an EU funded cognitive systems project). In contrast to many other frameworks, the system is such that researchers can quickly adjust the system to different problems and platforms, while allowing maximum reuse of components and abstractions, separation of concerns and extensibility.
In this paper we demonstrate (1) how a group of embodied artificial agents can learn to construct abstract conceptual representations of body postures from their continuous sensorimotor interaction with the environment, (2) how they can metaphorically extend these bodily concepts to visual experiences of external objects and (3) how they can use their acquired embodied meanings for self-organizing a communication system about postures and objects. For this, we endow the agents with cognitive mechanisms and structures that are instantiations of specific ideas in cognitive linguistics (namely image schema theory) about how humans relate motor and visual space. We show that the agents are indeed able to perform well in the task and thus the experiment offers a concrete operationalization of these theories and increases their explanatory power.
Open-ended language communication remains an enormous challenge for autonomous robots. This paper argues that the notion of a language strategy is the appropriate vehicle for addressing this challenge. A language strategy packages all the procedures that are necessary for playing a language game. We present a specific example of a language strategy for playing an Action Game in which one robot asks another robot to take on a body posture (such as stand or sit), and show how it effectively allows a population of agents to self-organise a perceptually grounded ontology and a lexicon from scratch, without any human intervention. Next, we show how a new language strategy can arise by exaptation from an existing one, concretely, how the body posture strategy can be exapted to a strategy for playing language games about the spatial position of objects (as in ?the bottle stands on the table?).
Colour naming games are idealised communicative interactions within a population of artificial agents in which a speaker uses a single colour term to draw the attention of a hearer to a particular object in a shared context. Through a series of such games, a colour lexicon can be developed that is sufficiently shared to allow for successful communication, even when the agents start out without any predefined categories. In previous models of colour naming games, the shared context was typically artificially generated from a set of colour stimuli and both agents in the interaction perceive this environment in an identical way. In this paper, we investigate the dynamics of the colour naming game in a robotic setup in which humanoid robots perceive a set of colourful objects from their own perspective. We compare the resulting colour ontologies to those found in human languages and show how these ontologies reflect the environment in which they were developed.
Humans maintain a body image of themselves, which plays a central role in controlling bodily movement, planning action, recognising and naming actions performed by others, and requesting or executing commands. This paper explores through experiments with autonomous humanoid robots how such a body image could form. Robots play a situated embodied language game called the Action Game in which they ask each other to perform bodily actions. They start without any prior inventory of names, without categories for visually recognising body movements of others, and without knowing the relation between visual images of motor behaviors carried out by others and their own motor behaviors. Through diagnostic and repair strategies carried out within the context of action games, they progressively self-organise an effective lexicon as well as bi-directional mappings between the visual and the motor domain. The agents thus establish and continuously adapt networks linking perception, body representation, action, and language.