Reimagining the language faculty: A multimodal model of language

Natural human communication is multimodal. We pair speech with gestures and combine writing with pictures in contexts from doodles to emoji to comics to advertising. These integrated expressions raise questions as to whether the bodily or graphic modalities become “part of language” or are merely auxiliary but related systems. I will argue that accounting for this complexity warrants a reimagining of the architecture of language and communication itself. Rather than viewing these as indivisible modalities that combine, each of our modalities are merely parts of a single, holistic communicative architecture.  I will thus outline a reimagining of the language faculty that can account for both unimodal and multimodal behaviors, embedded within Jackendoff’s (2002) model of a Parallel Architecture. I will break down the primary modalities of spoken, bodily, and graphic communication into their constituent parts, and then show that the resulting interactions have systematic characterizations and profiles. Altogether, different human expressions — speech, gesture, drawings, and their multimodal interactions — arise as emergent activation states out of this holistic cognitive architecture. This model directly contests the traditional thinking of language as an “amodal” system, and necessitates a multimodal paradigm with consequences for all aspects of the linguistic system and its study.

Organisé par LINGUAE.

LINGUAE est une équipe de recherche de l' Institut Jean-Nicod et l'une des équipes qui composent l'équipe transversale DEC-Linguistics.