BLOOMSBURY SEMIOTICS

VOLUME 4
Bloomsbury Semiotics
General Editor: Jamin Pelkey

Volume 1: History and Semiosis

Edited by Jamin Pelkey

Volume 2: Semiotics in the Natural and Technical Sciences

Edited by Jamin Pelkey and Stéphanie Walsh Matthews

Volume 3: Semiotics in the Arts and Social Sciences

Edited by Jamin Pelkey, Susan Petrilli and Sophia Melanson Ricciardone

Volume 4: Semiotic Movements

Edited by Jamin Pelkey and Paul Cobley
BLOOMSBURY SEMIOTICS

SEMIOTIC
MOVEMENTS
VOLUME 4

Edited by Jamin Pelkey and Paul Cobley

 BLOOMSBURY ACADEMIC
Bloomsbury Publishing Plc
50 Bedford Square, London, WC1B 3DP, UK
1385 Broadway, New York, NY 10018, USA
29 Earlsfort Terrace, Dublin 2, Ireland

BLOOMSBURY, BLOOMSBURY ACADEMIC and the Diana logo

are trademarks of Bloomsbury Publishing Plc

First published in Great Britain 2022

Copyright © Bloomsbury Publishing Plc, 2022

Jamin Pelkey and Paul Cobley have asserted their right under the Copyright,
Designs and Patents Act, 1988, to be identified as Editors of this work.

For legal purposes the Acknowledgements on p. xvii constitute

an extension of this copyright page.

Cover design: Tjaša Krivec

Cover illustration by Rebecca Heselton

All rights reserved. No part of this publication may be reproduced or transmitted in

any form or by any means, electronic or mechanical, including photocopying, recording, or any
information storage or retrieval system, without prior permission in writing from the publishers.

Bloomsbury Publishing Plc does not have any control over, or responsibility for, any third-party
websites referred to or in this book. All internet addresses given in this book were correct at the
time of going to press. The author and publisher regret any inconvenience caused if addresses have
changed or sites have ceased to exist, but can accept no responsibility for any such changes.

A catalogue record for this book is available from the British Library.

A catalog record for this book is available from the Library of Congress.

ISBN: HB: 978-1-3501-3940-4

ePDF: 978-1-3501-3942-8
eBook: 978-1-3501-3941-1
Set: 978-1-3501-3944-2

Series: Bloomsbury Semiotics

Typeset by Integra Software Services Pvt. Ltd.

To find out more about our authors and books visit www.bloomsbury.com
and sign up for our newsletters.­
 CONTENTS

L ist of F igures vii

L ist of T ables x
L ist of C ontributors xi
A cknowledgements xvii
L ist of A bbreviations xviii

Introduction 1
Paul Cobley

1 Communication Theory and Semiotics 15

Richard L. Lanigan

2 Media/Culture Studies and Semiotics 47

Sophia Melanson Ricciardone and Marcel Danesi

3 Digital Humanities and Semiotics 67

Alin Olteanu and Arianna Ciula

4 Systems Theory and Semiotics 87

Ricardo Gudwin and João Queiroz

5 Phenomenology and Semiotics 103

Peer F. Bundgaard

6 Hermeneutics and Semiotics 123

Ronald C. Arnett and Susan Mancino

7 Translation Studies and Semiotics 143

Evangelos Kourdis and Ritva Hartama-Heinonen

8 Pragmatics and Semiotics 163

Per Aage Brandt

9 Gesture Studies and Semiotics 183

Irene Mittelberg and Jennifer Hinnell

10 Multimodality and Semiotics 215

David Machin and Ariel Chen

11 Discourse Analysis and Semiotics 231

Kay L. O’Halloran and Sabine Tan
vi CONTENTS

12 Integrational Linguistics and Semiotics 253

Adrian Pablé

13 Cognitive Linguistics and Semiotics 269

Jordan Zlatev and Möttönen Tapani

14 Cognitive Science and Semiotics 293

Göran Sonesson

I ndex 313
 LIST OF FIGURES

1.1 Ruesch and Bateson discourse model of communicology 18

1.2 Category description of Ruesch and Bateson model 19

1.3 Discourse communicology model 20

1.4 Complexity description of semiotic systems 21

1.5 Etymology of C. S. Peirce logic typology 22

1.6 Scholastic trivium discourse model 24

1.7 Roman Jakobson communicology theory and applied model 26

1.8 Jakobson communicology process model 28

1.9 Jakobson chiasm matrix model 29

1.10 Semiotic phenomenology matrix model 30

1.11 Communicology matrix of elements and functions 31

1.12 Matrix of the logic and semiotic squares 34

1.13 Chiasm process matrix for apperception and apposition 35

1.14 Trope logic in rhetoric and semiotic 36

1.15 Communicology matrix of intelligibility and relevance 38

1.16 Discourse trope model of communicology 39

2.1 Two-step flow model 54

8.1 The complex functions of language 165

8.2 The rhetoric of image and text 168

­8.3 The semiotic constitution of power through enunciation 170

8.4 A sociogram example 171

8.5 The complex semiosis of music 172

8.6a The blending model 174

viii LIST OF FIGURES

8.6b Metaphor 175

8.6c Metonymy 175

8.7a The schema: ‘the road not taken’ 176

8.7b Irony 177

9.1 A seventeenth-century illustration of a gesture signalling antithesis 184

9.2 Thumbs-up emblem signalling approval 189

9.3 Pointing gesture (‘there is’) and metaphoric container gesture

(‘the main verb’) 191

9.4 Painting: P. Klee, Dance of a Mourning Child 1922 197

9.5 Motion-capture gesture diagram of travel itinerary: Stockholm,

Prague, Kiev, etc. 198

9.6 Metonymy: contiguity relations between hands and the virtual object
they seem to be holding. Metaphor: Imaginary object represents the
abstract entity (‘subcategory’) 200

9.7 Embodied contrast: On the one hand/on the other hand, bilateral
point sequence 203

10.1 UFIT protein shake and Slimfast protein shake 216

10.2 SCI-MX protein 225

10.3 Bounce protein energy ball 226

11.1 Multimodal analysis of Just Terror movie poster 236

11.2 Resemiotizations and intertextuality of Just Terror tactics and knife

attacks in Dabiq and Rumiyah magazines 240

11.3a Just Terror article: Image and text 243

11.3b Language analysis for Just Terror article 244

11.3c Image analysis for Just Terror article 245

11.4 K-Modes cluster visualization 245

11.5 Clusters with image categories, Watson categories and website

taxonomy and tonality 246

13.1 Meaning as construal, with some of the relevant structures 272

13.2 Diagrammatic iconicity between embodied and linguistic distance in

Paamese possessive 277
LIST OF FIGURES ix

13.3 Greece Flag ≡ Toilet paper, leading to the metaphorical interpretation

GREECE IS SULLIED, given appropriate knowledge on the
Sedimented and Situated levels 281

13.4 The Motivation & Sedimentation Model (MSM) with the upward
arrows signifying motivation, and downward arrow sedimentation 282
 LIST OF TABLES

11.1 Semiotic concepts for MDA 234

11.2 Interpretations of Just Terror 237

11.3 Bonding icons in violent extremist terrorist communication 239

11.4 MDA framework and IBM Watson and Clarifai algorithms 247

13.1 The sign systems of Language, Gesture and Depiction, with some of
their properties 285

14.1 The relations between principles, grounds and signs in the present
interpretation of Peirce 302

14.2 ‘The semiotic hierarchy’: Five types of selves, worlds and significations 304

14.3 The distinction of ipseity, dialogicity and neutrality in the phenomena

accessed and the modes of access 307
 LIST OF CONTRIBUTORS

Ronald C. Arnett is Chair and Professor in the Department of Communication and

Rhetorical Studies, the Patricia Doherty Yoder and Ronald Wolfe Endowed Chair in
Communication Ethics, and the Henry Koren, C.S.Sp., Endowed Chair for Scholarly
Excellence (2010–15) at Duquesne University, Pittsburgh, Pennsylvania, USA. He has
received eight book awards, co-edited seven books and authored/co-authored twelve
books, most recently Communication Ethics and Tenacious Hope: Contemporary
Implications of the Scottish Enlightenment (2022, Southern Illinois University Press). He
was named a Distinguished Scholar by the National Communication Association and is
currently President of the Semiotic Society of America.

Per Aage Brandt is Full Professor in the Department of Cognitive Science at Case
Western Reserve University, Cleveland, Ohio, USA. He is the author of a dozen books
and more than 150 published papers on cognitive and semiotic theory of language,
grammar, aesthetics, art and music. As a scholar trained in romance philology (French
and Spanish), he has worked his way through structural linguistics and structural
semantics, and elaborated a series of models – in particular those related to the
technical and formal representations of textual phenomena such as enunciation, diegesis
and modal schematisms – for describing patterns of meaning in the framework of a
discourse-oriented (Greimas) and later a formalized phenomenological (Thom, Petitot)
and cognitively (Talmy) oriented semiotics. In 2002, he was awarded the Grand Prix de
Philosophie by l’Académie française and was made Officier de l’Ordre des Arts et des
Lettres by the French Ministry of Culture.

Peer F. Bundgaard is Associate Professor in the Center for Semiotics, School of

Communication and Culture at Aarhus University, Denmark. He holds a PhD in general
semiotics. He is editor-in-chief of Cognitive Semiotics. His research interests are cognitive
linguistics and phenomenology as well as the cognitive semiotics of literary and visual art
and aesthetic experience. He has published articles in Synthese, Phenomenology and the
Cognitive Sciences and Semiotica. He has edited Investigations into the Phenomenology
and the Ontology of the Work of Art, Springer, open access (with Frederik Stjernfelt), and
Semiotics (4 vols), Routledge (with Frederik Stjernfelt).

Ariel Chen (PhD Cardiff University) is a researcher of media and communication studies at
Örebro University. Her research interests include social semiotics, visual communication,
food and health culture, and global media. She is currently working on a research project
entitled “Communication of ‘Good’ Foods and Healthy Lifestyles”. Using Multimodal
Critical Discourse Analysis, she examines the way the discourse of ‘good’ foods and
healthy lifestyles becomes colonized and shaped by commercial interests. Her recent
articles can be seen in Critical Discourse Studies; Discourse, Context and Media; and
Food, Culture and Society.
xii LIST OF CONTRIBUTORS

Arianna Ciula is Deputy Director and Senior Analyst in the King’s Digital Lab at King’s
College London, UK. She has broad experience in digital humanities research and
teaching, research management and digital research infrastructures. She holds a PhD in
manuscript and book studies. Her personal research interests focus on the modelling
of scholarly digital resources related to primary sources. She lectured and published on
humanities computing, in particular on digital manuscript studies and editing; she has
organized conferences and workshops in digital humanities and is an active member of its
international community.

­ aul Cobley is Full Professor and Deputy Dean (Research and Knowledge Exchange)
P
in the Faculty of Arts and Creative Industries, Middlesex University, London, UK.
He is the author of a number of books, including Cultural Implications of Semiotics
(2016) and Narrative, 2nd edn (2014). He is co-series editor (with Kalevi Kull) of
Biosemiotics, Communication and Cognition (de Gruyter Mouton) and co-editor
(with Peter J. Schulz) of the multi-volume Handbooks of Communication Sciences
(de Gruyter) and (with David Machin) the journal Social Semiotics. Among his edited
volumes are The Routledge Companion to Semiotics (2009), Theories and Models of
Communication (2013, with Peter J. Schulz), Semiotics and Its Masters vol. 1 (2017,
with Kristian Bankov), Realism for the 21st Century: A John Deely Reader (2009) and
The Communication Theory Reader (1996). He is the 9th Thomas A. Sebeok Fellow
of the Semiotic Society of America, President of the International Association for
Semiotic Studies (elected 2014) and secretary (since 2012) of the International Society
for Biosemiotic Studies.

Marcel Danesi is Professor Emeritus in the Department of Anthropology and ex-director

of the Program in Semiotics at Victoria College, University of Toronto, Canada. He was
editor of Semiotica from 2004 to 2019. Among his recent works are Semiotics of Emoji
(2016), Memes and the Future of Popular Culture (2019), Semiotics of Love (2019) and
Linguistic Relativity Today (2021). He is also a co-founder and currently co-director
of the Cognitive Science Network of the Fields Institute for Research in Mathematical
Sciences.

Ricardo Gudwin is Associate Professor at DCA-FEEC-UNICAMP, the School of

Electrical and Computer Engineering, University of Campinas, Brazil. He received a
BS degree in electrical engineering in 1989, an MS degree in electrical engineering
in 1992 and a PhD in electrical engineering in 1996, all of them from UNICAMP.
His current research interests include the study of cognitive architectures, cognitive
systems, intelligence and intelligent systems, intelligent agents, (Peircean) semiotics
and computational semiotics. He co-edited the books Artificial Cognition Systems
(2006), Semiotics and Intelligent Systems Development (2006), Computação, Cognição,
Semiose (2007) and Systems, Self-Organisation and Information – An Interdisciplinary
Perspective (2018). He was the editor-in-chief of the journal Controle & Automação,
published by the SBA, the Brazilian Society for Automation, from September 2004 to
December 2008, and the vice-president of SBCC, the Brazilian Society for Cognitive
Science, from 2013 to 2017.

Ritva Hartama-Heinonen is Senior Lecturer in Swedish Translation and Interpreting

Studies in the Department of Finnish, Finno-Ugrian and Scandinavian Studies at the
LIST OF CONTRIBUTORS xiii

University of Helsinki, Finland. She also holds the title of Docent in Translation Studies
at the same university. After finishing her doctoral dissertation Abductive Translation
Studies: The Art of Marshalling Signs (2008), a combination of translation studies and
Peirce’s semeiotic, she has continued to contribute to the semiotics of translation. In
addition, she has published widely on translation-theoretical issues, for instance, by
developing the notion of intracultural translation. Her recent research interests include
intralingual and interlingual translation into Easy Language.

Jennifer Hinnell is a Postdoctoral Research Fellow at the University of British

Columbia, Vancouver, Canada. As a cognitive linguist, her research applies corpus,
motion capture and experimental methods to the investigation of co-speech
gestures. She has (co-)published journal articles and book chapters in Language and
Cognition, Cognitive Linguistics, The American Journal of Semiotics and Frontiers
in Psychology – Cognition, among other venues, and co-edited Language and the
Creative Mind (2013). Jennifer has been an invited speaker in the United States,
Asia and Europe. Awarded the Governor General’s Gold Medal for her research in
2020, Jennifer holds an MA in linguistics from Simon Fraser University and a PhD
in linguistics from the University of Alberta. She is a member of the international
collaborative multimodality lab ‘Red Hen’ and Emerging Scholars Representative of
the International Cognitive Linguistics Association.

Evangelos Kourdis is Professor of Translation Semiotics in the Department of French

Language and Literature, Faculty of Philosophy, Aristotle University of Thessaloniki. His
main research interests include translation and semiotic theory, intersemiosis, semiotic
landscape, language ideology, social dialectology, cultural adaptation and cultural
communication. He is Director of the Joint Master Semiotics, Culture and Communication,
Aristotle University of Thessaloniki, and Director of the Laboratory of Semiotics (AUTH
SemioLab) of the Faculty of Philosophy, Aristotle University of Thessaloniki. He is also
the National Representative of Greece at the International Association for Semiotic
Studies and Vice-President of the Hellenic Semiotic Society.

Richard L. Lanigan is Executive Director and Laureate Fellow, International

Communicology Institute, Washington, DC, USA, and University Distinguished Scholar
and Professor of Communicology (Emeritus), School of Communication, Southern
Illinois University. He is Past President, Semiotic Society of America; Editor, The
American Journal of Semiotics (ten years); Past Vice-President, International Association
for Semiotic Studies; Senior Fulbright Fellow (China 1996, Canada 2007); Fellow of the
International Academy for Intercultural Research; Fellow, Polish Academy of Science
(Philology); ‘International Scholar Award in Philosophy of Communication’, Duquesne
University, USA (2011); elected member, American Philosophical Association; founding
member, Philosophy of Communication Division, National Communication Association
(USA) and International Communication Association. Festschrift: Communicology for
the Human Sciences: Lanigan and the Philosophy of Communication, ed. A. R. Smith, I.
E. Catt and I. E. Klyukanov (New York: Peter Lang, 2018). PDF publications: Academia.
edu/Richard L. Lanigan.

David Machin is Full Professor in the School of International Studies at Zhejiang

University, China. He is author of a number of books in the field of critical discourse
xiv LIST OF CONTRIBUTORS

studies, such as Doing Visual Analysis (Sage, 2018), Introduction to Multimodal

Analysis (Bloomsbury, 2020) and Visual Journalism (Palgrave, 2015). He has published
many papers in leading journals in the field. He is co-editor of the Taylor & Francis
journal Social Semiotics and the Bloomsbury book series Advanced in Critical Discourse
Studies.

Susan Mancino is Assistant Professor in the Department of Communication Studies,

Dance, and Theatre at Saint Mary’s College, Notre Dame, IN, USA. She is the co-editor of
Communication and Learning in an Age of Digital Transformation and An Encyclopedia of
Communication Ethics: Goods in Contention. Additionally, she is the author/co-author
of ten articles and nine book chapters. Her works have appeared in Versus, The Atlantic
Journal of Communication, Review of Communication and Empedocles: European Journal
for the Philosophy of Communication. Her research interests include semiotics, public
memory, religious communication, philosophy of communication and communication
ethics.

S­ ophia Melanson Ricciardone is a PhD candidate (ABD) with the joint programme of
Communication and Culture with York and Toronto Metropolitan Universities in
Toronto, Ontario, Canada. She is a PhD research assistant with Toronto Metropolitan
University’s Meaning Lab, which is dedicated to research into various facets of language
and culture and cognition. Broadly, her work examines the ways in which we offload
facets of cognition to digital technology, which invariably affects how we collectively
use language, how we reason through complex ideas about the world and how patterns
of thought (mental schemas) are altered in the process. She is currently working on a
research project empirically evaluating the impact of Twitter-bots on intersubjective
discourse. More specifically, she is exploring whether and to what extent bot-generated
contributions to political Twitter discourse stimulate the circulation of affectively charged
content and what consequences such arrangements impart on minds and bodies.

Irene Mittelberg is Full Professor of Linguistics and Cognitive Semiotics at the Institute
of English Studies of RWTH Aachen University (Germany), where she directs the Natural
Media Lab and the Centre for Sign Language and Gesture. She holds a PhD in linguistics
and cognitive studies from Cornell University. Her work combines semiotic theory
(e.g. Peirce, Jakobson) with embodied approaches to language, cognition, multimodal
interaction and the visual arts. Her recent interdisciplinary work includes the adoption
of Peirce’s universal categories for neuroscientific research into gesture. She wrote a
monograph on Metaphor and Metonymy in Language and Gesture (2006), co-edited
Methods in Cognitive Linguistics (2007) and has (co-)published over sixty journal articles/
book chapters. She serves on the board of the International Association for Cognitive
Semiotics (IACS) and the German Association for Semiotics (DGS).

Tapani Möttönen is a postdoctoral fellow of Finnish language in the Department of

Finnish, Finno-Ugrian and Scandinavian Studies at the University of Helsinki, Finland.
His general research interests include metatheory and methodological development of
linguistics and semantics, and his PhD research focused on the ontology of meaning
in cognitive grammar. He currently works on adapting methodology from cognitive
linguistics to the analysis of spoken interaction as a part of industrial design processes.
LIST OF CONTRIBUTORS xv

Kay L. O’Halloran is Chair Professor and Head of the Department of Communication

and Media in the School of the Arts at the University of Liverpool. Her research area
is multimodal analysis, involving the study of the interaction of language with other
resources in texts, interactions and events. A key focus of her work is the development of
digital tools and techniques for multimodal analysis, as well as mixed methods approaches
that combine multimodal analysis, data mining and visualization for big data analytics
within and across different media platforms.

Alin Olteanu is Postdoctoral Researcher and Publications Coordinator at the Käte

Hamburger Kolleg Cultures of Research of RWTH Aachen University, Germany.
Having worked on the dialogue between semiotic theories and educational philosophy,
Alin is currently interested in the potential of biosemiotics to contribute to society and
technology studies, with a focus on digitalization and sustainability.

Adrian Pablé is Associate Professor in the School of English at the University of Hong Kong
and Professor Extraordinarius at the University of South Africa (Pretoria). He is a Roy
Harris scholar and an integrational linguist, with an interest in the history of linguistics
and in ideologies of language. He is the secretary of the International Association of the
Integrational Study of Language and Communication (IAISLC) and the series editor of
the Routledge Advances in Communication and Linguistic Theory. His recent publications
include the monograph Signs, Meaning and Experience (de Gruyter, 2015, with Chris
Hutton) and the edited volume Critical Humanist Perspectives (Routledge, 2017). He has
published several articles on the relationship between semiotics and semiology, and he is
currently applying an integrationist critique to the posthumanist turn in linguistics.

João Queiroz is Professor at the Institute of Arts, Federal University of Juiz de Fora,
Brazil, and in the Postgraduate Program in Communication at the same institution, where
he coordinates the Iconicity Research Group (IRG). He has been teaching courses on
cognitive semiotics and intermediality studies. He has supervised PhD and master’s students
in the fields of semiotics, Latin-American art and literature and cognitive aesthetics. He
has several publications in international journals, books and conferences, including the
Commens Digital Companion to Charles S. Peirce (co-organized with Mats Bergman
and Sami Paavola). Queiroz is a member of the International Association for Cognitive
Semiotics (IACS), a member of Group for Research in Artificial Cognition (UEFS,
Brazil) and an associate researcher of the Linguistics and Language Practice Department,
University of the Free State (South Africa). Website: https://joaoqueirozsemiotics.
wordpress.com

Göran Sonesson is Professor Emeritus at the Division of Cognitive Semiotics, Lund

University, Sweden. He holds doctorates in general linguistics from Lund and in semiotics
from Paris. He has published numerous papers, both theoretical and experimental, on
pictorial, cultural and cognitive semiotics, as well as on the semiotics of communication
and translation and the evolutionary foundations of semiosis. Apart from anthologies,
his papers have appeared in journals such as Semiotica, Cognitive Semiotics, Cognitive
Development, Sign System Studies, Degrés, Signa, Signata, Sign and Society and Frontier
of Psychology. His main book-length works are Pictorial Concepts (1989), which is a
critique of the critique of iconicity, and Human Lifeworlds (2016), which is a study of
xvi LIST OF CONTRIBUTORS

cultural evolution. His new book, The Pictorial Extensions of Mind, will be published
next year by de Gruyter.

Sabine Tan is Senior Research Fellow at Curtin University, Western Australia. She
has a background in critical multimodal discourse analysis, social semiotics and
visual communication. She has applied multidisciplinary perspectives to the analysis
of institutional discourses involving traditional and new media. She has worked on
interdisciplinary projects involving the development of interactive software for the
multimodal analysis of images, videos and 360-degree videos for research and educational
purposes. She has also worked on projects to develop mixed methods approaches which
combine multimodal analysis, data mining and visualization for big data analytics in areas
such as violent extremism, news discourse and 360-degree video.

Jordan Zlatev is Full Professor of General Linguistics and Director of Research for the
Division of Cognitive Semiotics at Lund University. He served as the first president
for the Swedish Association of Language and Cognition (SALC), 2006–9, and for the
International Association for Cognitive Semiotics (IACS), 2013–14. His current research
focuses on polysemiotic communication and, more generally, on the nature of language in
relation to other semiotic systems like gesture and depiction, as well as to consciousness.
He is editor-in-chief of Public Journal of Semiotics.
 ACKNOWLEDGEMENTS

Out of all the hundreds of individuals involved in this project, we owe a big thank you
up front to Andrew Wardell, Senior Commissioning Editor of Linguistics at Bloomsbury
from 2017–20. Without Andrew’s invitation and his ensuing vision, enthusiasm and
patient support for this project, it would never have gotten off the ground; and it
certainly wouldn’t have grown from a single volume ‘companion’ into a four-volume
‘major reference work’. Andrew issued the invitation for this project while still serving
as Editorial Assistant to Gurdeep Mattu. Many thanks are also due to Becky Holland,
Editorial Assistant to Andrew from 2018–21; to Morwenna Scott, Senior Commissioning
Editor of Linguistics since 2020; and to Laura Gallon, Editorial Assistant to Morwenna
since the end of 2021 – each of whom played key roles in shepherding the project along
to publication against the bottomless backdrop of tragedy, uncertainty and delay that
marked the Covid-19 pandemic. Additional vital support from Bloomsbury during
the project’s final stages came from Production Editor, Elizabeth Holmes, and from
Dharanivel Baskar, Team Lead for project management at Integra Software Services, both
of whom engaged untold support from their own dedicated teams. It has been a pleasure
to work with all of you: thank you once again.
During the final two years of intensive work, this project benefitted financially from
a number of research grants, including a 2020–21 collaborative publishing grant from
the Faculty of Arts and Creative Industries, Middlesex University, London (via Paul
Cobley) and three grants from Toronto Metropolitan University – including a 2020–21
Faculty Research Grant from the Faculty of Arts (via Jamin Pelkey), a 2021 graduate
research assistant grant from the TMU-York Graduate Program in Communication and
Culture (via Stéphanie Walsh Matthews) and a 2021–22 Work Study Research Assistant
Grant from the Office of the Vice President of Research and Innovation (via Jamin
Pelkey). These funds paid for logistical and copy editing support involving seven student
researchers: three PhD researchers (Sophia Melanson Ricciardone, Richard Rosenbaum
and Jan Vykydal) and four undergraduate researchers (Sari Park, Irene Storozhuk,
Leonard Pamulaklakin and Kai Maurin-Jones). Many thanks to all.
Appreciation is also due to nine anonymous peer reviewers for their invaluable
guidance and feedback, including a converging series of recommendations that ultimately
led to the expansion of the project from one volume to four. The project also benefited
from countless consultations, conversations and inspirations afforded by connections with
semioticians around the world. Let us say thank you in this regard to colleagues in the
Semiotic Society of America, the International Association for Cognitive Semiotics, the
International Cognitive Linguistics Association, the International Society for Biosemiotic
Studies, and the International Association for Semiotic Studies, many of whom have
played a formative role, suggesting in the process the need for additional complementary
volumes of this nature to offer expanded scope, depth and range, in terms of topics,
angles of coverage, and qualified participants. Semiotics, after all, is still only just getting
started; and if these volumes serve the enterprise en route to its next milestone, it is
thanks to everyone involved.
 ­L IST OF ABBREVIATIONS

GUIDE TO CRITICAL PEIRCE EDITIONS

Charles S. Peirce: Primary Sources

CD Peirce, C. S. (1889–91), entries in The Century Dictionary: An

Encyclopedic Lexicon of the English Language, ed. W. D. Whitney, New
York: Century Co. Cited as CD, followed by page number.
CN Peirce, C. S. ([1901–8] 1975–9), Charles Sanders Peirce: Contributions to
‘The Nation’, 3 vols, ed. K. L. Ketner and J. E. Cook, Lubbock: Texas Tech
University Press, 1979. Cited as CN, followed by volume number and page
number.
CP Peirce, C. S. ([1866–1913] 1931–58), The Collected Papers of Charles
Sanders Peirce, vols 1–6, C. Hartshorne and P. Weiss (eds), vols. 7–8, A.
Burks (ed.), Cambridge: Harvard University Press. Cited as CP, followed
by volume number and section number.
EP 1 Peirce, C. S. ([1867–93] 1992), The Essential Peirce: Selected Philosophical
Writings, vol. 1, in N. Houser and C. Kloesel (eds), Bloomington: Indiana
University Press. Cited as EP 1, followed by page number.
EP 2 Peirce, C. S. ([1893–1913] 1998), The Essential Peirce: Selected
Philosophical Writings, vol. 2, ed. Peirce Edition Project, Bloomington:
Indiana University Press. Cited as EP 2, followed by page number.
LoF Peirce, C. S. ([1895–1910] 2019–21), Logic of the Future: Writings on
Existential Graphs, ed. A.-V.Pietarinen, vol. 1: History and Applications;
vol. 2/1: The Logical Tracts; vol. 2/2: The 1903 Lowell Lectures; vol. 3/1:
Pragmaticism; vol. 3/2: Correspondence, Berlin: De Gruyter. Cited as LoF,
followed by volume number and page number.
LI Peirce, C. S. ([1891–1910] 2009), The Logic of Interdisciplinarity: The
Monist Series, ed. E. Bisanz, Berlin: Mouton de Gruyter. Cited as LI,
followed by page number.
MS Peirce, C. S. ([1857–1914] 1787–1951), The Charles S. Peirce Papers
Harvard University, Houghton Library, MS Am 1632. Individual papers
are referenced by manuscript number in R. Robin (ed), Annotated
Catalogue of the Papers of Charles S. Peirce, Amherst: University of
Massachusetts Press, 1967, and in Robin (1971), ‘The Peirce Papers: A
Supplementary Catalogue’, Transactions of the Charles S. Peirce Society 7:
37–57. Cited as MS, followed by manuscript number and page number.
­LIST OF ABBREVIATION xix

Note: Houghton Library catalog available online: https://hollisarchives.lib.

harvard.edu/repositories/24/resources/6437.
Many of Peirce’s papers and letters are available in the Microfilm Edition,
Harvard University Library, 38 Reels (1966-70). Digital images of the
microfilm, in the Robin catalog number sequence, are also available online:
https://rs.cms.hu-berlin.de/peircearchive/pages/home.php.
NEM Peirce, C. S. ([1866–1913] 1976), The New Elements of Mathematics, 4
vols, ed. C. Eisele, The Hague: Mouton Press. Cited as NEM, followed by
volume number and page number.
PoM Peirce, C. S. ([1888–1908] 2010), Philosophy of Mathematics: Selected
Writings, ed. M. E. Moore, Bloomington: Indiana University Press. Cited as
PoM.
PPM Peirce, C. S. ([1903] 1997), Pragmatism as a Principle and Method of Right
Thinking: The 1903 Harvard Lectures, ed. P. Turrisi, New York: SUNY
Press. Cited as PPM, followed by page number.
SS Peirce, C. S. and V. L. Welby ([1903–11] 1977), Semiotic and Significs:
The Correspondence between Charles S. Peirce and Victoria Lady Welby, ed.
C. Hardwick and J. Cook, Bloomington: Indiana University Press. Cited
as SS, followed by page number.
SWS Peirce, Charles S. ([1894–1912] 2020), Selected Writings on Semiotics,
1894–1912, ed. F. Bellucci, Berlin: Mouton de Gruyter. Cited as SWS,
followed by page number.
W Peirce, C. S. ([1857–92] 1982–2010), Writings of Charles S. Peirce, 7 vols
(1–6, 8), ed. Peirce Edition Project, Bloomington: Indiana University Press.
Cited as W, followed by volume number and page number.
xx
 Introduction
Semiotic Movements

PAUL COBLEY

This fourth volume of Bloomsbury Semiotics might be best introduced – perhaps

surprisingly – with reference to a play first performed in 1670. Molière’s Le Bourgeois
gentilhomme (The Bourgeois Gentleman) features the attempts of a middle-class man to
realize, through instruction, his aristocratic aspirations. Monsieur Jourdain, wishing to
ascend to the noble class, enlists the services of a number of teachers who will educate
him in the ways of the aristocracy. In line with his ambition, he desires union with a
Marchioness and asks his philosophy master to help him compose a love note to her. The
philosophy master asks him whether he wants the note composed only in prose. Then:

Monsieur Jourdain: No, I don’t want either prose or verse.

Philosophy Master: It must be one or the other.
Monsieur Jourdain: Why?
Philosophy Master: Because, sir, there is no other way to express oneself than with
prose or verse.
Monsieur Jourdain: There is nothing but prose or verse?
Philosophy Master: No, sir, everything that is not prose is verse, and everything that
is not verse is prose.
Monsieur Jourdain: And when one speaks, what is that then?
Philosophy Master: Prose.
Monsieur Jourdain: What! When I say, ‘Nicole, bring me my slippers, and give me
my nightcap,’ that’s prose?
Philosophy Master: Yes, Sir.
Monsieur Jourdain: By my faith! For more than forty years I have been speaking
prose without knowing anything about it, and I am much obliged to you for
having taught me that [. . .]
(Molière, Le Bourgeois Gentilhomme, Act II, Sc. IV)

This brief interaction dramatizes what has become known as the ‘Jourdain effect’. An
amusing episode, it demonstrates the naivete of Jourdain in failing to recognize the
dominance of prose in discourse but also, more importantly, the fact that he is not
conscious of the very architecture of his own speech. This lack of awareness of the
mechanics of signification processes even while one is using them is common over a
2 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

number of domains. Many people around the globe talk to each other every day without
ever registering that they are employing verbs, nouns, tenses and cases. Yet, the effect
operates also at the level of academic fields and disciplines: no matter how self-conscious
and methodologically precise any discipline might be, it is still susceptible of utilizing
procedures and assumptions which it barely acknowledges but which are central to its
epistemology and its operations.
Among the many things that semiotics does and the many areas of scholarly endeavour
that it pervades, there are still fields and domains where a semiotic perspective is only
found to be implicitly at work. Sometimes that perspective is acknowledged without
necessarily being named (e.g. cognitive theory); sometimes it occupies a major part of the
domain (e.g. cultural studies); sometimes it is unacknowledged but not necessarily named
(e.g. linguistics); sometimes it is deliberately repressed for methodological purposes
(e.g. information theory and, again, linguistics); sometimes it is repressed with bad
faith (e.g. biology). The founding father of contemporary semiotics, Thomas A. Sebeok,
frequently referred to the Jourdain effect or ‘factor’ (1998: 35) in his interdisciplinary
studies (e.g. 1991: 45) and, indeed, as Danesi (2011: 116) points out, made a vast number
of converts to semiotics in this way (cf. Houser 2016: 321; Favareau 2010: 44; Aroni,
Chapter 12, Vol. 2). It should be pointed out, too, that Roman Jakobson (1960: 356)
made exactly the same point about the Jourdain effect in relation to metalanguage in
one of the most famous contributions to semiotics and communication theory, which
appeared in a book edited by Sebeok over sixty years ago. The general contention, then, is
that in many disciplines where signification is a central – or even marginal – process, any
consideration of it, however unsystematized it may be, amounts to a work of semiotics.
This may seem a straightforward issue at first glance. Since semiotics is concerned with
signification of all kinds, it seems to follow that any field which features sign trafficking
and refers to this is, effectively, doing semiotics. Indeed, that field will be in want of
semiotics, particularly if it takes signification for granted, treats it as an unconsidered
trifle or, worse, represses the fact that its processes involve signs in action. Something
of each of these scenarios is often present in the approaches of a number of disciplines.
Biology, for example, features sign action at higher, organismic levels (animal and
plant signals) as well as at lower thresholds (the neurons, mRNA). Yet, when biology
is taught, a semiosic entity such as mRNA is usually pronounced to be a ‘messenger’
effective in the production of proteins. Vanishingly rare is the case that there will be,
as part of biology as a disciplinary field, explication of the messenger mechanism and
its relation to other biological functions with palpably semiotic constitutions. Indeed,
addressing this lacuna in biology is one of the aims of biosemiotics within general
semiotics. However, confronting the matter of semiosis in terms of the Jourdain effect
may not always be well advised. Norris (1986: 229) suggests that figuring semiotics in
this fashion may seem like a threat to instal it as a ‘master-science’ while also depriving
it of the possibility of enjoying any clear disciplinary status. Certainly, this would not fit
well with the orthodoxy of, say, postmodernism and, in particular, its customary claim
that there is incredulity towards meta (or master) narratives – such as semiotics in this
figuring – and that universals need to be superseded by the specifics of the local political
conjuncture. Norris suggests that the semiotician says, with ‘boundless confidence’,
humani nil a me alienum putum. Well, since Norris was writing, the situation has got
worse than that because, as the aforementioned biosemiotics has attested over the last
thirty years, the domain of semiotics has expanded beyond the human to all known
life forms.
INTRODUCTION 3

In the face of postmodernism and its decline as an academic trend, many disciplines have,
while remaining alive to the importance of local configurations, gone back, or continued,
to explore the possibility that there may exist universals. In the current volume, such
universals include interpretative and communicative processes (chapters on hermeneutics,
phenomenology, communication theory), the operation of systems (chapter on systems
theory), human processes of knowing (chapters on digital humanities, media and cultural
studies and on translation studies), human modes of signification (chapters on pragmatics,
multimodality, discourse analysis and integrationism) and cognition (with chapters on
cognitive linguistics and cognitive science generally). Potentially, there could be other
fields, disciplines and domains where semiosis and semiotic understanding are involved
in the investigation of what some consider to be universal processes and attributes. These
include medicine (see Tredinnick-Rowe and Stanley, Chapter 6, Vol. 2), finance and
economics (see Oakley, Chapter 10, Vol. 2), religion and the idea of a supreme being (see
Leone, Chapter 4, Vol. 3), mainstream linguistics, cultural anthropology (see Ness and
Coleman, Chapter 2, Vol. 3), palaeontology, the possibility of artificial intelligence, law
(see Chapdelaine-Feliciati, Chapter 8, Vol. 2) and performance studies.
So, in pursuit of some of these processes, the current volume is concerned with
semiotic movements. Whereas Volume 1 featured contributions on movements and
areas of prominence in the history of semiotics, with Volume 2 featuring contributions
specifically concerned with areas of the natural and technical sciences which have been
infused with semiotics, and Volume 3 containing the same for the arts and social sciences,
the current volume slightly re-figures the term ‘semiotic movements’. Rather than being
movements that have characterized the development of semiotics as a disciplinary field,
the contributions here are largely concerned, in line with the Jourdain effect, with
aspects of signification – but have not necessarily declared, fully realized or completely
embraced that fact. They are movements that are semiotic, but often proceed with areas
of functioning that do not necessarily advertise their semiotic bearing.
Communication theory (Lanigan, Chapter 1, this volume) has had a relationship to
semiotics which has often been so close that it has been impossible to determine the
boundary line (Sebeok 1977: 185). Indeed, it has been closer, perhaps, than the field
of communication studies in general, which has been overwhelmingly concerned with
human communication, particularly as it might be approached from a social sciences
perspective (Cobley and Schulz 2013). Qualifying this last observation, it should be noted
that there are indications that the two fields, with communication theory at the forefront,
are inexorably moving closer (Self 2013). Although the convergence of semiotics,
particularly in its more contemporary concerns beyond human communication, and
media/culture studies (Melanson and Danesi, Chapter 2, this volume) cannot be said to
be proceeding as rapidly as the convergence of semiotics and communication theory,
there has been progress. The second part of the media and cultural studies couplet,
particularly in some aspects of ‘posthumanism’, has started to acknowledge common
grounds and approaches, albeit often with its own preliminary agenda. Indeed, digital
humanities (Olteanu and Ciula, Chapter 2, this volume) as a whole has, as its raison
d’etre, a concern with signification – both in the differences in signification between
digital and non-digital methods for research and between the same for dissemination
of knowledge. As with communication theory, systems theory (Gudwin and Queiroz,
Chapter 4, this volume) has also had an immeasurably close relationship with semiotics,
arising, initially, from the post–Second World War ‘great expectations’ (Sperber 2008)
of a unified science featuring cybernetics, information theory, linguistics, semiotics and
4 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

game theory. Cannizzaro (2010, 2013, 2014) argues, persuasively, that systems theory
is effectively a continuation of semiotics, particularly in the centrality of the concept
of modelling after the work of Lotman from the 1950s onwards. Where it differs from
media/culture studies, communication theory and digital humanities is that systems
theory has not harboured a semiotic wing itself but, rather, studied systems in a manner
that is very much analogous and allied to that of semiotics’ approach.
Semiotics has also been a concern for particular developments in philosophy in the
last century, in terms of its implications both for ontology and for epistemology. In
this volume, phenomenology and hermeneutics have been chosen as emblematic of this
concern as a ‘semiotic movement’. Although semiotics has been resolutely committed to
an ‘anti-psychologistic’ (Stjernfelt 2014) perspective on the existence of signs – sometimes
in contrast, sometimes overlapping with contemporary speculations from cognitive
perspectives – it has acknowledged, in various ways, the crucial role of first-person
experience in the fortunes of semiosis. One reason why phenomenology and semiotics can
be so closely related is that the latter, while seeming to propose an entity called ‘the sign’
and a larger one called ‘semiosis’, both of which are putatively irreducible and stable, has
never, in practice, presented such a proposition. In the oft-quoted definition from Peirce,
for example, a sign ‘stands to somebody for something in some respect or capacity. It
addresses somebody that is, creates in the mind of that person an equivalent sign or
perhaps a more developed sign’ (c.1899: CP 1.564). The sign, in this sense, is very much
bound up with phenomenology (Bundgaard, Chapter 5, this volume), notwithstanding the
fact that ‘somebody’ in this formulation need not even be human. This phenomenological
orientation of the sign, of course, entails that signs are variably interpreted. Originally
applied to sacred texts, the close study of written sources that provided the crucible for
hermeneutics (Arnett and Mancino, Chapter 6, this volume) is to be found in the work
of ‘Plato, Augustine, Humanism, the early Heidegger and Husserl’, according to Grondin
(1995: xi). He adds (1995: 36) that phenomenology gave birth to hermeneutics, almost
unwittingly or against the wishes of Husserl, one of modern phenomenology’s founders.
What hermeneutics does carry out, however, is a kind of intense phenomenological
semiotics of texts which is allied to other kinds of textual semiotics such as narratology
but, because of its institutional history, has managed to remain distinct.
Effectively, a form of textual semiotics is quite explicitly carried out by four more of
the ‘semiotic movements’ covered in this volume. In the burgeoning field of translation
studies (Kourdis and Hartama-Heinonen, Chapter 7, this volume), there is specific focus
on how one text, with all the determinants specific to it, might undergo a transmutation
into another text, often involving shared determinants but also involving quite different
ones. The circumspection of the formulation in the last sentence will be noted because, in
demotic conceptions of translation, what is taken to be involved is merely a conversion
from one national language to another. What semiotics has demonstrated for translation
studies is that national language correspondences and divergences are merely the tip of
the iceberg. Translation takes place across a number of semiotic modes, both verbal and
non-verbal, with ‘interlinguistic’ translation only constituting a small part of a general
process featuring in myriad transmissions at lowly organismic levels upwards (Marais
2020). Similarly, the studies of multimodality (Machin and Chen, Chapter 9, this volume)
that have come to dominate the analysis of discourse in the last twenty-five years have
arisen from fairly modest beginnings. Mainstream linguistics, constantly reinventing itself
and attempting to protect against extinction, started to realize that contemporary culture
has involved other modes than the word. Most commonly identified has been pictorial
INTRODUCTION 5

representation, especially in digital interfaces, with the move from page to screen (Kress
2003). Often, ‘semiotic’ has been taken by linguists as a synonym for that which is not
solely linguistic; however, the field of multimodality studies has developed in a limited
way, focusing on the pictures and the visual, with research into other non-verbal modes
(e.g. Jaworski and Thurlow 2011) still a rarity.
Nevertheless, both translation studies and multimodality studies retain and share a
fairly generalist remit with semiotics. Discourse analysis and pragmatics, by contrast, are
more avowedly specific in their concerns. One part of linguistics’ reinvention in the face
of semiotics over the last fifty or sixty years has involved fragmentation. In addition to
multimodality studies, linguistics has also developed sociolinguistics, discourse analysis,
Critical Discourse Analysis (CDA), systemic functional linguistics (SFL), ordinary language
philosophy, pragmatics, relevance theory, (its own version of) ethnomethodology,
conversation analysis (CA), psycholinguistics, generative linguistics, generative semantics,
neurolinguistics, cognitive linguistics and numerous others, many of them continuing to
focus on linguistics’ traditional moieties of morphology, phonology, phonetics, grammar,
syntax, lexicon and semantics. Yet, it is worth remembering that a more general division
which has applied in linguistics was actually proposed from within semiotics as early as
1938. In Foundation of the Theory of Signs, Charles Morris identified the semantical
dimension of semiosis (the relation of signs to objects), the pragmatical dimension of
semiosis (the relation of signs to interpreters) and the syntactical dimension of semiosis
(the formal relation of signs to one another) (1938: 6–7). Taken up also by Carnap (1948:
8–11) in the service of logic and deduction, the division effectively launched the linguistic
field of pragmatics which has enjoyed immense success. Enjoying the benefit of multiple
journals, a plethora of textbooks, specifically titled professors and a well-subscribed
international association, pragmatics has expanded the study of the relation of signs to
interpreters with an extensive armoury of technical applications. Arguably launched as
a discipline in ordinary language philosophy and, especially, the work of Austin ([1955]
1962), pragmatics has been, from its wider semiotic beginnings, principally a pursuit
confined to linguistics (Moeschler 2021). As a semiotic movement (Brandt, Chapter 8,
this volume), its attempts to move beyond a curricular prison-house of language are still in
gestation (Wharton 2009; Scarantino 2017; Xie, Yus and Haberland 2021), although the
benefits of pragmatics in understanding the movement from non-verbality to acquisition
of language by children have been made palpable (Clark 2017; Bohn and Frank 2019).
In its relation to some multimodality studies, discourse analysis, by contrast with
pragmatics, has made significant inroads in the move beyond language (Machin 2013).
Nevertheless, it is once more a semiotic movement whose main profile has primarily been
in the field of language. Developing in tandem with other linguistic text-focused semiotic
movements such as narratology and myth criticism, discourse analysis (O’Halloran and
Tan, Chapter 10, this volume) has both a general existence and a more specified one.
The more general one arises out of the impetus to analyse features of language beyond
or bigger than the traditional linguistic domain of the sentence. Arguably, this impetus
has a lineage which runs in parallel with the relatively recent (Harris 2001) establishment
of linguistics in the academy. It has often been manifest in fairly consistent concerns
with figures in language such as metaphor, metonymy, simile, irony and so forth. Yet,
along with the development of Saussurean semiotics (or semiology) in Europe during
the 1960s, and especially the ‘invention’ of the ‘text’ (Marrone 2014), such concerns
turned increasingly towards a more critical analysis which exposed the broadly political
implications of the options taken within discourse. Roland Barthes, whose work differed
6 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

from discourse analysis in that he took linguistic models and expanded them to analyses
of non-verbal phenomena from the outset, nevertheless provided inspiration for CDA
(Waugh, Masaeed, Hong Do and Renigar 2016; Catalano and Waugh 2020). As with
Barthes’s early essays (1973, 1977), discourse analysis increasingly attempted to reveal
not just the mechanics of texts, but the ‘agenda’ of any instance of discourse: how it
worked – largely unconsciously – to instil particular messages which were often far from
being disinterested ideologically, politically or both. As discourse analysis developed in the
1980s and 1990s, the question of power became absolutely central, with many analyses
learning from Foucault – not an avowedly semiotic thinker – that institutions, currents of
thought and even whole societies might act in a programmed fashion under the yoke of
one or more discourses. Critical discourse analysis (in upper case – CDA), developed by
Norman Fairclough (1989), is a particular case in point, ossifying into a method which
involves a kind of close reading or semiotics of texts in order to demonstrate a larger point
about how a given text instantiates the embedding of power relations in sign systems.
One benefit of early discourse analysis, perhaps, was its relative lack of methodological
precision. It could combine a number of approaches and adjust its own overall approach
in line with the situations that gave rise to the texts upon which it focused. Later,
CDA and critical discourse studies enjoyed a great deal of success and are sometimes
considered as rigorous methods, supported by a number of journals invoking their
name. Yet, as Peter Jones (2007) argues, CDA has been successful because it flatters
the culture which sponsors it. As the holders of power in contemporary Western social
formations have become more aware of the importance of sign systems, so critical
discourse analysts – with good intentions to criticize unequal power relations and
even promote social change – have responded in a way which not only affirms the
current state of affairs but reifies the phenomenon under scrutiny: that is, language.
Jones’s argument is mounted from the position of the thoroughgoing critique of the
‘language myth’ (Harris 1981) which characterizes integrationism (Pablé, Chapter 11,
this volume). Rather than ‘segregating’ communication processes (the actual activities in
which sign users engage) from language (routinely conceived as an entity, or an abstract
system), an integrationist approach, as the name implies, brings both together as one
process. As a result, integrationism is an anomaly in a long history, as well as amidst
contemporary practices, of conceiving signification in terms of the myth of language as
something that can be located, quantified and labelled. For Jones (2007: 367), ‘thinking
about language in terms of “codes”, “models”, “frameworks” of rules, rule systems,
or choices within inventories or networks is not part of the solution but part of the
problem’. As a semiotic movement, then, integrationism or ‘integrational semiology’
and contemporary semiotics have been steadily converging, as the latter has moved away
from its decades-long (1960s to 1980s) fixation on codes and invariants in signification
(Pablé and Hutton 2015).
Similar issues are evinced in the development of what might be the quintessence of a
certain kind of semiotics, recapitulating linguists’ synonym for that which is not solely
linguistic: gesture studies. Again, gesture studies, as a particularly strong branch of non-
verbal communication studies in general, has sometimes been riven by the relationship
of semiosis and invariance. Sebeok (2001a: xiii) refers to the work of his ex-University of
Chicago office mate Ray Birdwhistell, whose kinesics introduced a programme devoted
to gesture but which became ‘effectively moribund’ owing to its attempt to align non-
verbal communication with grammar. Foremost in semiotics, with programmes devoted
to a more semiotic rather than codified perspective on gesture, are the works of Adam
INTRODUCTION 7

Kendon (e.g. 2004) and David McNeill (e.g. 2007). Yet, as a field of study, gesture
itself has tended to be emblematic of the division of non-verbal communication in the
academy. In communication study, for example, non-verbal communication has become
a massive endeavour (Hall and Knapp 2013), but this is within a context in which
‘communication’ implies only human communication and largely focuses on bodily
communication other than endosemiosis. Despite the inaugural work of Ruesch and
Kees (1956), there have been very few attempts to situate gesture and human bodily
communication within the wider context of the non-verbal communication which
includes the whole complex of visual communication in general, endosemiosis and non-
human communication (Sebeok 2001b). Nevertheless, the proximity of gesture as a field
to that of human communication, along with its address of some fundamental issues, has
maintained its central place in semiotics.
Equally enmeshed in a complex relationship with semiosis and invariance have been
the semiotic movements associated with the so-called ‘cognitive revolution’. Brandt
(2011: 49) has suggested: ‘If semiotics studies meaning, and cognitive science studies
the mind, then cognitive semiotics is the study of mind and meaning – the way meaning
exists and works in human minds (and ideally, in animal minds in general).’ This is
not at all unreasonable. Furthermore, the fact that his statement bears upon ‘cognitive
semiotics’, a major strand of general semiotics, with its own successful society, regular
conferences and an esteemed journal makes it seemingly uncontentious as a statement.
Notwithstanding the Jourdain effect, however, a strong case could be made for
cognitive linguistics and cognitive science as semiotic movements – that is, as studying
‘meaning’, in Brandt’s formulation, from the very start. Both of them trade, primarily,
in signs or the concept of signs. Indeed, despite the fact that he did not commit it to
print, more than once Thomas A. Sebeok referred to research in cognitive science as
‘semiotics plus money’. Sebeok was referring not so much to the coded semiotics of
cultural theorists, but to the dynamic sign as conceptualized by Peirce and a semiotics
derived from the new conceptualization of language in which he had participated in the
immediate post–Second World War period. For Sebeok, whose own work contributed to
resolving the impasse of behaviourist research created by the likes of his mentor Charles
Morris, the perspective of Chomsky as well as the cognitive linguistics and the cognitive
revolution that followed were pivotal (see Cobley 2016: 34–8). Cognitive linguistics
(Zlatev and Möttönen, Chapter 13, this volume), stemming from Chomsky’s work in
the first instance, forced semiotics (as well as other fields) to consider the prominent role
of syntax. Syntax, of course, had already been introduced into the semiotic programme
by Morris, but was afforded comparatively lowly status. Cognitive linguistics raised the
stakes by insisting that the very definition of language lay in the innateness, as well as
the formality, of syntactic rules (Chomsky 1957). Yet, what was to characterize cognitive
science was the ‘plus money’ aspect of Sebeok’s formulation – that is, cognitive science
(as the name may even suggest) conceived itself in scientistic fashion, seeking to carry out
experiments and empirical investigations rather than laying out formal rules (Sonesson,
Chapter 14, this volume). This would not necessarily guarantee that it became a science;
however, empirical work can provide considerable cachet in the world of grant-driven
research, particularly when it purports to deal with something as fundamental as
consciousness. Even so, a glimpse of the paraphernalia of much thinking in cognitive
science – signalling, schema, rules, models, as well as semantics, pragmatics, syntactics
and aspects of existing linguistics – makes it hard to avoid concluding that cognitive
science is a semiotic movement.
8 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

As has been argued, the semiotic movements featured in this volume are, in their own
way, attempts to come to terms with universals, although not necessarily by utilizing
universalizing principles. Thus, they have tended to correspond to historical landmarks
in the development of semiotics in the last two-and-a-half thousand years. As possibly the
founder of semiotics, Hippocrates (c.460–370 BCE) certainly seems to have inaugurated
the systematic investigation of signs in the construction of the corpus of symptoms and
treatments, observing that a symptom exists as such when it is found to be identical
in Delos, Scythia and Libya (Hippocrates [430-330 BCE] 1983: 185). At the very
root of semiotics, then, there is an attempt, for good reasons, to identify a potentially
universal phenomenon through signs. Yet, of course, diagnosis or prognosis is not always
watertight; as Peirce acknowledged (e.g. 1893: CP 5.265), it should be understood as
subject to fallibilism.
Does this entail that there are some semiotic rules in some domains and different
ones in others? Is it the case that some disciplines are more suited to rigid definitions
in signification or actively demand them? Volume 2 of this Bloomsbury Companion,
featuring contributions to the natural and technical sciences, deals with fields that have
tended to rely on invariance in signification. Yet, the contributions to that volume reveal,
in consonance with contemporary semiotics, the extent to which variance in sign use very
much dictates the results and orientations of fields and disciplines. This is probably to be
expected in, say, jurisprudence (Chapdelaine-Feliciati, Chapter 11, Vol. 2); yet it is likely
that most lay people would be unaware of the extent to which this is true of biology,
ecology, zoology, medicine and even mathematics (Kull and Favareau, Chapter 2;
Maran, Chapter 3, Tønnessen, Chapter 4; Tredinnick-Rowe and Stanley, Chapter 4;
Stjernfelt and Pietarinen, Chapter 1; respectively, all in Vol. 2). The semiotics which has
arisen since the twentieth century has had a natural home in the arts and humanities;
so, it is perhaps unsurprising that its interpretative orientation has infused fields from
musicology, education and archaeology (Dougherty, Chapter 10; Olteanu and Stables,
Chapter 7; Tamm and Preucel, Chapter 3; respectively, all in Vol. 3). Nevertheless, what
is demonstrated overall is that some intellectual movements, as well as those inhabiting
the Jourdain effect, have become semiotic or realized some of the implications of their
significatory proclivities.
Put yet another way, intellectual movements through a semiotic lens traverse grounds
where it is now possible to contemplate the project of those movements in their
oscillation between certainty and doubt. Even Hippocrates, whose project was committed
to the goal of certainty lest a shortfall of certainty should precipitate harm, would have
been compelled to operate with some residual doubt. As the groundbreaker for both
a semiotics of interpretation and a semiotics of invariance, effectively his work is the
ultimate reference point certainly for a semiotic history of the universe as adumbrated in
Volume 1, but also for all the movements covered in the current volume. Interpretative
and communicative processes (hermeneutics, phenomenology, communication theory)
seek ways to establish both principles of invariance and invariant properties while
stressing the mutability, locality and indeterminacy of interpretation. The same can be
said, from the opposite direction, in respect of the operation of systems (systems theory),
where the fallibility of systems corresponds to the interplay of agency in semiosis in
relation to the constraints of semiosis. That same phenomenon of agency, associated most
readily with the interpretative behaviour of humans’ specific forms of oscillation between
certainty and doubt in acts of knowing (digital humanities, media and cultural studies,
translation studies), also characterizes the very modes of human signification (chapters on
INTRODUCTION 9

pragmatics, multimodality, discourse analysis, gesture and integrationism). It may even

be a manifestation of cognitive processes generally (anthropological linguistics, cognitive
linguistics, cognitive science).
Much more than doubt and fallibilism, however, semiotics – particularly in relation to
semiotic movements – finds itself confronting a particular challenge which was mentioned
at the start of this Introduction to Volume 4. That is, there is fear that semiotics might
be posed as a ‘master-science’ while at the same time being a discipline with no purchase
at all. Certainly, its lack of a firm institutional basis – strong as a subject area, with many
international journals and handbooks in semiotics such as this one, but less strong as
an embedded subject area outside Latin America, with relatively few courses and
professorial appointments – suggests some truth in the latter. Indeed, it is possible that
semiotics’ alleged positing of itself as a master-science and its weak institutional basis are
intimately connected. Yet, however marginalized semiotics has been in the academy since
Hippocrates, it is a survivor, adapting to the demands of rapidly changing circumstances
in the realm of knowledge and, as this volume attests, being ready to support, facilitate
and stabilize some intellectual movements whether they explicitly acknowledge semiotics
or not.
The contributions to the current volume present a range of semiotic movements and, in
addition to locating them historically in a manner corresponding to this Introduction, they
present specific perspectives on the controversies, debates and challenges that make up the
‘state of the art’. The volume opens with Richard L. Lanigan’s overview of communication
theory. Proceeding from the distinction between competence and performance, Chapter 1
goes on to outline a ‘communicology’, drawing from phenomenology but also from some
generalized models which have haunted communication theory: the human observer
model; the human discourse model; the trivium model; and the chiasm model of discourse.
The chapter sees the interface of semiotics and communication theory in, particularly, the
sign’s ability to mediate a consciousness of time and in the mind which is simultaneously
possible in the mind of the other, the interlocutor, a human that is not me.
Melanson and Danesi’s Chapter 2 shows how media/culture studies have employed
terms and strategies that draw on semiotics, such as opposition theory, code, text,
iconicity and deconstruction. Especially important to modern media theory has been the
work of McLuhan. Not usually associated with semiotics’ history, this chapter shows
how McLuhan’s work reveals media to be an extension of the human’s sensorium, also
creating a global village which is akin to a semiotic life of the mind. This is not only
changing media but also changing what it is to be human. Where the old models of orality
and literacy, with their semiotic overtones, served for previous media formations, both
now have to be extended for the world of memes and perpetual electronic connection
into digital literacy, computer literacy, mobile literacy and virtual worlds literacy.
In their consideration of education, Olteanu and Ciula propose new avenues for digital
humanities. Taking the concept of modelling, which has been influential in Lotman’s work
and in biosemiotics, Chapter 3 suggests that it is especially apposite in understanding
mediality. Following on from Melanson and Danesi’s discussion of mediality in terms of
extensions of sensoria, this chapter argues that the digital humanities inculcate a special
kind of literacy, one that is akin to the process of modelling in humans, an awareness
of which might be taught in order to foster digital skills in general rather than simply
teaching each localized digital skill in turn. In addition, Olteanu and Ciula see possibilities
for semiotic and ecological approaches in digital humanities to encourage a ‘biosphere
consciousness’.
10 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Gudwin and Queiroz’s Chapter 4 suggests that systems theory and semiotics – in
particular, the semiotics of Charles Sanders Peirce – could mutually assist one another.
What they share is the ability to see and even assess entities as processes rather than just as
agents and objects. Semiosis, then, is not to be conceived simply as subject to the vagaries
of all agents; nor does the chapter argue that semiosis is an unchanging system. Instead, it
explores the possibility of a semiotic system – a stable ‘cluster of semiotic processes’. This,
it is argued, has significant implications for the understanding of computer simulations.
Despite the fact that Peirce proposed a phenomenology (his ‘phaneroscopy’),
Bundgaard’s Chapter 5 suggests that this is not necessarily evident in the development
of semiotics. The chapter cites Greimas and Peirce specifically. Yet, Bundgaard argues,
a widened perspective reveals that Husserl’s phenomenology has decisively influenced
the development of semiotics since the early twentieth century. Closely reading Husserl,
Chapter 5 makes similar conclusions to Chapter 1, observing that a phenomenological
semiotics invokes not a self-contained language system but, rather, interaction between
acting intentional subjects giving rise to intersubjective communicating consciousnesses.
As Chapter 6 by Arnett and Mancino avers, semiotics and hermeneutics have an
intellectual objective in common. Comparing them with the work of Gadamer, and
considering the themes of dialogue and semiosis, the chapter analyses two works by
Eco: his rich treatise, A Theory of Semiotics, and his last, plot-driven novel, Numero
Zero. It argues that hermeneutics explicates texts in a manner that exemplifies dialogic
understanding; semiotics, as a complement, is discussed in terms of signs’ relation to other
signs and their translatability. Ultimately, Arnett and Mancino’s conclusion suggests that
hermeneutics and semiotics carry strong traces of their mutual progenitor, Augustine.
Although Augustine stressed signification’s ability to beget further, future-orientated,
signification, the idea of semiotics as translation, particularly intersemiotic translation,
was properly set in motion by Roman Jakobson. Chapter 7, by Kourdis and Hartama-
Heinonen, identifies three main currents of thinking translation semiotically after
Jakobson: from the perspective of Peircean semiotics; from the perspective of the Paris
School, paying particular attention to ‘intertextuality’; and from the perspective of
the Tartu-Moscow School. These have led to broadly different approaches depending
on whether the objects of translation are linguistic or non-linguistic, whether they are
cultural, cognitive, sociological or technological in nature, or whether they are more
broadly concerned with objects of different kinds whose common denominator is that
they are made up of signs.
A movement as influential, particularly in communication studies, as pragmatics
clearly deserves its own chapter in a volume on semiotic movements. Brandt’s Chapter 8
considers the interface of pragmatics and semiotics, finding it sufficiently significant to
warrant the proposal of a semio-pragmatics. The chapter proceeds from the conception
of meaning as both ‘immanent’ and ‘transcendent’, conceptual and contextual. Brandt
sees semiotics as principally concerned with the former in these couplets and pragmatics
as principally concerned with the latter. The chapter suggests that, in fact, pragmatics
is constantly prevented from drifting off into sociology, linguistics or anthropology by
semiotics’ insistence that pragmatics is called back to the ‘fundamental life of signs in
human life’.
Chapter 9, by Mittelberg and Hinnell, focuses on gesture, or ‘how interlocutors
gesturally indicate objects, ideas, locations, or events and enact physical habits such as
interpersonal interaction, movement patterns, or object manipulation, as well as more
abstract, yet deeply embodied, schemata of experience’. Seemingly not a semiotic
INTRODUCTION 11

movement as such, the field of gesture studies invokes other ‘movements’ – cognitive
science and multimodality, especially – and its concern with both human communication
and the interpretation/invariance couplet, as noted above, offers it the flavour of a
movement. Unlike many other discussions of gesture in semiotics which either adapt
linguistic models or develop their own for discussing non-verbal communication, this
chapter draws on Peirce, particularly his understanding of how signs relate to their
objects and the embedding of these relations in habits. Mittelberg and Hinnell survey
theories and empirical work, ultimately concluding that gesture studies not only throw a
great deal of light on human gestures in a developmental ecology of human non-verbal
communication, but also provide indications for understanding iconicity, indexicality and
habit in semiosis more widely.
Signs in human life partake of many modes. Machin and Chen’s Chapter 10 makes the
point at the outset that ‘Communication was never “mono-modal” but always involved
things like images, fonts, types of paper, gestures, sound, music’. Multimodality, as an
approach that exemplifies the recognition that there are many modes of communication,
has also customarily indicated that ‘choices’ of mode are very significant and that, for the
addressee of communications, any choices offered are usually partial and limited. Taking
a case study of food packaging which presents comestibles purporting to be healthy,
and does so in a gendered way, the chapter shows how, say, energy or protein food bars
aimed at men feature packaging that connotes science and technology, whereas those
for women connote ‘nature’. Multimodality is used here, then, to expose how food
marketing companies use communication modes strategically in order ‘to colonise and
shape discourse and practices of taking care of our bodies’.
In Chapter 11, O’Halloran and Tan trace the origins of multimodality and social
semiotics to the seminal work of Michael A. K. Halliday, finding that some key
concepts which his work allowed to flourish – system, metafunction, intersemiosis and
resemiotization – provided the underpinnings for contemporary multimodal discourse
analysis. Carrying out a number of approaches from the latter, including cluster
visualization, along with the application of IBM Watson and Clarifai models, the chapter
finds that the implementation of such mixed methods with current computer technology
offers considerable possibilities for textual and contextual understandings through
collection of metadata and existent semantic classifications.
Any combination of semiotics with other movements necessarily entails an embrace
of what semiotics understands to be the ontology of the sign. Yet, is it possible that the
ontology of the sign for many in semiotics has become a little too settled? This is the key
question asked by Pablé in Chapter 12. As opposed to the version offered by Saussure
and Peirce, Pablé presents a Third Ontology of the sign, based on human creativity rather
than a ‘standing for’ function. Derived from the integrationist principles of Roy Harris,
the chapter outlines an ontology in which signs are what sign-makers make them. It
suggests that integrationism is antithetical to a global semiotics because human activities
and non-human activities ‘must ultimately be treated as “activities” tout court which
trigger signs’ although it must be conceded that ‘integrationism is deeply humanist and
anthropocentric’.
Chapter 13 by Zlatev and Möttönen, whilst noting the Peircean tradition’s attempt to
‘dementalize’ semiosis but also overlooking Sebeok’s incorporation of post-Chomskyan
linguistics into semiotics, observes that semiotics and cognitive linguistics seem to be
miles apart. Yet, drawing attention to construal, iconicity, metaphor and multimodality,
the chapter suggests that the subjectivist conception that dominates cognitive linguistics
12 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

and the emphasis on intersubjective structures in semiotics is resolved to some extent by

the relatively recent development of cognitive semiotics, encompassing phenomenology
and cognitive science generally, as well as semiotics and cognitive linguistics.
Finally, Chapter 14, pace the Sebeok formulation on cognitive science cited earlier
in this chapter, treats semiotics and cognitive science as two distinct research traditions.
Semiotics, according to Sonesson, is concerned with ‘meaning’; cognitive science is
concerned with consciousness. His chapter compares and contrasts semiotics and
cognitive science, drawing out the stages of the latter, commencing with a focus on
artificial intelligence and culminating, in recent years, with a movement from its fringes
towards a more phenomenologically based concern with the nature of consciousness. As
in the previous chapter, this one draws attention to the emergence of cognitive semiotics.
Observing that linguistics is not to be considered as a third ingredient of cognitive
semiotics, because it is already embedded in semiotics and cognitive science, Sonesson
nevertheless notes that linguistics’ penchant for setting up rules and regularities is carried
over in aspects of cognitive semiotics.

REFERENCES
Austin, J. L. ([1955] 1962), How to Do Things with Words: The William James Lectures
Delivered at Harvard University in 1955, ed. J. O. Urmson and M. Sbisà, Oxford:
Clarendon Press.
Barthes, R. (1973), Mythologies, trans. A. Lavers, London: Paladin.
Barthes, R. (1977), Image – Music – Text, ed. and trans. S. Heath, Glasgow: Fontana.
Bohn, M. and M. Frank (2019), ‘The Pervasive Role of Pragmatics in Early Language’, Annual
Review of Developmental Psychology, 1: 223–49.
Brandt, P. A. (2011), ‘What Is Cognitive Semiotics? A New Paradigm in the Study of Meaning’,
Signata, 2: 49–60, https://journals.openedition.org/signata/526.
Cannizzaro, S. (2010), ‘On Form, Function and Meaning: Working Out the Foundations of
Biosemiotics’, Hortus Semioticus, 6: 40–52.
Cannizzaro, S. (2013), ‘Where Did Information Go? Reflections on the Logical Status of
Information in a Cybernetic and Semiotic Perspective’, Biosemiotics, 6 (1): 105–23.
Cannizzaro, S. (2014), ‘Transdisciplinarity for the 21st Century, or “Biosemiotics as Systems
Theory”’, Cybernetics and Human Knowing, 21 (3): 45–59.
Carnap, R. (1948), Introduction to Semantics, Cambridge, MA.: Harvard University Press.
Catalano, T. and L. Waugh (2020), Critical Discourse Analysis, Critical Discourse Studies and
Beyond, New York: Springer.
Chomsky, N. (1957), Syntactic Structures, The Hague: Mouton.
Clark, E. V. (2017), Language in Children, London: Routledge.
Cobley, P. (2016), Cultural Implications of Biosemiotics, Dordrecht: Springer.
Cobley, P. and P. J. Schulz (2013), ‘Introduction’, in Cobley and Schulz (eds), Theories and
Models of Communication, 1–16, Berlin: de Gruyter.
Danesi, M. (2011), ‘The Semiotic Foundations of Knowledge: Remembering Thomas A.
Sebeok’, in P. Cobley, J. Deely, K. Kull and S. Petrillil (eds), ‘Semiotics Continues to
Astonish’: Thomas A. Sebeok and the Doctrine of Signs, 115–22, Berlin: de Gruyter.
Fairclough, N. (1989), Language and Power, London: Longman.
Favareau, D. (2010), ‘An Evolutionary History of Biosemiotics’, in Favareau (ed.), Essential
Readings in Biosemiotics: Anthology and Commentary, 1–80, Dordrecht: Springer.
INTRODUCTION 13

Grondin, J. (1995), Sources of Hermeneutics, Albany: SUNY Press.

Hall, J. A. and M. L. Knapp, eds (2013), Nonverbal Communication, Berlin: de Gruyter.
Harris, R. (1981), The Language Myth, London: Duckworth.
Harris, R. (2001), ‘Linguistics after Saussure’, in P. Cobley (ed.), The Routledge Companion to
Semiotics, 118–33, London: Routledge.
Hippocrates (1983), Hippocratic Writings, ed. A. Lloyd, Harmondsworth: Penguin.
Houser, N. (2016), ‘Semiotics and Philosophy’, Cognitio: Revista de Filosofia, 17 (2): 313–36.
Jakobson, R. (1960), ‘Closing Statement: Linguistics and Poetics’, in T. A. Sebeok (ed.), Style in
Language, 350–77, Cambridge, MA: MIT Press.
Jaworski, A. and C. Thurlow (2011), ‘Gesture and Movement in Tourist Spaces’, in C. Jewitt
(ed.), The Routledge Handbook of Multimodal Analysis, 363–72, London: Routledge.
­Jones, P. E. (2007), ‘Why There Is No Such Thing as “Critical Discourse Analysis”’, Language
& Communication, 27: 337–68.
Kendon, A. (2004), Gesture: Visible Action as Utterance, Cambridge: Cambridge University
Press.
Kress, G. R. (2003), Literacy in the New Media Age, London: Routledge.
Machin, D. (2013), ‘What Is Multimodal Critical Discourse Studies?’ Critical Discourse Studies,
10 (4): 347–55.
Marais, K. (2020), A (Bio)Semiotic Theory of Translation: The Emergence of Socio-cultural
Reality, London: Routledge.
Marrone, G. (2014), The Invention of the Text, Milan: Mimesis.
McNeill, D. (2007), Gesture and Thought, Chicago: University of Chicago Press.
Moeschler, J. (2021), Why Language? What Pragmatics Tells Us about Language and
Communication, Berlin: de Gruyter.
Morris, C. (1938), Foundations of the Theory of Signs, (vol. I, no. 2, of International
Encyclopedia of Unified Science), Chicago: University of Chicago Press.
Norris, C. (1986), ‘Semiotics in Great Britain’, in T. A. Sebeok and J. Umiker-Sebeok (eds), The
Semiotic Sphere, 229–51, New York: Plenum.
Pablé, A. and C. Hutton (2015), Signs, Meaning and Experience: Integrational Approaches to
Linguistics and Semiotics, Berlin: de Gruyter.
Peirce, C. S. ([1866–1913] 1931–58), The Collected Papers of Charles Sanders Peirce, vols. 1–6,
C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University
Press. Cited as CP.
Ruesch, J. and W. Kees (1956), Nonverbal Communication: Notes on the Visual Perception of
Human Relations, Berkeley: University of California Press.
Scarantino, A. (2017), ‘How to Do Things with Emotional Expressions: The Theory of
Affective Pragmatics’, Psychological Inquiry, 28 (2–3): 165–85.
Sebeok, T. A. (1977), ‘Ecumenicalism in Semiotics’, in Sebeok (ed.), A Perfusion of Signs,
180–206, Bloomington and London: Indiana University Press.
Sebeok, T. A. (1991), Semiotics in the United States, Bloomington and London: Indiana
University Press.
Sebeok, T. A. (1998), ‘The Estonian Connection’, Sign Systems Studies, 26: 20–39.
Sebeok, T. A. (2001a), Global Semiotics, Bloomington and London: Indiana University Press.
Sebeok, T. A. (2001b), ‘Nonverbal Communication’, in P. Cobley (ed.), The Routledge
Companion to Semiotics and Linguistics, 14–27, London: Routledge.
Self, C. (2013), ‘Who’, in P. Cobley and P. J. Schulz (eds), Theories and Models of
Communication, 351–68, Berlin: de Gruyter.
14 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Sperber, D. (2008), ‘Claude Levi-Strauss, a Precursor?’ European Journal of Sociology/Archives

Européennes de Sociologie/Europäisches Archiv für Soziologie, 49 (2): 309–14.
Stjernfelt, F. (2014), Natural Propositions: The Actuality of Peirce’s Doctrine of Dicisigns,
Boston, MA: Docent Press.
Waugh, L., Catalano T., K. Masaeed, Do T. Hong and P. Renigar (2016), ‘Critical Discourse
Analysis: Definition, Approaches, Relation to Pragmatics, Critique, and Trends’, in A.
Capone and J. Mey (eds), Interdisciplinary Studies in Pragmatics, Culture and Society,
71–136, New York: Springer.
Wharton, T. (2009), Pragmatics and Nonverbal Communication, Cambridge: Cambridge
University Press.
Xie, C., F. Yus and H. Haberland, eds (2021), Approaches to Internet Pragmatics, Amsterdam:
John Benjamins.
 ­C HAPTER ONE

Communication Theory
and Semiotics
RICHARD L. LANIGAN

‘For all human beings, and only for human beings, language is the vehicle of mental
life and communication’, says Roman Jakobson as he begins the task of suggesting just
how communication theory and semiotics relate to one another as verbal comportment
(1972; Holenstein 1974). While it helps enormously to view what is semiotic about
communication and how semiotics assists our understanding of communication (Nöth
2014), the issue of the current moment is why they must be explicated together as we
encounter our futures in the symbolic systems of our cultures (Whorf 1952; Ruesch
1967; Wilden 1972, 1987; Lotman 1994). The emergent answer is the human trait of
wonder – the experienced uniqueness of speech expression wherein the perception of
language founds an ability to express the mind’s consciousness (Fuchs 2018). The moment
and event of wonder (ékstasis) are our own human embodied mental life (consciousness)
and the awareness (experience), as Charles Peirce (1902: CP 3.621) would say, that others
also have this communication capacity to intertwine expression as perception (mental
causation) in the modality of discourse tropes of meaning (Shapiro 1988; Lotman 1990;
Rota 1997; Ruthrof 1997; Ehring 2011; Bondi 2012).
Human beings have the capacity to first learn a language from others, yet use that
very tool to identify both the similarity and the difference between one’s self and the
other – as object or not-me person (Bühler 1934; Ruesch and Bateson 1951). In a
process of auto-communication (Lotman 1991; Jakobson 1960b), the act of speaking is
simultaneous with that of hearing and language is developed as performance (verbal and
gestural). Yet this experience occurs after hearing others speak and seeing them gesture
to us, when language is learned as embodied competence (expression and perception)
(Britton 1939: 2; Ruthrof 1997, 2000, 2020). Competence we know as culture (our
entry into all semiotic systems) just as we understand performance to be the awareness
of self-identity or mind (our entry into all communication systems: linguistics, logics,
mathematics).
Because these two basic sign-systems of mind and culture are simultaneous in human
beings, i.e. in logic and language (Sapir 1931; Ruesch 1953; Alexander 1988), human
symbolic capacity stands apart from machines (they must be built and maintained by
humans) and apart from animals (who lack the ability to move referents in space-time
[mediate by symbolic forms]) (Wiener 1948a, b, 1950; Fuchs 2018: 22, 167, 170; Lanigan
2013). It is Ernst Cassirer (Urban 1949b; 1957; DePalo 2010; Chang 2013; Lanigan
16 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

2017a, b) who proposes that the phenomenology (conscious experience) of being human
consists in three capacities:
1. The ability to perceive objects (the world of icons investigated by natural science);
2. The ability to perceive expressions (the world of indices explored by human science);
3. T
­ he ability to communicate by speaking (the world of symbols created and learned
by human beings).
Human beings have the unique capacity to combine these abilities as the ‘art and science
of . . . ’ (Ruesch 1953; Eco 1976). As Fuchs (2018: 22) summarizes, ‘we do not perceive
stimuli or images, rather gestalt units, meanings and affordances. Perception avails itself
of the mediating process, in order to establish a direct relation to things – in other words,
a mediated immediacy.’ The paradigm tool for exploring this art and science combination
is the Discourse Model of Communication (symbolic meaning as the medium for both
Addresser and Addressee) that is prior to, and contextualizes, the Exchange Model of
Information (signal signification as the channel between either Sender or Receiver).
The discourse model is generally known by the disciplinary name Communicology
(Hunt 1970; DeVito 1978; Lanigan 1992, 2018a), while the exchange model is called
Informatics, often more a ‘computer metaphor or metonymy of the mind’ vaguely labelled
as inorganic ‘computer science’ or organic ‘cognitive science’ (Jakobson [1962] 1981;
Marcus 1974; Gleick 2011).

COMMUNICOLOGY MODELS
While it is a commonplace assumption that ‘communication’ and ‘information’ are
synonymous, we must realize that this is a scientific vulgarity comparable to saying today
that ‘chemistry’ is synonymous with ‘alchemy’ (Gleick 2011). Our analysis will not explore
the mathematics of ‘information theory’ (or its statistical cousins ‘computer science’ or
‘cognitive science’), inasmuch as ‘information’ is simply a numerical-indexical referent
for an object. And, only human beings create such symbolic reference systems (linguistic
communication) for objects in the natural world within which they live (Lanigan 2013).
Making this judgement is an example of what we shall explore: how human beings
observe (mediate) their own behaviour (immediate) in the moment (time binding) as a
meaningful event (space binding). Moments account for our consciousness and events
record our experiences as signs (sēmeion). Moments and events are a continuing process of
observation known simply as intentionality (awareness of consciousness as the experience
of living). Human development founds a series of learning stages wherein the perception
of others constitutes a sense of similarity and difference from which we abstract an idea of
who (nóēsis) and what (nóēma) we are (Dreyfus 2009). Importantly, the combinatory idea
(symbolon) guides our understanding of moments (index) as past, present, future and of
events (icons) that name [ónoma] what is taken as mediated self–other (measure) and
immediate same–different (magnitude). The best example in the human Umwelt is (1) the
transmutation from real sound (speech) to symbolic sound (language), which we know as
speaking and writing, and (2) the translation of one symbol system into another, which we
call linguistics and mathematics (Holenstein 1974: 159; Powell 2017). As Ernst Cassirer
(1942: 3) summarizes, ‘Speech and writing are considered as the origin of measure; for
they are characterized above all by the ability to fix the fleeting and variable and to
remove it from the accidental and arbitrary’.
COMMUNICATION THEORY AND SEMIOTICS 17

The human observer model

As in any scientific publication, I shall use figures to condense and present large amounts
of factual material, the evidence from which we make judgements. In particular, figures
are intended to visually simplify complex relationships of process and provide tabular data
categories for further exploration in the references given. In addition, I should note that
there is only one comprehensive theory of human communication (Roman Jakobson);
yet, there are a great many models of aspects of the theory. A theory is a comprehensive
narrative account that purports to explicate a complete depiction of a process used for
future prediction. A model is a selective account (often just pictorial) of a few variables
abstracted from the theory. The only complete model for a theory is the textual narrative
explicating it (what you are reading right now; or for example, see Alexander 1988).
Some parts of a narrative can be condensed into content tables; other parts are connected
by forms of dynamic relationship requiring visual figures. Please be aware that just as a
text is a continuous narration, sequential figures cross-refer to one another and are a
progression of thought complexity grounded in logic.
The Observer Model of Human Communication was proposed by Jürgen Ruesch
and Gregory Bateson (1951). Their book articulates a human-centred view of science
discussed at the 1950s New York Macy Conferences by a group that included Bateson,
Ruesch, Margaret Mead, Norbert Weiner, Ernst Cassirer, Chiam Perelman, Alfred
Schütz, Charles Morris, Claude Lévi-Strauss and Roman Jakobson (Lanigan 2018b). All
were concerned to account for the full range of observation from the mind inside the
human to the body outside, from psychiatry (Ruesch) to cultural anthropology (Bateson),
from logio-mathematics (Weiner) to linguistics (Jakobson), from rhetoric (Perelman) to
poetic (Jakobson) in discourse – all combinatory symbolic forms (Lanigan 2017a, 2018b;
Rota 1997). Ruesch and Bateson (Figures 1.1 and 1.2) suggest that human observation
begins ‘within one’ as the embodiment of consciousness designated as Intra-Personal
communication (Level I). This level of language (langage) defines the symbolic systems
(codes) learned by a human being as a result of being born into a given culture. Observation
is a search for similarity that is constricted by tone of voice heard (gērys). The child learns
by measuring the future channel utility of particular sounds (verbal functions) and their
contextual facial expressions (nonverbal functions). The child begins to experiment with
the magnitude of expressive production (rhetoric) of these sounds and the response they
get (network levels as a medium of communication).
These initial events lead to Inter-Personal behaviour (Level II) where ‘one to one’
dialogue emerges. The priority for the child is the discovery of the Self, finding one’s
own voice as speaking (parole). The task is sometimes difficult in a bilingual family, but
it is universally difficult with parents whose private language ‘baby talk’ is useless and
confusing compared to the soon-to-be-needed ‘public talk’. Interpersonal communication
is distinguished by the ability to recognize the ‘one-to-one’ difference (perspectives)
between the Self and the Other person as a voice of embodiment in conversation.
Public talk (langue) engaging other people in a social group constitutes Social
communication (Intra-Group Level III) as an intra-group function. At this stage, culture
makes a noticeable entrance into conversational events in the form of social preference for
human actions and their names. Cultures have a preference for how groups are organized
and graded with respect to activity. Some cultures prefer an egocentric focus (individual/
identity by rule, e.g. gender designation) where people are an aggregate collection (e.g.
United States; ‘one to many’, ‘a bag of marbles’). Other cultures prefer a sociocentric focus
18 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.1 Ruesch and Bateson discourse model of communicology.

(group/identity by role, e.g. family or professional) where there is an organic collection

(e.g. PR China; ‘many to one’, ‘a pile of salt’).
When and where groups begin to encounter one another, we have the discovery of
human difference manifest as wholistic, unified communicative preference (discours)
when people engage in group-to-group interaction. Cultural communication (Inter-
Group Level IV) is the highest level of complexity inasmuch as social preference expands
to cultural choice adherence. Differences between persons become differences among
groups by preferred affordance (measured magnitude). Groups organize by choice of
rule (egocentric; space binding) or choice of role (sociocentric; time binding) for making
analogue choices about ‘normal, acceptable, good, bad, evil’ behaviour (cultural mores).
Culture becomes a reality guidepost for ‘many to many’ future interactions. Examples are
commonplace on both a positive (‘peace’) and negative (‘war’) note.
A word of caution here, the comparisons and contrasts that I depict are preferences
meaning that all cultures have affordances for individuals and groups. Their practices are
simultaneously social and cultural, so that choice is a matter of habitual practice carried
COMMUNICATION THEORY AND SEMIOTICS 19

FIGURE 1.2 Category description of Ruesch and Bateson model.

forward as anticipated future actions. Echoing Charles Peirce, Norbert Wiener (1915:
570, 574) long ago argued about meaning functions that ‘the life of every branch of
mathematics lies in a habit’ and those habits are perceived first as ‘words not symbols’.
Communication as discourse is an attempt to perceive those actions (words – embodied
speaking) as expressions (symbolic formations; Gestaltung). For Cassirer, the habits of
culture are the source of all symbolic forms: language, art, myth and religion (Lanigan
2017a, b). Peirce (1906: 196–7) calls such forms the commens of mind constituted by
communication or simply commind as ‘double consciousness’ (1897: CP 1.153).

The human discourse model

Linguistic anticipation is our orientation to the unknown (ágnōstos), to the future.
Expression is an anticipation that someone will engage in perception. Our tool use of
language assumes a hierarchy of utility for such expressive and perceptive behaviour
(Figure 1.3). These expressions are typically spoken as oral messages where we get our
sense of connotative or informal meaning, or they are written texts with a code or style
that suggests a more formal denotation. The result is our sense of personal narrative
expression (my story, spoken by me) compared to a public narrative perception (my
history, spoken by others). As communication forms, these functions can be combined in
autobiography (story of my history = same-self = discourse trope) or biography (history
of my story = other-different = discourse figure). These expressive levels or registers
(Figure 1.3) have a definite social and cultural value implication about us as a person.
20 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.3 Discourse communicology model.

We use the register to orient our communicator perspective as either the person speaking
(Addresser) or as the one listening in a communicatee perspective (Addressee).
How the speaking is encoded or nominated by the Addresser (metaphor = A—as—B)
is then decoded or predicated in reverse order by the Addressee (metonymy = B—is—A).
This binary analogue interaction we tag as metaphor in the use of analogy (Ruthrof 1997:
172–4). A non-technical account of this concept of a logic constituted by founding tropes
in speech is the semiotician Yuri Lotman (1990: 36–53) and the mathematician Rota
(1997: 188–9). The more detailed explication is Ehring (2011) and the relation to Hegel’s
foundational logic as the semiotic dash [—] is Comay and Ruda (2018). Nonetheless,
keep in mind that Aristotelian logic per se derives from the consideration of oral rhetoric;
hence, tropes become informal (rhetoric) and formal (logic) laws of analogy to which we
adhere as beliefs (Lanigan 2014, 2019b, c; Houser 2020).
COMMUNICATION THEORY AND SEMIOTICS 21

However, before we can explore this founding trope logic of chiasm in its dynamic
process (A : B :: b : A = A as B, if b is a), it is helpful to preview the complexity system
of communication upon which we are embarking. Figure 1.4 summarizes where we have
been (choice : preference :: habit : affordance) and where we are going with logic and
semiotic models. We begin with Husserl’s (1948: 115, 116, 154) concept of the Founding
Logic [Fundierung] of Chiasm, more commonly called tropic or rhetorical logic (Lotman
1990). Figure 1.5 gives the working definitions that correlate to the semiotics of Charles
S. Peirce ([1866–1913] 1931–58) that also will guide us.
For context, Figure 1.6 illustrates the Scholastic trivium definitions of logic, rhetoric
and grammar that are the historical concept base for all contemporary theoretical work

FIGURE 1.4 Complexity description of semiotic systems.

22 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.5 Etymology of C. S. Peirce logic typology.

in communicology, but especially for Peirce, Husserl, Heidegger and Merleau-Ponty.

Other popular authors who worked in this tradition are Umberto Eco (1976) and Michel
Foucault (Lanigan 1992). Noting the ‘rhetorical branch of linguistics’, Roman Jakobson
also uses this foundation logic to specify the discourse model of communication as the
symbolic form (‘text’) of tropes (speaking) and figures (writing). As noted in Figures 1.3
and 1.6, for Jakobson the master tropes are metaphor (qualitative meaning) and metonymy
(quantitative signification) as interactive combinatory elements (poetic function) in every
register of communication (metalinguistic function); this is also Heidegger’s analysis of
poiēsis (1929: 63; cf. view of Heidegger in Britton 1939: 170).

The Basic Model of Georges Gusdorf

Gusdorf was a philosophy professor in charge of exam tutoring at the famous École
Normale Supérieure (ENS) in Paris. He began his career by succeeding Maurice Merleau-
Ponty in the job. He is famous for writing a small tutoring book called Speaking [La Parole]
COMMUNICATION THEORY AND SEMIOTICS 23

(1957) used by generations of French scholars, most notably Louis Althusser and Michel
Foucault (Bondi 2012, 2014). The book outlines the philosophy of speech foundational
for the analysis of all human sciences (Buyssens 1943, 1968; Swiggers 1987; Mulligan
1990; Rollinger 2009, 2010). Gusdorf (1966–93) also authored a twelve-volume history
of the human sciences during his lifetime for which this book is the key. The basics of this
wonderful little book about speaking are summarized in Figure 1.3.
The figure displays a hierarchy of perception, who a listener encounters in speech.
Recall this is a decoding process in which the trope of metonymy is operating with
a whole analysed by its parts. The cultural criterion of language [langage] is used to
identify the speaker’s code and the judgement value for this evidence is difference. The
conversational listener is always thinking by analysis, i.e. divide into parts, assemble
into wholes.
For example, does this person speak my language? If I speak English, I recognize
English, otherwise the speaking is noise. But then this discovered similarity of code
continues to suggest difference, e.g. the tone or accent is new to me, some of the
words are not known to me, etc. This is group identity; the speaker is using American
English; I know because I learned a different version, British English. So, we, speaker
and listener, are now the same, but different! We share a cultural logic in language
(code), but our grammar is showing signs of difference (context). This American
is using lots of personal (informal) references (‘I, me, you’) that make me slightly
uncomfortable (I prefer more formal reference [contact], that is what ‘one’ does).
Social usage [langue] is suggesting this American probably does not have much in
common with me. I start to perceive that a great deal of the speaker’s vocabulary
is adjectives, making most nouns incoherent by exaggeration! I have the British
preference for specific nouns and no adjectives. My exception for conversation
[discours] is not being met.
I am becoming very aware of how I speak [parole] and it is not like this! Now, my
oral experience is helped or hindered by the matching nonverbal codes that contextualize
the situation: how close is the American standing (Proxemics), how long will this take
(Chronemics), can I tolerate the way his clothing looks (Ocularics), will this person ever
leave (Kinesics), do I have to shake hands on departure (Haptics), can I tolerate the
accent (Vocalics), can I take the smell any longer (Olfactorics)? Every culture has exacting
affordance norms for the non-verbal codes that are not as flexible as the verbal codes,
especially when social preference operates inside the cultural context (the source of class,
caste, etc.) (Ruthrof 2000).
Figure 1.4 helps correlate the issues we are analysing. Our example took us through
the Gusdorf discourse model and introduced us to a working example of communication
network levels. For analytic purses, Husserl and Jakobson formalize our levels of conscious
experience as founding relations of implication. Humans organize these experiences as
levels of meaning or what Peirce calls typologies (universal categories) consisting of types
(particular wholes), their tokens (particular parts) and their tones (singular attributes).
In order to interpret the relationships of all the connected levels (gestalt), Peirce uses
the traditional division of logic into its types (according to its semiotic function in
communication): (1) ADduction (Interpretant), (2) ABduction (Symbol), (3) DEduction
(Index) and (4) INduction (Icon).
We need to recall that ever since the Greeks, logic and language have used triadic
(three parts; think grammatical subject, verb, object) relations of process that are called
appositions (each part related to the other two). If one of the parts is the context, then the
24 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.6 Scholastic trivium discourse model.

other two become an opposition. However, it is also critical to remember that appositions
also have a context which is the definition of a logic (qualitative) or a mathematic
(quantitative) called a quadratic (four attributes; think a ‘window frame’ 2 × 2 matrix).
The ‘Peirce Semiotic’ in Figure 1.4 is an example. While Figure 1.4 gives us a general
summary of our analysis so far, Figure 1.5 provides an accurate definition of Peirce’s
usage names and definitions over time so that his terminology is not confusing.

The trivium model of the Scholastics

Scholasticism was the dominant philosophical orientation in Europe from about 1100
to 1700 CE and was centred mainly at Paris, France, although there is a heavy German
influence including Cassirer, Husserl and Heidegger. The most notable persons were
Albertus Magnus and his assistant, Thomas Aquinas. These scholars were focused on
two goals:
1. How do you come to understand things (philological analysis)?
2. How do you communicate this knowledge to others (logical analysis)?
­ hey decided the key was listening to orators and reading books for evidence, then
T
writing out summaries that could be orally presented (lectures). Of course, these
written synthetic summaries and critical commentaries became the new (text) books
to be read and then discussed. How to do this efficiently? They devised a four-part
COMMUNICATION THEORY AND SEMIOTICS 25

method that applied to speaking (lecture format) and listening (note taking) called
explication de texte:
1. Text – Literal Meaning: Factual Summary of the content of a text (oral [lógos] or
written [lexis] evidence); Peirce’s term (see Figures 1.4 and 1.5) for this category in
Logic as expression is Case, and in Semiotic as perception is Symbol (Immediate/
Intentional Interpretant; Data = what is given).
2. Grammar – Moral Meaning: Description of what is perceived by typology (argument
type, token, tone); Peirce’s term for this category in Logic as expression is Result, and in
Semiotic as perception is Index (Dynamical/Effectual Interpretant; Acta = what is done).
3. Rhetoric – Allegorical Meaning: Reduction of the description to definition as semantic
tropes or syntactic figures; Peirce’s term for this category in Logic as expression is
Rule, and in Semiotic as perception is Icon (Final/Communicational Interpretant;
Verba = what is said) (Charaudeau 1983: 77–80; Lotman 1990: 36–44).
4. Logic – Anagogical Meaning: Interpretation by evaluation of the definition for the
future (hermeneutic judgment); Peirce’s term for this category in Logic as expression
is Representamen, and in Semiotic as perception is Interpretant (Commens/
Commind; Capta = what is taken).
A contemporary example of the method occurs significantly in the work of Maurice
Merleau-Ponty and Michel Foucault (Lanigan 1992: 89, 119, 127ff.). Of course, anyone
who has been to law or medical school encounters this method as learning by case studies,
the art foundation of doing science (Hunt 1970; Houser 2020). I should also note that
the case method is foundational to the Ruesch (psychiatrist) and Bateson (anthropologist)
model (Figure 1.1) where r/receiving (Self) is literal meaning, c/channel (Other) is moral
meaning, s/sending (Same) is allegorical meaning and e/evaluation (Different) is anagogical
meaning, and each semantic function is dominant in one of the communication Levels
(I to IV) of signification (Hunt 1970). The tabular information in Figure 1.6 illustrates the
basic organization of the Trivium as both:
1. A general application of philological interest in the code used by the Addresser and
the message understood by the Addressee in communication, whereas
2. The special application of logical (dialectic) interest is the medium of contact created
by the conjunction with the context for the message. The right-hand column in
Figure 1.6 notes the Jakobson communication elements and functions derived from
the trivium, and Figure 1.8 illustrates those categories as a process.

THE DISCOURSE COMMUNICOLOGY

OF ROMAN JAKOBSON
Roman Jakobson was a polyglot whose main work takes up nine large volumes in English,
French and several Slavic languages. Like others discussed thus far, he is at the centre of
work on language, semiotics and their application to the human sciences based in linguistic
theory. He brings the Ruesch and Bateson model (Figure 1.1) into the larger action
perspective (praxeology) of the human sciences on the one hand; yet, he also focuses on
the place of language within the hierarchy of those sciences of life: Linguistics, Semiotics,
Cultural Anthropology, Economics and Biology (Figure 1.7; tabular version is Figure 1.11).
26 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.7 Roman Jakobson communicology theory and applied model.

COMMUNICATION THEORY AND SEMIOTICS 27

Basic models of semiotics and communication

All this is another way of saying that the rhetoric of philology founds the logic of
philosophy. Jakobson’s Human Science Model provides a context or fame for his
Communicology Model. The elements or content categories of his theory constitute
the basic, unique parts of the human communication phenomenon. The elements and
functions are found at each level:
1. Intra-personal – the thought process in our own mind where the active voice of
consciousness is a simultaneous dialogue of Addresser (‘I’) and Addressee (‘Me’);
2. Inter-personal – the dyadic conversation between the passive voice of one Self and
an Other person (‘You [thee] / You [thou]’);
3. Intra-group – the social dialogue of middle voice among three or more Similar
people as the collective voice of a family or a work team (‘We / Us’); and
4. Inter-group – the interaction of mainly ethnic groups where cultures form and
encounter one another as Different (‘They/Them’) ways of giving particular voice
(a reality) to the experienced natural world of actuality (Lanigan 2015c).

Tropes: Complexity of elements and functions

What makes the communication elements discrete is their function, i.e. how each element
is perceived by the Addressee as an orientation, an indication of the future consciousness
(intentionality) of the Addresser. All human encounters begin with such nominated
expectations – future predictions that are meant to be descriptions of behaviour with
which we are familiar already in the present situation. The basic comparison (what
Husserl calls founding implication) that we engage with human communication is the
result of a shared code, both as expressed intentionality (Addresser representamen, to use
Peirce’s term) and as perceived interpretation (Addressee interpretant). Figure 1.8 gives
a static picture of this process event. The encoding process is the speaker’s attempt at
intelligibility for the present, while the listener’s decoding process is the counter-attempt
at relevance for the future. The whole domain of misunderstanding (ambiguity of past
referents) comes about because the speaker’s code-to-message construction sequence
must be reversed by the listener as a message-to-code deconstruction. Unfortunately, the
reversal brings along with it a tendency to now view the message/code (listener view) as
reflexive (message different from listener’s view) and reflective (code same as listener’s
view). Most adults discover this referent confusion phenomenon in conversation when
using pronouns, only to later (future) discover the assumed (past) noun referents of
speaker and listener are different. The semiotic point here is that the code gives us an
expression type, but the message gives us a perception token, and only the comparison
of both the type and the token allows a symbolic judgement of tone. Indeed, the semiotic
function of logic is the combinatory foundation of unity in difference (see the Fundierung
column in Figure 1.4).
While we have explored the complexity of message and code as elements and functions,
we must now consider several more levels of complexity. This amounts to saying that
we must take the basic pairs of Code /Message and combine them with Contact/Context
[note the way elements and functions are illustrated in Figure 1.10]. Then, we must
consider that all of them can interrelate as pairs (oppositions), triads (appositions) and
28 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.8 Jakobson communicology process model.

as quads (reverse oppositions and appositions) with equally interrelated functions. The
static list of elements and function in Figure 1.7 becomes a process matrix as illustrated
in Figure 1.9. For convenience, the matrix is called a chiasm and is represented by the
logic symbol .
It is helpful to gather the various parts of our analysis thus far into a summary that
is visually static in Figure 1.10. The manner in which the communication elements and
functions work from the practical perspective of thinking (logic), writing (grammar) and
speaking (rhetoric) constitutes Figure 1.11 (see Figure 1.7). Note here that Figure 11
COMMUNICATION THEORY AND SEMIOTICS 29

FIGURE 1.9 Jakobson chiasm matrix model.

depicts the integration of subjectivity (elements) and intersubjectivity (functions) so that

we become aware of what is called double articulation or ‘double judgement’ (Peirce
1893: CP 4.52–3). This is a way of saying that while in the actual world of experience,
there are not just two voices talking (speaker and listener) but four reality referent voices
that dialogue as Addresser (one embodied speaker [voice-1] and listener [voice-2]) and
Addressee (an other embodied speaker [voice-3] and listener [voice-4]). We know this
process as two people talking to one another while each person is thinking about what
the other person is saying and doing. Karl Jaspers calls this human conjunction of minds
the encompassing.
The four-part interaction is called the Perspectives Model of Communication; a
detailed account is Lanigan 2015, and the logic is detailed in Peirce (1903: CP 1.347 and
1.370). The chiasm square illustration in Figure 10 is an adequate visual representation
of the four perspectives for most purposes. For our immediate need, the following
30 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.10 Semiotic phenomenology matrix model.

quadratic schema of perspectives is a sufficient explication of the talk/thought semantic

field as in (1):
(1) Open : Blind :: Hidden : Unknown
Knowing (public) : Imagining (overt) :: Imagining (covert) : Thinking (private)
  ⬅ Known to Self and Others ➡ OPEN
   : ⬅ Not Known to Self yet Known to Others ➡ BLIND
    :: ⬅ Known to Self yet Not Known to Others ➡ HIDDEN
    : ⬅ Not Known to Self and Others ➡ UNKNOWN
COMMUNICATION THEORY AND SEMIOTICS 31

FIGURE 1.11 Communicology matrix of elements and functions.

The chiasm model of discourse

For most readers, it is enough to know that the founding trope logic of chiasm is a simple
ratio expressed as:
1. A : B :: b : a
Our ‘common sense’ version of the ratio is implication thinking by analogy (Rota
1997), whereby:
2. A as B if b is a
This analogical implication thinking (‘if’) is the Husserl and Jakobson source for
metaphor (‘as’) and metonymy (‘is’). Or, we should note its traditional French trope
version of le même et l’autre (Descombes 1979):
3. Self : Other :: Same : Different
32 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

The linear illustration is deceptive; keep in mind that (1) the four terms make up a
‘square’ (as in Figure 1.10, but see the Note in Figure 1.9 for visual help) and (2) the
four are a ‘mirror’ reversal (vertically and horizontally; Jakobson’s poetic function; see
[1962] 1981) such that ‘Self: Other’ outside the mirror is reflected backwards and upside
down inside the mirror as ‘Same: Different’ [the symbol :: would be the edge the mirror
glass] (Sonesson 2015). You experience this visual chiasm when you touch your face
while looking in a mirror and have trouble distinguishing ‘right side’ from ‘left side’, up
from down, since you forget to first pick a perspective of ‘inside’ or ‘outside’! There is an
extended cultural explication in Lanigan (2015c).
The concise point is this: a quadratic (four parts) is a combination that allows any
one term (one part) to be the context for the other three terms (three parts). The
context has a valence (positive/negative) so that the remaining triad (three parts) is
in an apposition. Any one of the three parts can take the opposing valence (negative/
positive) as a criterion to select compatibility (similarity) with one of the two terms
remaining as a dyad (making two parts). The chosen part now has the same valence as,
at least, one other part, leaving the last one part to have the reverse valence – by force
of division this part stands in opposition and is a monad. Thus, we have philosophical
terms of convenience, i.e. monad (singularity, ‘one’; Position; Peirce’s Firstness as
spontaneity [1892: CP 3.422]), dyad (particular, ‘two’; ‘Some’; Opposition; Secondness
as dependence) and triad (universal, ‘three’; ‘All’; Apposition; Thirdness as mediation).
Do not forget their context which provides the quad (universal particular, ‘fourth’;
Peirce call this an instance [1905: CP 4.537]). An analysis parallel to that of Peirce, but
in comparison to Husserl, is Ruthrof (2020).
Cultural preferences for the positive/negative valences of conduct create semantic
meaning (metaphor), while the quantitative syntactic signification (metonymy) creates
our pragmatic choices for future behaviour (affordance, comportment). We just used
the cultural tropic logic at work here, i.e. deictic expressions/perceptions such as inside/
outside, right/left, up/down. Each deictic expression (one part) displays (is the context
for) an apposition selection (one part) of the remaining opposition choice (two parts).
Precisely this semiotic process allows a human being to make a choice, yet ‘change
their mind’ by choosing an alternative context (a new quad, i.e. changing both logical
levels and semantic registers). Recall we started with a quad and choose one part; this
means we have three choices left, and we assign them a place in our memory called
present, past and future. You learned this process as grammar (rules for listening) then
learned to control it as rhetoric (rules for talking), all the time assuming it was the
correct logic to use (rules for thinking). The semiotic creativity of human beings is this
capacity to be wrong (Peirce’s Fallibilism), change our mind and create a different future,
because there is an absolute continuity (Synechism) and probability (Tychism) for our
conscious experience (Lanigan 2014, 2018b, 2019a; [see the levels of apperception in
Figure 1.13]).
Jakobson’s use of the chiasm discourse model is heavily contextualized by its
use among French scholars, most notably Merleau-Ponty and Kristeva on issues of
perception, and Foucault and Lacan on issues of expression (Kristeva 1974, 1981;
Lanigan 2018b, c; Bondi 2012, 2014; Lacan 1956). Following Edmund Husserl’s
work on logic (Holenstein 1974: 145) and Claude Lévi-Strauss’s work on chiasm in
semantic exchange (Lévi-Strauss 1958: 228; Lanigan 2017b: 210), Jakobson formulates
the chiasm in linguistic terms as Stawarska (2015: 185) summarizes: ‘Instead of a
distinction between deep structural level and the surface phenomenon of speech, there
is a crisscrossing or a chiasm between expression and language systems.’ The crossover
COMMUNICATION THEORY AND SEMIOTICS 33

poetic function occurs as the encoding/decoding reverse conjunction (see Figure 1.8)
between metaphor (the relation between substance and attribute, often called ‘content’
nomination) and metonymy (the relation between whole and part, often referred to as
‘form’ predication). The result is a chiasm implication model defined as:

Substance : whole :: part : attribute

Herein metaphor is a semantic (trope) relation of substance and attribute that is
mediated by metonymy as a syntactic (figure) relation of whole and part. Metaphor
is the paradigmatic/synchronic (‘vertical’ now/here) process of selection, substitution
and similarity. Metonymy is the syntagmatic/diachronic (‘horizontal’ there/then)
process of combination, contexture and contiguity (Holenstein 1974: 139, 159).
Jakobson ([1962] 1981) shorthands the reversible process explication by calling it
simply poetic function (transmutation; see Figure 1.3) and metalinguistic function
(translation) – the combination is synonymous with the Greek aitia process concept.
Note the hypercode here, the phonology of the word aitia itself is an iconic palindrome
chiasm: a - i — t — i - a.
Sebeok (1972: 17) argues in agreement, affirmed by Deely (1994: 71), that Jakobson’s
poetic function (semantics/temporality) and metalinguistic function (syntactics/spatiality)
are a coding capacity for judging logical ‘sameness or difference’ (chiasmatic exchange)
that is ‘exclusively human’ as a biological boundary between the human and animal
Umwelt (see Figure 1.7). The boundary is manifest as a habit in human language by Sapir’s
(1944: 23; Sebeok 1972: 23) concept of semantic grading wherein any process category
(capacity) is marked by analogue degree differences of both ‘mores’ and ‘lesses’ replacing
a ‘dichotomy with a continuum’, e.g. the English static case set: ‘good, better, bad, worse,
of average quality, interesting’ or causative case set: ‘to fall, to fell’ (1944: 125, 155).
Another example is the contemporary German da, meaning by analogue abessive case set:
‘not-here, there, here, because, in-order-to’.
Such a chiasm model illustrates the complete integration of the trivium categories
of logic, rhetoric and grammar as the ‘immediate inference’ of communication per se.
Lanigan (2017a, 2017b) is a summary of this long tradition, but for the technically minded
Figure 1.12 gives a summary of its use in semiotics, especially the popular Greimas Semiotic
Square that is built upon the Scholastic version of Aristotle’s aitia [αἰτία] metaphysics, i.e.
(1) explication by ‘mental causation’ implication (material, formal, efficient, final answers
[thematics] to questions [problematics]) and (2) explication by enthymeme as a chiasmatic
‘syllogism’ (Peirce 1893: CP 2.450; Lanigan 1974, 1995). For an applied example of the
Greimas theory and model, see Lanigan (2019d).
These explications are used by Peirce to account for his ‘argument cycle’ (Figure 1.5;
Lanigan 1974, 1995). Aristotle’s aitia is also tagged with the Husserl/Jakobson ‘laws of
foundedness’ (Figure 1.12). Figure 1.13 summarizes the over-reaching analysis made by
Franz Brentano, Edmund Husserl, Anton Marty and others with regard to the function of
apperception in the chiasm structure (Rollinger 2009, 2010). I should note that Jakobson’s
theory of marked and unmarked terms functions semantically as this chiasm relation,
thereby providing a systematic account of Karl Bühler’s (1934) analysis of anaphoric
and cataphoric deixis (DePalo 2010; Chang 2013; Lanigan 2018b). Last, Figure 1.14
summarizes the basic convergence of the classical and medieval view of the trivium as it
bears on Jakobson’s linguistics reflecting the influence of Husserl and converging with
Merleau-Ponty’s enduring work on chiasm when he prematurely died at age fifty-four in
1961 (Lanigan 1992: 119).
34 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.12 Matrix of the logic and semiotic squares.

MODELLING THE FUTURE: INTELLIGIBILITY

AND RELEVANCE
A discussion of the relation between chiasm and the combinatory future of communication
theory and semiotics necessarily begins with Maurice Merleau-Ponty. He takes up
Martin Heidegger’s discourse definition of human Being as a projection (Geworfenheit;
‘thrownness’ = Greek symbolon; Da—Sein, ‘there—Being’) into the future (the ‘law
of projection’ in Peirce (1892: CP 3.419; Ijsseling 1970; Rota 1997). Merleau-Ponty
(1964/1968: 170; Wayne J. Froman 2005) offers us these working notes:
‘a new effort of expression’, ‘Being in Heidegger’s sense’ [Dasein], ‘which, apprehended
by philosophy in its universality, appears as containing everything that will ever
COMMUNICATION THEORY AND SEMIOTICS 35

FIGURE 1.13 Chiasm process matrix for apperception and apposition.

be said, and yet leaving us to create it’ (Proust): it is the λόϒοϛ ένδίάϴϵτος [lógos
endiathetos; the speech that remains within; private narrative, interpretive] which calls
for the λόϒοϛ προϕορικόϛ [lógos prophorikos; the speech manifests without; public
narrative, revelatory] – –.
(my bracket inserts and Greek transliteration to Latin and English)

We already know this invoked semiotic methodology of existential discourse as

phenomenology and revelatory discourse as hermeneutics, respectively, the Scholastic lógos
and lexis. The simultaneous semiotic combination (metaphor) and division (metonymy)
36 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.14 Trope logic in rhetoric and semiotic.

COMMUNICATION THEORY AND SEMIOTICS 37

is precisely Jakobson’s use of chiasm, captured so well by Merleau-Ponty’s reference to

Proust. Figure 1.15 provides a contextual view of the binary analogue (unity of division
in combination) operation of everyday human communication, whether as encoding,
decoding or both together as the hypercode allowing simultaneous transmutation and
translation: Symbolicity.

­Binary analogue model of communicology

Husserl’s model of perception (core—field—horizon) allows us to see the existential
core (Self) as opposed to the revelatory field (Other), and the analogue boundary
of the horizon (Same/Different; lesser vs greater scale) that moves from low to high
intelligibility (left-to-right), yet simultaneously is balanced by an equal degree of
relevance (low to high) moving right-to-left. Compare Figures 1.8 and 1.9 to see the
process dynamic at work.
Figure 1.15 also incorporates the basic logic of founding implication by specifying the
basic elements of chiasm perception (a : b :: B : A) and chiasm expression (A : B :: b : a) as
they relate to the discourse hierarchy in Figure 1.3. The main communication components
of Figure 1.15 are intelligibility and relevance as ‘implicational argument’: ‘The possibility
of intelligible communication is the ultimate postulate of all thought, all knowledge. It
itself cannot be explained but is the presupposition of all knowledge and science’ (Urban
1929: 46). ‘Mutual acknowledgment on the part of transcendent minds of common
meanings and values is as much a condition of communication and genuine knowledge as
the independent being of the meanings and values thus acknowledged. Transcendence of
both Object and Subject is presupposed’ (Urban1949b: 147). Yet again, Peirce’s commind
is affirmed by Urban’s description of ‘double transcendence’ that echoes Husserl’s theorem
that ‘Subjectivity is Intersubjectivity’.
Intelligibility is a human foundational belief (performance; depiction) in discourse
meaning (lógos endiathetos): ‘Now, there are three “well-nigh invariable beliefs”
certainly necessary propositions of intelligible communication, namely, that I exist
and others like me, inhabiting a world.’ ‘They are not the outcome of reflection and
communication, but their co-implication’ (Urban 1929: 47–8; 1939: 634; 1949a: 144).
Encoding is our shorthand for naming discourse (vocabulary selection that nominates)
which creates reason [lógos in Figure 1.15], the capacity to be understood, i.e. allows
others to comprehend our meaning [páthos]. Decoding is the reverse process leading to
a determination, a judgement of reasonableness [eulógos]. ‘The condition of intelligible
communication is, indeed, logical form – an intelligible logic is one that must be
interwoven with our natural language’ (Urban 1929: 122–3). As Heidegger (1929: 136)
affirms, ‘lógos mean thinking. Lógos means the word, discourse, and legein means to
discourse, to talk’.
Relevance is a human foundational belief (competence; prediction) in discourse
signification (lógos prophorikos): As William James suggests, ‘we can choose which way
of thinking to adhere to and which to disregard’ (Schütz 1970: xiv). Here, decoding is the
parallel shorthand for implication (articulation selection that predicates). In this regard,
Schütz (1970: 70) suggests a model for relevance which moves perceptually from:
1. Topical connection to the point at issue [páthos], then to combinatory comparison of
2. Interpretation of meaning [eulógos], and then
3. Determination of value [ḗthos].
38 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

This is, of course, Peirce’s (1905: CP 4.538) sequence of type, token and tone, but
more specifically, his use of the Scholastic term, proposition and argument (he specifies
the ‘argument’ as the Greek dèlôma (‘revealed by speech’) which is contextualized by
communication (1902: CP 3.621; 1906). Heidegger (1929: 63–8) and Lanigan (2015b)
make this same analysis about dèlôma and its ‘gathering’ [naming] function in discourse
by saying: ‘Ónoma in the wider sense is dèlôma tèi phönéi peri tén ousian: a revelation
by means of [hearing tonal] sound in relation to, and in the sphere of, the Being of being’
(my bracket trans. insertion; Peirce defines tone exactly this way as hearing one’s unique
embodied voice [gērys] where ónoma literally means ‘made symbolic by speech’).

FIGURE 1.15 Communicology matrix of intelligibility and relevance.

COMMUNICATION THEORY AND SEMIOTICS 39

FIGURE 1.16 Discourse trope model of communicology.

Trope matrix model of intentionality

Recall the tropic structure of thinking, as Jakobson conceives it (Figure 1.9). With
Figure 1.16, we flesh out the various concepts of tropic structure developed thus far in
our analysis. The overall matrix arrangement displays the chiasm by grouping related
explications of causal mentality (tropes). Thus, the levels of communication (Figure 1.1)
are placed with those of trope description (substance, attribute; whole, part) as chiasm
ratios (A—B; b—a) and the logic correlation to Aristotle’s syllogism is noted. The schema
allows the easy recognition of types of tropic thinking: Metaphor/Paradox; Metonymy/
Dilemma; Simile/Irony; Asyndeton/Zeugma; Prosopopoeia/Synecdoche (see Figures 1.13
and 1.14 for definitions).
40 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

As such, each trope structure suggests a mediating analogue (more/less) combination

(both/and) context that, in turn, frames an immediate binary (either/or) division by
choice. While Figure 1.16 is a static illustration, it allows us to visualize why the horizon
boundary in Figure 1.15 is dynamic as a moving analogue (lesser/greater) boundary
between intelligibility and relevance meaning. For example, an Addresser may overcode
the message (‘talk too much, too long’, etc.) causing the Addressee to undercode the
message (increasingly ‘stop listening, loose focus’, etc.). Here, speaker presence turns
into listener absence leading to misunderstanding, misinterpretation and the like. The
speaker’s intentionality (aseity: conscious experience as self-sufficiency) fails as a symbolic
form for the listener’s intentionality (ipseity: conscious experience as self-identity).
While the symbolic form is shared as signification, the symbolic content constitutes a
meaning ambiguity (good) or paradox (bad) as lived (Peirce’s commind). ‘The content of
consciousness is a sign resulting from an inference’ (Peirce 1893: CP 5.313).
Our analysis began in the wonder of human language, the symbolic capacity to be
conscious of time and space, embodied and mediated, within our own mind and, yet,
immediately present within other people’s mind. As such, in the claim to immediacy, we
humans come to discover the chiasm transcendent moments and events wherein we may
communicate:
1. [a] Depict (encode) and
2. [b] Predict (decode) the future of
3. [B] Communicology (consciousness), and
4. [A] Semiotics (experience) as simply a continuing discourse expression:
A : B :: b : a.
The subjective and intersubjective matrix founds discourse as culture, if nature is
explicated in language by human beings. Communicology is human intentionality as
an embodied three-part discourse relation: ‘something means something for somebody’
(Fuchs 2018: 37). Perhaps Charles S. Peirce (1905: CP 5.506) says it best as an abduction:
‘No communication [Rule] of one person to another [Result] can be entirely definite,
i.e., non-vague [Case].’ Or even better, Martin Heidegger’s famous Dasein/Da—Sein
chiasm: ‘Discourse which Expresses Itself is Communication’ (Heidegger 1926: 211;
1939: 25, para. 27; Lanigan 2015b). Expressed as a Greimas Square (see Figure 1.12),
we have: Dasein [not-here/Being] ↑ Mitsein [being with] Da— [there] ↑ —Sein [Being];
note that the chiasm crossover means both past ‘because-motive’ [Folglewelt] and
future ‘in-order-to motive’ [Vorwelt] (Schütz 1970; Lanigan 1974), i.e. the human ability
to ‘change your mind’.
By way of conclusion, let me provide a summary definition of the Discourse Model of
Communication:
Communicology, as a founding logic, (1) names (ónoma) the wonder (ékstasis) of the
human mind (logismós) that is the transmutation (sēmeion) and translation (symbolon)
of the past (experience) into the future (consciousness) by meaning (discours, the voice
of the Other) and (2) expresses (parole, the voice of the Self) itself (langue, the voice of
Society) as communication (langage, the voice of Culture) in the continuing discovery
(Mitsein) that both not-here as time (Dasein) and there as place (Da—Sein) signify
understanding (dianoia) as being humane (logós/ḗthos/páthos). In short, we come to the
semiotic Theorem of Human Communication: Speech as Language, if a Sign is a Symbol,
more than a Signal.
COMMUNICATION THEORY AND SEMIOTICS 41

REFERENCES
Text page references are to the English translation where one exists.
Alexander, H. G. (1988), The Language and Logic of Philosophy, Lanham: University Press of
America. Reprinting of revised and enlarged edn by University of New Mexico Press, 1972,
of Language and Thinking: A Philosophical Introduction, Princeton: D. Van Nostrand Co.
Inc., 1967.
Bondì, A. (2012), ‘Le sujet parlant comme être humain et social’, Cahiers Ferdinand De
Saussure, 65: 25–38.
Bondì, A. (2014), ‘L’expérience de la parole: le thème du sujet parlante’, Texto ! Textes &
Cultures, 19: 1–19.
Britton, K. (1939), Communication: A Philosophical Study of Language, London, UK: Kegan
Paul, Trench, Turner & Co. Ltd.
Bühler, K. (1934), Sprachtheorie: Die Darstellungsfunktion der Sprache, Jena/Stuttgart: Gustav
Fischer Verlag.
Buyssens, E. (1943), Les Langages et le Discours: Essai de linguistique fonctionnelle dans le
cadre de la sémiologie, Brussels: Office de Publicité.
Buyssens, E. (1968), ‘Le Langage et la Logique. Le Langue et la Pensée’, in A. Martinet (ed.), Le
Langage, 76–90, Paris: Éditions Gallimard.
Cassirer, E. (1942), ‘Zur Logik der Kulturwissenschaften: Fünf Studien’, Götesborgs Högskolas
Arsskrift, 48: 1–139.
Cassirer, E. ([1957] 1995), The Philosophy of Symbolic Forms: Vol. 1: Language, Vol. 2:
Mythical Thought; Vol. 3: Phenomenology of Knowledge; Vol. 4: The Metaphysics of
Symbolic Forms, trans. R. Manheim, New Haven, CT: Yale University Press.
Chang, H. (2013), ‘Cassirer, Benvenisté, and Peirce on Deictics’ and “Pronominal”
Communication’, Sign Systems Studies, 41 (1): 7–20.
Charaudeau, P. (1983), Langage et Discours: Éléments de sémiolinguistique (Thêorie et
pratique), Paris: Hachette.
Comay, R. and F. Ruda (2018), The Dash – The Other Side of Absolute Knowing, New York:
MIT Press.
­Deely, J. (1994), The Human Use of Signs or: Elements of Anthroprosemiosis, Lanham: Rowan
& Littlefield Publishers, Inc.
DePalo, M. (2010), ‘Le “je”, la phénoménologie et le discours: Bühler, Benvenisté et Husserl’,
Beiträge zur Geschichte der Sprachwissenschaft, 20: 155–65.
Descombes, V. (1979), Le Même et L’Autre: Quarante-Cinq Ans de Philosophie Française
(1933–1978), Paris: Éditions de Minuit.
DeVito, J. A. (1978), Communicology: An Introduction to the Study of Communication, New
York: Harper and Row, Publishers, Inc.
Dreyfus, H. L. ([2001] 2009), On the Internet, 2nd edn, New York: Routledge; Taylor &
Francis Group.
Eco, U. (1976), A Theory of Semiotics, Bloomington: Indiana University Press.
Ehring, D. (2011), Tropes: Properties, Objects, and Mental Causation, New York: Oxford
University Press.
Froman, W. J. (2005), ‘Merleau-Ponty and the Relation between the Lógos Prophorikos
and the Lógos Endiathetos’, in A.-T. Tymieniecka (ed.), Logos of Phenomenology and
Phenomenology of Logos, 409–16, [Book One], New York: Springer.
Fuchs, T. (2018), Ecology of the Brain: The Phenomenology and Biology of the Embodied Mind,
New York: Oxford University Press.
42 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Gleick, J. (2011), The Information: A History, A Theory, A Flood, New York: Pantheon.
Gusdorf, G. (1953), La Parole, Paris: Presses Universitaires de France.
Gusdorf, G. (1966–93), Les Sciences Humaines et la Pensée Occidentale, Tomes 1–12, Paris:
Payot.
Hardwick, C. E., ed. (1977), Semiotics and Significs: The Correspondence between Charles S.
Peirce and Victoria Lady Welby, Bloomington, IN: Indiana University Press.
Heidegger, M. ([1926] 1962), Sein und Zeit, 7th edn, Tübingen: Neomarius Verlag, trans. J.
Macquarrie and E. Robinson (eds), New York: Harper & Row, Publishers.
Heidegger, M. (1929), Einführung in die Metaphysik, Tübingen: Max Niemeyer Verlag.
Heidegger, M. (1939), Die Geschichte des Seyns, Frankfurt Main: Vittorio Klostermann GmbH.
Holenstein, E. (1974), Roman Jakobson’s Approach to Language: Phenomenological
Structuralism [from German Habilitationschrift, Zurich 1974; French edn, Jakobson, ou sur
le structuralisme phénoménologique, Paris, FR: Editions Seghers, 1974].
Houser, R. E. (2020), Logic as a Liberal Art: An Introduction to Rhetoric and Reasoning,
Washington, DC: Catholic University of America Press.
Hunt, E. (1970), ‘Classical Rhetoric and Modern Communicology’, Western Speech, 34 (1): 2–7.
Husserl, E. (1948), Erfahrung und Urteil: Untersuchungen zur Genealogie der Logik, Hamburg:
Klassen and Govert.
Ijsseling, S. (1976), ‘Who Is Actually Speaking Whenever Something Is Said?’ in Rhetoric and
Philosophy in Conflict: An Historical Survey, 127–36, Boston: Martinus Nijhoff.
Jakobson, R. O. ([1962] 1981), ‘Linguistics and Poetics’, in Stephen Rudy (ed.), Roman
Jakobson Selected Writings III: Poetry of Grammar and Grammar of Poetry, The Hague:
Mouton Publishers.
Jakobson, R. O. (1960b, 1961), ‘Linguistics and Communication Theory’, in R. Jakobson (ed.),
Structure of Language and Its Mathematica Aspects, vol. 12; vol. 20, 570–9, Providence:
American Mathematical Society.
Jakobson, R. O. (1962–2002), Selected Writings, 9 vols: vol. 1, Phonological Studies, 1962, 2nd
edn 1971, 3rd edn 2002; vol. 2, Word and Language, 1971; vol. 3, Poetry of Grammar and
Grammar of Poetry, S. Rudy, ed. (1981); vol. 4, Slavic Epic Studies, 1966; vol. 5, On Verse,
Its Masters and Explorers, S. Rudy and M. Taylor, eds (1979); vol. 6, Early Slavic Paths and
Crossroads: Part 1 and Part 2, S. Rudy, ed. (1985); vol. 7, Contributions to Comparative
Mythology; Studies in Linguistics and Philology, 1972–82, S. Rudy, ed. (1985); vol. 8,
Completion Volume One: Major Works, 1976–1980, S. Rudy, ed. (1988); vol. 9, A Complete
Bibliography of His Writings, compiled and edited by Rudy (1990). Berlin: Mouton de
Gruyter. [Unless noted, all vols. edited by Jakobson.]
Jakobson, R. O. (1972, September), ‘Verbal Communication’, Scientific American, 227 (3):
37–44.
Kristeva, J. (1974), La révolution du langage poétique, Paris: Éditions du Seuil.
Kristeva, J. (1981), Le Langage, cet inconnu, Paris: Éditions du Seuil.
Lacan, J. ([1953] 1966), Fonction et champ de la parole et du langage en psychanalyse, vol. 1,
Paris: Éditions du Seuil.
Lanigan, R. L. (1974), ‘Enthymeme: The Rhetorical Species of Aristotle’s Syllogism’, Southern
Speech Communication Journal, 39 (3): 207–22.
Lanigan, R. L. (1988), ‘A Treasure House of Preconstituted Types: Alfred Schütz on
Communicology’, in R. L. Lanigan (ed.), Phenomenology of Communication: Merleau-
Ponty’s Thematics in Semiology and Communicology, Pittsburg: Duquesne University Press.
Lanigan, R. L. (1988), Phenomenology of Communication: Merleau-Ponty’s Thematics in
Semiology and Communicology, Pittsburgh, PA: Duquesne University Press.
COMMUNICATION THEORY AND SEMIOTICS 43

Lanigan, R. L. (1992), The Human Science of Communicology: A Phenomenology of Discourse

in Foucault and Merleau-Ponty, Pittsburgh, PA: Duquesne University Press.
Lanigan, R. L. (1994), ‘Capta Versus Data: Method and Evidence in Communicology’, Human
Studies, 17 (1): 109–30.
Lanigan, R. L. (1995), ‘From Enthymeme to Abduction: The Classical Law of Logic and
the Postmodern Rule of Rhetoric’, in L. Langsdorf and A. R. Smith (eds), Recovering
Pragmaticism’s Voice: The Classical Tradition, Rorty, and the Philosophy of Communication,
49–70; notes 278–83, Albany: State University of New York Press.
Lanigan, R. L. (2012), ‘The Two Senses of a Phenomenology of the Weltanschauung: An Essay
in Honor of Émile Benvenisté’, The American Journal of Semiotics, 28 (1–2): 63–72.
Lanigan, R. L. (2013), ‘Information Theories’, in P. Cobley and P. Schulz (eds), Theories
and Models of Communication, vol. 1, 58–83, Berlin, DE; Boston, MA: De Gruyter
Mouton.
Lanigan, R. L. (2014), ‘Charles S. Peirce on Phenomenology: Communicology, Codes, and
Messages, or, Phenomenology, Synechism, and Fallibilism’, The American Journal of
Semiotics, 30 (1–2): 139–58.
Lanigan, R. L. (2015a), ‘Semiotic Paradigms of Self and Person: The Perspectives Model of
Communicology as the Logic Foundation of Human Science’, Language and Semiotic
Studies, 1 (1): 106–29.
Lanigan, R. L. (2015b), ‘Philosophy of Communicology: “Discourse Which Expresses Itself Is
Communication” [Heidegger]’, Review of Communication, 15 (4): 332–58.
Lanigan, R. L. (2015c), ‘Semiotic Confusion in the Phenomenology of Perception West Meets
East, One Actuality Becomes Two Realities’, Chinese Semiotic Studies, 11 (2): 227–62.
Lanigan, R. L. (2017a), ‘Cassirer on Communicology: The Symbolic Forms of Language, Art,
Myth, and Religion in Cultural Semiotics’, The American Journal of Semiotics, 33 (3–4):
135–40.
Lanigan, R. L. (2017b), ‘Ernst Cassirer’s Theory and Application of Communicology: From
Husserl via Bühler to Jakobson’, The American Journal of Semiotics, 33 (3–4): 181–231.
Lanigan, R. L. (2018a), ‘Franklin H. Knower (1901–93): A Model for Communicology’,
Review of Communication, 18 (4): 391–413.
­Lanigan, R. L. (2018b), ‘Communicology Chiasm: The Play of Tropic Logic in Bateson and
Jakobson’ (Special Issue: The Play of Signs/The Signs of Play), Language and Semiotic
Studies, 4 (2): 67–92.
Lanigan, R. L. (2018c), ‘The Rhetoric of Discourse: Chiasm and Dialogue in Communicology’,
in R. C. Arnett and F. Cooren (eds), Dialogic Ethics, 215–43, Philadelphia, PA: John
Benjamins Publishing Co.
Lanigan, R. L. (2019a), ‘Applied Communicology in Organizational PR and R&D: Peirce on
Synechism, Fuller on Synergetics, Gordon on Synectics, and Alinsky on Socialism’, Language
and Semiotic Studies, 5 (4): 1–27.
Lanigan, R. L. (2019b), ‘Immanuel Kant on the Philosophy of Communicology: The Tropic
Logic of Rhetoric and Semiotics’, Semiotica, 227: 273–315.
Lanigan, R. L. (2019c), ‘Perelman’s Phenomenology of Rhetoric: Foucault Contests Chomsky’s
Complaint about Media Communicology in the Age of Trump Polemic’, Semiotica, 229:
273–328.
Lanigan, R. L. (2019d), ‘Crossing Out Normative Boundaries in Psychosis: The Logic of a
Social Semiotic Passage in Dickens’, Bleak House’, The American Journal of Semiotics, 35
(3–4): 335–64.
Lévi-Strauss, C. (1958), Anthropologie Structurale, Paris: Pion.
44 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Lotman, Y. M. (1994), The Universe of Mind: A Semiotic Approach to Culture, Bloomington:

Indiana University Press.
Marcus, S. (1974), ‘Fifty-two Oppositions between Scientific and Poetic Communication’, in
C. Cherry (ed.), Pragmatic Aspects of Human Communication, 83–96, Boston: D. Reidel
Publishing Co.
Merleau-Ponty, M. (1964), Le Visible et l’invisible, Paris: Librairie Gallimard.
Mulligan, K. ed. (1990), Mind, Meaning, and Metaphysics: The Philosophy and Theory of
Language of Anton Marty, Boston: Kluwer Academic Publishers.
Nöth, W. (2014), ‘Human Communication from the Semiotic Perspective’, in F. I.-San Juan
and T. M. Dousa (eds), Theories of Information, Communication and Knowledge: A
Multidisciplinary Approach, 97–119, New York: Springer Science & Business Media.
Peirce, C. S. (1906), March 9, 52-page draft letter to Lady Welby (under Robin L493, p. 200;
EP 2: 478), ms. pp. 24–30 excerpted in Hardwick 1977: 195–201.
Peirce, C. S. ([1866–1913] 1931–58), The Collected Papers of Charles Sanders Peirce, vols. 1–6,
C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University
Press. Cited as CP.
Powell, E. A. (2017 May/June), ‘When the Ancient Greeks Began to Write: Newly Discovered
Inscriptions Help Explain How Literacy Spread’, Archaeology, 3: 43–9.
Ruesch, J. ([1953] 1972), ‘Synopsis of the Theory of Human Communication’, in T. Sebeok
and D. J. Umiker (eds), Semiotic Approaches to Human Relations, 47–94, Paris: Mouton.
Ruesch, J. ([1967] 1972), ‘Social Control of Symbolic Systems’, in Ruesch, Semiotic Approaches
to Human Relations, 277–300, Paris, FR: Mouton.
Ruesch, J. and G. Bateson (1951), Communication: The Social Matrix of Psychiatry, New York:
Norton.
Rollinger, R. D. (2009), Austrian Phenomenology: Brentano, Husserl, Meinong, and Others on
Mind and Object, New Brunswick, NJ: Ontos Verla.
Rollinger, R. D. (2010), Philosophy of Language and Other Matters in the Work of Anton
Marty: Analysis and Translations, New York: Rodopi.
Rota, G.-C. (1997), ‘Three Senses of “A is B” in Heidegger’ [Husserl’s Fundierung]’, in
F. Palombi (ed.), Indiscrete Thoughts, 188–91, Boston, MA: Birkhäuser.
­Ruthrof, H. (1997), Semantics of the Body: Meaning from Frege to the Postmodern, Toronto:
University of Toronto Press.
Ruthrof, H. (2000), The Body in Language, New York: Cassell.
Ruthrof, H. (2020), ‘Phenomenology’, in Oxford Research Encyclopedia of Literature, New
York: Oxford University Press.
Sapir, E. (1931), ‘Communication’, Encyclopedia of the Social Sciences, 78–81, New York:
Macmillan.
Sapir, E. ([1944] 1949), ‘Grading: A Study in Semantics’, in D. G. Mandelbaum (ed.), Selected
Writings of Edward Sapir in Language, Culture, and Personality, 122–49, Berkeley:
University of California Press.
Sebeok, T. A. (1972), ‘Coding in the Evolution of Signaling Behavior’, in Perspectives in
Zoosemiotics, 7–33, Paris: Mouton.
Schütz, A. (1970), Reflections on the Problem of Relevance, New Haven: Yale University Press.
Shapiro, M. (1988). Figuration in Verbal Art, Princeton: Princeton University Press.
Sonesson, G. (2015), ‘The Mirror in-between Picture and Mind: A Phenomenologically
Inspired Approach to Cognitive Semiotics’, Chinese Semiotic Studies, 11 (2): 159–80.
Stawarska, B. (2015), Saussure’s Philosophy of Language as Phenomenology: Undoing the
Doctrine of the Course in General Linguistics, New York: Oxford University Press.
COMMUNICATION THEORY AND SEMIOTICS 45

Swiggers, P. (1987), ‘Eric Buyssens’s Les Langages et le Discours: A Functional Analysis of

Man’s Use of Signs’, in T. A. Sebeok and J.-U. Sebeok (eds), The Semiotic Web 1986, New
York: Mouton de Gruyter.
Urban, W. M. (1929), The Intelligible World: Metaphysics and Value, London: George Allen &
Unwin.
Urban, W. M. (1939), Language and Reality: The Philosophy of Language and the Principles of
Symbolism, London: George Allen & Unwin.
Urban, W. M. (1949a), Beyond Realism and Idealism, London: George Allen & Unwin.
Urban, W. M. (1949b), ‘Cassirer’s Philosophy of Language’, in P. A. Schilpp (ed.), The
Philosophy of Ernst Cassirer (LLP vol. VI), 401–44, Evanston: The Library of Living
Philosophers.
Whorf, B. L. (1952), ‘Language, Mind, and Reality’, Et cetera, 9 (3): 203–26.
Wilden, A. ([1972] 1980), System and Structure: Essays in Communication and Exchange, 2nd
edn, London: Tavistock Publications Ltd.
Wilden, A. (1987), The Rules Are No Game: The Strategy of Communication, New York:
Routledge and Kegan Paul.
Wiener, N. (1915, 14 October), ‘Is Mathematical Certainty Absolute?’ The Journal of
Philosophy, Psychology and Scientific Methods, 12 (21): 568–74.
Wiener, N. (1948a), ‘Time, Communication, and the Nervous System (Teleological
Mechanisms)’, Annals of the New York Academy of Sciences, 50 (4): 197–220.
Wiener, N. ([1948b] 1961), Cybernetics: Or Control and Communication in the Animal and the
Machine, 2nd edn, Cambridge: MIT Press.
Wiener, N. (1950), The Human Use of Human Beings: Cybernetics and Society, Boston:
Houghton Mifflin Co.
46
 ­C HAPTER TWO

Media/Culture Studies
and Semiotics
SOPHIA MELANSON RICCIARDONE AND MARCEL DANESI

INTRODUCTION
The emergence of media/culture studies as an autonomous discipline is traced to the
1940 work The Invasion from Mars: A Study in the Psychology of Panic by Princeton
psychologist Hadley Cantril. In it, Cantril investigated the likely source of the public
panic created by the 1938 CBS radio docudrama based on H. G. Wells’s 1898 novel
about interplanetary invasion, War of the Worlds, directed by Orson Welles as part of
his Mercury Theater of the Air Presents. The broadcast interspersed fake news reports of
Martian landings in New Jersey, which were so realistic that many listeners believed that
they were genuine, despite periodic announcements that the programme was a fictional
dramatization. Some citizens in the New Jersey area even fled their homes and phoned
the local authorities to report the purported landing. After interviewing 135 subjects,
Cantril concluded that better-educated listeners were more inclined to recognize the
broadcast as fiction than less-educated ones. The study was criticized on several counts,
since the statistical correlation between the radio broadcast, educational background and
the degree of reported panic was weak. Moreover, the panic may have been exaggerated
by subsequent media stories.
Despite the critiques, the implications of Cantril’s research could not be ignored by a
world saturated with mass media communications and culture. So, by the early 1950s, the
academic study of the relationship between mass media and mass culture (henceforward
abbreviated as media studies) started taking shape (Williams 1950; McLuhan 1951; Gerbner
1956), becoming an autonomous discipline by the 1960s. In 1957, French semiotician
Roland Barthes, along with his colleagues at the Centre d’Études des Communications
de Masse (the Centre for Mass Communication Studies, or CECMAS), brought out the
benefits of analysing popular mass media texts and performances (spectacles, movies,
consumer products, etc.) in semiotic terms, with his book Mythologies. In 1962, CECMAS
began publishing the journal Communications, which traced the semiotic dimensions of
communication and culture (Simonson and Durham Peters 2008). Although Barthes did
not name his approach in any specific way, eventually it came to be called media semiotics,
becoming a branch of both media culture studies and cultural semiotics.
This chapter will deal with the interface between general media studies and media
semiotics. It will cover several thematic areas including the disciplinary origins of both
fields of study; the main critical and socio-psychological approaches to the media; the
48 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

media semiotic approach that traces its roots to Barthes (classic media semiotics); and
semiotic views of digital media (new media semiotics).

ORIGINS
The Cantril study made it obvious that the mass media had psycho-behavioural effects on
individual consumers of media content. At first, the study of these effects was incorporated
as a target of investigation by psychology, sociology and anthropology. But soon after, the
importance of the mass media culture that was spreading broadly caught the attention of
scholars from across the academic landscape. This led eventually to the establishment of
media studies as an autonomous discipline, characterized by the integration of findings
and ideas emanating from the different fields (above). By the mid-1960s, departments
of media studies started cropping up on campuses across America, Europe and other
countries where the mass media played a critical role in daily life.
One of the first to examine the mass media in terms of its relation to technology
was the Canadian communications theorist Marshall McLuhan. In several key works
(e.g. 1951, 1962, 1964), McLuhan developed the theory that media are extensions or
amplifications of human faculties – biological, cognitive and emotional. So, for example,
tools such as an axe, a wheel and a telescope are media, in this sense, because they extend
(amplify) specific biological capacities – an axe extends the ability of the human hand
to break wood, the wheel of the foot to cover greater distances and the telescope of the
eye to see further. In this framework, communications media extend the ability of the
individual to go beyond the immediate environment, increasing the frequency, efficacy,
range and rapidity of communication across time and space. The advent of the electronic
mass media in the twentieth century further extended this ability and, in the process,
became a factor in changing the world.
McLuhan coined the catchphrase the medium is the message to convey that
communication and meaning-making are shaped by the nature of the communications
media we devise and use, because they condition the codes for expression and the
modes of experience that emerge within specific social environments for messages to be
conceived and delivered. He used the example of a light bulb to illustrate this. A light bulb
has no meaning or content in itself. However, it enables people to see in the dark and
thus creates ‘an environment by its mere presence’ (1964: 8) – a physical environment in
which people can carry out activities involving sight. Clearly, as environments become
altered by a medium, the modalities of meaning, expression and perception are adjusted
accordingly, thus influencing the ‘message’.
In his 1967 book The Medium Is the Massage: An Inventory of Effects, co-authored with
Quentin Fiore, McLuhan used the word ‘massage’, rather than ‘message’, to reinforce the
notion of media creating environments that directly affect human beings, as if they were
being given a massage. He elaborates as follows:

All media work us over completely. They are so persuasive in their personal, political,
economic, aesthetic, psychological, moral, ethical, and social consequences that they
leave no part of us untouched, unaffected, unaltered. The medium is the massage.
Any understanding of social and cultural change is impossible without a knowledge
of the way media work as environments. All media are extensions of some human
faculty – psychic or physical.
(McLuhan and Fiore 1967: 26)
MEDIA/CULTURE STUDIES AND SEMIOTICS 49

McLuhan’s work on media, which goes back initially to his 1950 book The Mechanical Bride,
showed how a culture cannot be studied apart from the dominant communications media
that it uses. As a result of McLuhan’s ground-breaking work, departments, journals, book
series and associations of media studies started burgeoning in the 1960s, 1970s and 1980s.
By the 1990s, the field had carved a significant niche for itself in academia. Today, media
studies is a flourishing discipline across the world, encompassing a broad range of theories
and interests. Within the discipline, branches have emerged focusing on ‘area-specific’
study – for example, television, radio, internet and video games. Of particular relevance is
the study of digital media and their effects on culture – a line of investigation that clearly
continues McLuhan’s work, albeit often in an unwitting and thus unacknowledged way.
As mentioned above, the foundational work for media semiotics, as separate from, but
integrated with, general media studies, was established by Barthes’s Mythologies. Guided
by Barthes’s contributions, CECMAS became a leading institution where the mass media
became the target of serious semiotic analysis – what was then termed semiology. Barthes
saw the mass media culture as a ‘bastard form’ plagued by ‘humiliated repetition’ and
generating a constant need for ‘new books, new programs, new films, news items, but
always the same meaning’, as he emphatically put it later (Barthes 1975: 24). For Barthes,
the content of mass media texts and performances (blockbuster movies, sports spectacles,
etc.) was a mere recycling of the classic myths (as will be discussed below). Despite
Barthes’s ideological take on mass media culture, it is his semiotic approach to mass
culture that has remained a significant point of reference not only in semiotics proper but
in media culture study generally.
The ideas of philosopher and literary theorist Mikhail Bakhtin, which predate those
of Barthes, going back to the late 1920s (see Bakhtin 1981), are now included under the
rubric of media semiotics retroactively. Bakhtin claimed that the more profane aspects
of modern mass culture are similar to the medieval carnivals, since they allow common
people to laugh and vent their frustrations as a unified community – in contrast to the
idealistic and individualist culture produced by the nobility and by artists of the Romantic
period. The language used at carnivals is polyphonic, exchanged between performers
and audiences with no regard to any pre-existent hierarchical social relations among the
speakers; it is a form of dialogue where everyone’s voice has value since there is no one
voice of authority imposing itself on the dialogical system. Bakhtin saw polyphony as
even a feature of great literature, from Dante and Rabelais to Goethe and Dostoyevsky,
because readers feel that they are in an internal dialogue with the author – a communicative
relation that became a key one within semiotics and literary criticism in the analysis of
literary texts a few decades ago.
Media semiotics today has spread its tentacles broadly, including within its purview
everything from film, television, video games, the algorithmic generation of signifying
content, the structure of websites to the evolution of new languages in cyberspace. Since
the late 1990s and into the millennium, it has produced several textbook-like surveys and
introductions (e.g. Jensen 1995; Nöth 1997; Bignell 2002; Danesi 2018). Overall, media
semiotics aims to explore the implicit or unconscious signifying structures and practices
present in mass media content. As an analytical and conceptual framework, it has proven
itself to be particularly well-suited to deciphering media effects on cultural trends and on
how meaning systems inherent in human codes, such as language, are transformed for
various purposes by the media. Like the other branches of the discipline, media semiotics
interweaves insights and findings from cognate fields in order to unravel the modalities
implicit in ‘mediated semiosis’ (Danesi 2018).
50 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

CRITICAL APPROACHES TO MEDIA STUDIES

Long before the rise of the academic study of mass media culture in the 1950s, social critics
of the Industrial Age were already assailing mass culture as being vulgar and profane. The
nineteenth-century British social critic Matthew Arnold (1869), for example, attacked
the mass culture of his era as insipid and homogenized, warning his contemporaries that
it was a threat to civilized society, turning the clock back to an undomesticated stage
of humanity. Arnold’s book, titled Culture and Anarchy, can be seen retrospectively to
constitute the initial work in what can be called a critical approach to mass culture. Known
today as the ‘mass society thesis’, Arnold’s main contention was that a mass culture based
on materialism and affluence had a deleterious effect on human growth and potential.
Also falling under the rubric of a critical approach is the 1922 book Public Opinion
by American journalist and essayist Walter Lippmann. Lippmann argued in it that mass
culture had a powerful negative impact on people’s minds and behaviours, producing
‘pictures in our heads’, and thus adversely affecting public opinion and politics, a view
espoused as well by Harold Lasswell a little later in Propaganda Technique in World War
I (1927). In the 1950s, this view was adopted by English literary critic Frank R. Leavis
(1952), who saw the spread of an ever-expanding mass media culture as a product of
banal bourgeois tastes and thus evidence of the decline of civilization.
But perhaps the best-known critical approach emerged from the Frankfurt Institute
for Social Research, founded in 1922, which condemned the mass-mediated cultures
of modern capitalist societies from a specific philosophical angle – Marxism. Members
of the School, which included Theodor W. Adorno (1941), Max Horkheimer (1947),
Leo Lowenthal (1949) and Herbert Marcuse (1972), characterized capitalist culture
as an anti-aesthetic one, merely reproducing art forms, such as music, as if they were
‘commodities’ to be sold and quickly discarded in the marketplace just like manufactured
commodities. As Adorno saw it, works of art in a mass capitalist culture, literature, music
and so on followed the same pattern of reproducibility. In contrast, true art ennobles the
human spirit and cannot be reproduced so easily. Despite the many counter-arguments
to Adorno’s views, they continue to have a broad intellectual appeal today, because of
their implications for studying new media, where cultural products can be produced,
consumed and reproduced instantly and collectively ad infinitum, and often take the
form of viral content on the internet. Adorno and Max Horkheimer coined the term
culture industry to characterize culture in a mass capitalist society – in analogy with the
industrial manufacturing of commercial products. Marcuse argued further that the mass
media indoctrinate and manipulate common people, promoting a false consciousness of
the self that plays into the hands of the system of power that oversees the social order.
Following on the coattails of the Frankfurt School was the Centre for Contemporary
Cultural Studies (CCCS), founded at the University of Birmingham in 1964 to study mass
culture from the standpoint of hidden discursive practices within it. The CCCS was one
of the first to study systematically the negative representation of underprivileged classes
and people. Leading members of the Centre included Richard Hoggart (1957), E. P.
Thompson (1963), Raymond Williams (1977), Stuart Hall (1978), Dick Hebdige (1979)
and Angela McRobbie (1991). Their focus was on the struggle that powerless people faced
in achieving a cultural voice. Under the leadership of Stuart Hall, the CCCS conducted an
important research project that led in 1978 to the publication of Policing the Crisis, which
showed how African Americans were misrepresented in the media and the negative social
consequences this entailed (for an overview of the CCCS, see Turner 2002).
MEDIA/CULTURE STUDIES AND SEMIOTICS 51

The CCCS was also instrumental in promoting feminist theories of media culture,
focusing on how representations, or misrepresentations, of women in the mass media kept
women subjugated and repressed as artists and free thinkers. This line of inquiry no doubt
had a profound impact on changing attitudes in how media representations of gender are
realized and in bringing about a greater gender-equality. In the 1970s and 1980s, feminist
critics argued that images of women were highly biased towards patriarchal views of
gender. But as society changed, by the 1990s the feminist approach had also changed,
leading to the so-called post-feminist movement, which did not see female sexuality in
media as exploitation, but rather as a transgressive form of representation against previous
and largely religious restrictions on the public display of women’s bodies (on feminist
theories of the era, see Van Zoonen 1994; Phoca and Wright 1999). However, social
commentator Ariel Levy (2005) has asserted that this particular brand of post-feminism
only implied an emancipation from the shackles of patriarchy through sexuality and
was, thus, a superficial enticement merely distorting reality. According to Levy, the mass
media has simply appropriated feminism and commodified the male perspective through
the production of a ‘raunch’ culture that seduces women into asserting their so-called
liberation by accommodating a highly sexualized model that is heavily contoured by a
nuanced projection of male desire. Recent events around the #metoo movement during
the second decade of the twenty-first century have begun to challenge these masculinized
framings of female sexual liberation that accompanied the mediation of ‘raunch’ culture
during the 1990s. The semiotic salience of the hashtag used to circulate awareness of
movements such as #metoo will be discussed later in this chapter in relation to memetic
(meme-based) internet culture.
The critical approaches of both the Frankfurt School and the CCCS are sometimes
classified under Hegemony Theory – a concept developed initially by Italian Marxist
Antonio Gramsci (1937) – which states that genuine culture under modern capitalism
is improbable, since media representations are typically nothing more than forms of
propaganda designed to pacify the masses and disguise social inequalities. In this system,
true artists and opposing voices are punished either directly (by incarceration) or indirectly
and thus more effectively (by critique and marginalization). The concept of hegemony
is inherent in another major critical approach, called Propaganda Theory, a framework
associated with Edward Herman and Noam Chomsky (1988), which maintains that those
in power, such as the government and media owners, influence how the media present
news coverage for the simple reason that they control the funding. In consequence, the
media tend to be little more than a propaganda arm of those in power or of particular
media owners. The media are thus set up to ‘manufacture consent’. They do this by
selecting the topics to be showcased, establishing the tone of the issues that are discussed,
and filtering out any contradictory information.

­CRITICAL APPROACHES TO NEW MEDIA

More recent critical scholarship in new media studies has revitalized former approaches and
philosophical framings of how power is distributed and negotiated across media networks.
Philosopher and sociologist Bruno Latour, for example, emphasizes the importance of
analysing non-human activities, such as computational operations, mediating networks
and the machinations of the political economy, in order to trace how they shape the
contours of social and cultural life. He refers to this sociological approach as the Sociology
52 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

of Associations and has put forth an associated theory, called Actor Network Theory
(Latour 2007: 7–9). Latour argues that to fully ascertain the nature of ‘power relations’
within society, one must trace the processes by which various associations are constituted
(86). Recall that McLuhan had made similar observations in his assertion that any medium
is an extension of human faculties, and will therefore transform the environments that are
inhabited, altering the cultural conditions and the social ordering of everyday life in the
process. This perspective aligns well with the general semiotic law of media which stipulates
that ‘ . . . as the medium changes, so too do the patterns of semiosis’ (Danesi, in Pericles
Trifonas 2015: 485). One may conclude, therefore, that as the various machinations of
new media influence the signifying order of power relations within society, the rhythms of
social and cultural life become altered. Thus, understanding how new media influence the
patterns of signification in everyday life is integral to ascertaining the conditions of social
and cultural existence at any given time.
Along a similar though perhaps more deterministic vein, sociologist Maurizio Lazzarato
(2014) borrows from Deleuze and Guattari’s conception of machinic enslavement to
describe the new media conditions outlined above. He remarks that a-signifying processes
of new media – the signalling qualities of algorithms and digital processes that have no
semantic dimension – capture human subjectivities within a technosocial machine, and he
explains that human subjects are rendered conceptually contiguous with machines within
the logic of the new media conceptual system (Lazzarato 2014: 27). Lazzarato outlines
the process through which new media operationalize traces of human behaviour online
as terminals for a-signifying, symbolic and signifying processes (96), and that within
such arrangements, ‘ . . . signs function in place of the object to which they refer . . . [and
through this] they influence subjectivity’ (100).
Also borrowing from Guattarian perspectives, communications scholar, Ganaele
Langlois (2014), put forward a mixed semiotic approach to ascertain how meaning
emerges from the confluence of material, social, technical, economic, psychological and
political processes, and proposes a theoretical framework to map the ways in which
various interactions coalesce into meaning-making processes. She argues that interactions
between disparate units within the circuitry of new media create meaning machines,
which render mediated forms of semiosis susceptible to the manipulative interference of
economic and political powerbrokers (19).
Guattari has suggested that the recent miniaturization of technology has enabled
powerbrokers to equip individuals with devices that permit increased management
of their perceptions and interpretations of meaning by plugging them into what
he refers to as the machinic phylum – an ongoing process through which singular
units belonging to disparate wholes are dismembered and assembled into something
else (Guattri 1995: 103). As new media hardware continues to contract in size, the
algorithmic machinations of new media that define the neuro-technological ordering
of signifying content online have become progressively more portable and increasingly
more accessible. Congruent with Guattari’s observations, cognitive semiotician
Merlin Donald (2010) notes that as intimacy between humans and their mediating
devices intensifies, society becomes increasingly dependent upon technology and its
mediating logic – what he terms exographic media – in making decisions and planning
various courses of action (2010: 76) on the micro- and macro-levels, some of which
undoubtedly impact the meaning, distribution and relations of power.
Interactions between technological circuitry, artificial intelligence (AI), artificial neural
networks, human digital and physical activity, human cognition and the political economy
MEDIA/CULTURE STUDIES AND SEMIOTICS 53

within cyberspace generate a myriad of simulacra – representations or abstractions of

the real – that through the affordances of algorithms signal the production, assemblage
and reproduction of signifying matter ad infinitum. Within these conditions, digital
impressions of our identities are abstractly contoured as simulacra (Baudrillard 1981)
of the embodied self, which are translated into bits and bytes, rendering abstractions of
the ‘self’ legible to the algorithmic (or, statistically salient) computations of new media.
The resulting digital transmutations of the self are what communications scholar, John
Cheney-Lippold, refers to as our measurable type (Cheney-Lippold 2017: 66) – a term
inspired by Max Weber and Erving Goffman’s conception of the ‘ideal type’. Cheney-
Lippold argues that these datafied, abstracted identities are constructed by algorithms
in the background of cyberspace according to the programmed priorities of commercial
and political powerbrokers without our knowledge, thus reinforcing existing monopolies
of power.

PSYCHOLOGICAL-SOCIOLOGICAL APPROACHES
The psychological analysis of the mass media, as mentioned, started with the Cantril
study, which led shortly thereafter to the first cognitive theory of media, called the
Hypodermic Needle Theory (HNT), which claims that the mass media can directly
influence minds and behaviours with an analogous impact to the one that a hypodermic
needle has on the body. In its initial form, HNT was critiqued substantively, although
recent versions that apply to digital media have been put forth, such as the idea that
we have become overly dependent on digital media or psychologically primed by
media effects. People habituated to large doses of information and visual stimulation,
constantly navigating the Web, or being constantly involved in social media chatter,
tend to become psychologically dependent on these in ways that parallel substance
dependency. Indeed, research has suggested that lower working memory and reduced
attention control result from prolonged use of mobile devices (Unsworth, McMillan,
Brewer and Spillers 2012). Furthermore, Hadlington established a significant probability
that excessive use of the internet and mobile phones is linked with common cognitive
failures, such as physically walking into other people and objects (Hadlington 2015).
Another symptom is an abiding preoccupation with all aspects of the internet, including
compulsive anticipations of digital contact from online interlocutors. Research scattered
across medical and psychology journals is suggesting, in fact, that this kind of addictive
behaviour might potentially be having several negative effects (e.g. Carr 2011, 2015;
Greenfield 2015).
So, while a revised version of HNT is gathering momentum today, the original
one, which was associated with electronic media before the advent of the internet,
was critiqued at first by American sociologist Paul Lazarsfeld (Lazarsfeld, Berelson and
Gaudet 1948), who found that the media had very little (if any) ability to change or
influence people’s minds directly. A little later, Lazarsfeld and Elihu Katz (1955) showed
that audiences react to media content as members of interpretive communities – families,
unions, neighbourhoods, churches, peer groups and so on. In such communities, there
are ‘opinion leaders’ (e.g. church ministers) who influence how the members will
interpret a media event. So, in contrast to HNT, which portrays media impact as a one-
step flow reaching a homogeneous and passive audience directly, the work by Lazarsfeld
and his co-researchers indicate that it is more accurately a two-step flow, as illustrated
in Figure 2.1.
54 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 2.1 Two-step flow model.

Katz and Lazarsfeld also argued that the mainstream media have a conservative social
function, rather than a disruptive one. For example, the representation of violence and
deviancy on TV crime programmes is hardly a causative factor of real violence or crime
in society, because such representations have a moral subtext built into them, warning
people about the dangers of violence and crime. This came to be known as Cultivation
Theory, which claims that the mainstream media ‘cultivate’, not threaten, the status
quo. The over-representation of violence actually reinforces an abiding respect for law
and order in most people. And even if there are copycat criminals, it was suggested that
they would be inclined to perpetrate their misdeeds anyway. In a related 1976 study,
George Gerbner and Larry Gross looked at the beliefs of habitual television viewers
and those of non-viewers, finding that awareness of violence was higher in the former
than it was in the latter. Those watching television seemed to believe that there is more
violence in society than there actually is. In a related previous study, Katz (1959) argued
that audiences use the media for their own purposes and gratifications, known as Uses
and Gratifications Theory. In 1984, Elisabeth Noelle-Neumann demonstrated that
exposure to the media impels many simply to go along with the opinions that the media
propagate. Known as Spiral of Silence Theory, she claimed that those who perceive their
opinion as being a minority one tend to remain silent fearing ridicule or marginalization
(Scheufele and Moy 2000). Thus, the media reinforce the majority opinion by ‘silencing’
dissenting voices.
One final psychological approach that implies an incorporation of some of the
implications of the HNT is Agenda-setting Theory, which claims that the mainstream
mass media influence opinions because they select which ideas and events are worthy
of public exposure. The choices then get transferred to public and political agendas –
an unconscious process called salience transfer. The theory was introduced by Maxwell
McCombs and Donald Shaw in a 1972 study of the impact of media coverage on the
1968 American presidential election. They found a correlation between the degree of
coverage and the rankings of the interviewees. In effect, voter opinions were conditioned
MEDIA/CULTURE STUDIES AND SEMIOTICS 55

by the quantity and nature of media information. Media emphasis thus translates into
public emphasis.
New media, which includes the internet and its related mediating technologies, tends
to be cognitive by nature. This cognitive dimension of new media, what Latour refers to as
mental software (2007: 3), introduces a new dynamic to mediation, in that it has become
highly participatory – we participate within the technological assemblages of mediated
content, and new media participates with us through the semi-autonomous automations
of algorithms. Canadian neuroanthropologist and cognitive semiotician, Merlin Donald,
asserts that technologically meaningful objects ‘ . . . can be said to have cognitive “lives”
inasmuch as they enter into the mind, solidify and channel shared experience and define
the social order . . . ’ (Donald 2010: 71). Donald notes that the proliferation of new media
produces significant revolutionary implications for human cognition since it supplies a
means of performing cognitive work collaboratively across hybrid networks that link
many minds with new technologies. He claims that as new media have produced hybrid
human-machine networks, the broader cognitive-cultures that society has constructed
have changed, producing a detailed division of cognitive labour (Donald 2010: 72).
Borrowing from Deleuze and Guattari, and in line with Donald’s insights, Katherine
Hayles argues that cognitive assemblages are the result of these hybrid human-machine
networks, and states that the interconnection between human minds and technical systems
affects the cognitive processing of mediated information on the conscious, unconscious
and embodied levels, as sensory systems carry mediated signals across the nervous system
(Hayles 2017: 118). Given the analytical rigour of seasoned semioticians, media semiotics
is likely the most appropriate intellectual means of assessing these perspectives and making
sense of how meaning is shaped within the confluence of cognitive assemblages rendered
within hybrid human-machine networks.

CLASSIC MEDIA SEMIOTICS

The type of semiotic approach to traditional media, from print to television and cinema,
can be called classic media semiotics, while the one that applies to digital and computer
media of various kinds can be called, instead, new media semiotics (see below). Even
though they share much theoretical ground, there are some techniques in the latter that
apply specifically to the emergence of the new media.
Classic media semiotics, as discussed above, began with Barthes’s Mythologies, in
which he argued that mass-mediated texts and spectacles do nothing more than recycle
ancient mythic themes, characters and settings, adapting them to the modern world.
So, for instance, the action hero Superman possesses the same kinds of superhuman
powers that an Atlas or a Hercules had; he also has a tragic flaw, like Achilles. Because
of the unconscious power of myth, such fictional media heroes become cultural icons,
symbolizing virtue, heroism and righteousness above and beyond the comic and movie
scripts in which they appear. They are recycled mythic personae. To distinguish between
the original myths and their contemporary media versions, Barthes designated the latter
mythologies. Barthes claims that the code (system of meanings) is the same, but its
textualization and representation are adapted and updated to fit the present day. Classic
media semiotics involves penetrating the mythological (coded) level of all kinds of media
texts and performances.
As a structuralist, Barthes adopted various techniques, such as the notion of opposition,
initially called différence by Ferdinand de Saussure (1916), to aid in the decoding of
56 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

media texts and spectacles. Essentially, this meant that if there was a hero in a story,
then there would also have been a villain in it, and the différence between the two comes
out in signifying elements (signifiers) such as their diverse appearances, behaviours and
so on. Mythic oppositions are found throughout media texts. Now, despite its obvious
utility, the structuralist approach was challenged as early as the 1960s, by the movement
called post-structuralism, spearheaded by the late French philosophers Michel Foucault
(1972) and Jacques Derrida (1976, 1978). According to Derrida, all sign systems are
self-referential – signs refer to other signs, which refer to still other signs, and so on ad
infinitum. Thus, what appears stable and natural turns out to be illogical, not oppositional
in the structuralist sense. In contrast to Saussure’s idea of différence, Derrida (1978) coined
the word différance (spelled with an ‘a’, but pronounced in the same way) to overtly
critique Saussurean theory. As Nesselroth (2007: 442) has remarked, Derrida’s approach
was one that ‘constantly decentres fixed meanings and puts into question the ontological
status of language (both written and spoken) and of communication in general’. Now that
the movement has subsided somewhat, it is becoming evermore evident that structuralist
notions, such as those used by Barthes, still have value in understanding all kinds of
mediated artefacts, spectacles, genres, etc. since they are based on oppositional codes that
recycle ancient mythic notions and language.
Another critique of opposition theory that emerged in semiotics at around the
same time as poststructuralism was that it ignored the role of metaphorical meaning in
representational activities – a critique dovetailing with the crystallization of cognitive
linguistics (Pollio, Barlow, Fine and Pollio 1977; Lakoff and Johnson 1980, 1999;
Fauconnier and Turner 2002). But as it turns out, the concept of conceptual metaphor
within cognitive linguistics is hardly incompatible with that of opposition. A simple
linguistic metaphor such as ‘My sister is a tiger’ is a token of an associative structure
that portrays personality in connotative (coded) animal terms (‘Sam is a gorilla’, ‘Sarah
is a puppy’, etc.). That structure is the conceptual metaphor, people are animals. Now,
conceptual metaphors are formed through image schemata, as Lakoff and Johnson
have cogently argued (Johnson 1987; Lakoff 1987; Lakoff and Johnson 1999). The
image schematic source for the people are animals conceptual metaphor seems to be
an unconscious perception that human personalities and animal behaviours are linked
in some way. In other words, it is the output of an ontological opposition: humans-as-
animals, constituting an example of how opposition manifests itself as an associative
phenomenon. In this case, the two parts of the opposition are not contrasted (as in
night vs day) but equated: humans-as-animals. This suggests that oppositional structure
operates in a non-contrastive way at the level of figurative meaning.
As mentioned, poststructuralism has receded, but it has left several interesting
ideational residues that are now incorporated into media semiotics. One of these is the
claim that media texts have rhetorical structure, which, as just mentioned, is covered by
conceptual metaphor theory. Actually, in a 1964 article, ‘The Rhetoric of the Image’,
Barthes had established a method for fleshing out rhetorical meaning in visual texts,
leading shortly thereafter to the emergence of Visual Rhetoric (VR), which has since
become a branch of media semiotics and other disciplines, from psychology to graphic
design (Barthes 1964/2004). The Belgian Group µ, founded in 1967, further developed
Barthes’s original ideas. Their 1970 publication A General Rhetoric reformulated classical
rhetoric in Barthesian ways, and in their Traité du signe visuel (1992) they elaborated
a rhetorical grammar of the image. More recently, Hawreliak (2018) has applied the
MEDIA/CULTURE STUDIES AND SEMIOTICS 57

notion of rhetorical structure to the multimodality of video games, showing that classic
media semiotics still has a significant role to play in the context of current digital media.
It should be mentioned that aspects of VR analysis, as envisioned by Barthes and
illustrated in his 1964 article, are consistent with Charles Peirce’s notion of iconicity
(Peirce 1866–1913) and how it shapes the understanding of many media texts, such
as common ads. Reductively, iconicity is defined as the process of making signs and
texts resemble their referents. Iconic brand names and logos, for example, dominate the
marketing world – Splash (detergent) evokes what is done with the product through
sound imitation (‘splashing’); the Polo logo represents the sport of polo visually with a
horse and a rider dressed in polo garb; etc. (Danesi 2018). The relation between Peircean
and Barthesian analysis of media is beyond the present scope. Without going into detail
here, suffice it to say that a combination of both approaches has characterized classic
media semiotics since its inception.

NEW MEDIA SEMIOTICS

Though the term new media emerged in common parlance alongside digital media in the
1990s (Manovich 2001: 31), the cyberculture within which we operate today has long
been percolating beneath society’s surface during its shift from a post-industrial state
in the 1970s (Bell 1976) to the network society (Castells 2011) of the late twentieth
century, which materialized fully equipped with a service and information economy.
AI, as the pulse of digital new media, was first conceived in 1958 by Frank Rosenblatt
and evolved throughout the 1960s to automate the statistical induction of probability in
order to detect patterns and make predictions about the world based on data fed into a
machine. However, as Minsky and Papert (1969) noted at the time, the AI theories and
technologies of the period were inadequate to fully accommodate the vision of what
AI might accomplish in simulating the neural networks of the human brain. Thus, AI
pursuits were temporarily suspended and the dominant trend in computer science until
the early 1990s was based on the expert systems model, which necessitated the tedious
task of manually programming specific knowledge into software. Dissemination of this
knowledge involved the integration of networks and the maintenance of cumbersome
databases. These dynamics of computer programming and software performance defined
the hypertexted, multi-mediated culture of the late 1990s and early 2000s.
The advent of the internet – what has been coined the Big Data revolution of the
mid-2000s – coupled with the development of powerful computer hardware for gaming
consoles in the 1990s provided the extensive data and processing speed needed to
reinvigorate new developments in AI (Hosanagar 2019), which today fuels what is called
new media. Contemporary forms of AI are equipped with deep learning algorithms and
artificial neural networks that have transformed the more static and anchored quality of
internet content at its nascence to the more kinetically adaptive and distributed content
that is curated and personalized for each individual user today. Herein lies the distinction
between mass media and new media: the homogeneous quality of mass-mediated popular
culture contrasts with the adaptive and customized quality of participatory digital culture.
This distinction will be expanded upon further below – it is a direct result of significant
technological transformations.
Media theorist Matteo Pasquinelli (2017) offers Peirce’s differentiation between
abduction and induction to supply insights into the potential limitations of new media
58 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

AI (2017: 12). Peirce discerns that inductive inference only determines ‘ . . . the value of a
quantity. It sets out with a theory and it measures the degree of concordance of that theory
with fact’ (1903: CP 5.145). Pasquinelli echoes Peirce’s proclamation that abduction is
much more dynamic and serves as the engine of creative insight and invention. Though
artificial neural networks perform a myriad of complex statistical inductions and may
even come close to accomplishing an anaemic form of abduction in producing statistically
constituted predictions, the human brain’s capacity for invention and creativity remains
out of reach for the artificial neural networks of AI. That said, within cyberspace, the
human brain responds to the signifying content and assemblages curated and assembled
by deep learning algorithms, and so the computational operations of AI remain a target
of semiotic analysis among contemporary theoreticians. In other words, the signifying
output of AI computations becomes real in its consequences as human minds interpret
the resulting assemblage of signifying matter projected through the user interface.
These interpretations, guided by mediated processes, are schematically mapped within
individual and collective memory. However, as Pasquinelli so aptly summarizes, ‘if pattern
recognition via statistical induction is the most accurate descriptor of what is popularly
termed Artificial Intelligence, the distorting effects of statistical induction on collective
perception, intelligence and governance . . . are yet to be fully understood’ (Pasquinelli
2017: 2). New media semiotics is tasked with the charge of understanding these effects.
In addition to the description of AI logistics offered above, the participatory and
convergent dimensions of new media also warrant some consideration. New media,
according to Henry Jenkins, is ‘convergent’ media, through which ‘old and new media
collide’ (Jenkins 2006: 3) and where content flows across a multitude of media platforms
that reproduce and remediate traditional mass media forms. Lev Manovich refers to this
process as meta-mediation – media that mediates other media forms (Manovich 2002: 32).
Such convergence certainly impacts the signifying order of content and the negotiation of
meaning within current new media environments. Convergent media also represents the
ways in which digital technologies enable the intersecting of media industries and media
consumers, who collaboratively generate the ephemera of internet content. This dynamic
has resulted in an effusion of participatory culture, which contrasts with previous notions
about the mass media as passive media, cultivating a docile spectatorship (Manovich
2002: 3). In consequence of this participatory dynamic of new media, the circulation
of signifying matter within cyberculture operates according to a logic of individual
customization rather than that of mass standardization (Manovich 2002: 51).
Within the participatory culture of convergent new media, humans collaborate in
acts of semiosis alongside computational codes and a-signifying orders, which generate
shifting modes of interpretation and tenuously distractible patterns for interaction. The
vacillating and variable nature of participatory new media cultivates conditions within
which the meaning of culturally constituted signifiers becomes destabilized where every
meaning-making instance is subjected to the influence of what Lévi-Strauss refers to as the
‘floating signifier’, which animates ‘the disability of all finite thought’ (Lévi-Strauss 1987:
63). This particular mode of suspending finite thought is perpetuated by what Heidegger
(2010) describes as a ‘ . . . restlessness and excitement from continual novelty and changing
encounters’ (166) brought into existence by transient experiences with signifying matter
online – a semio-technical simulation of ‘place’ that may be conceptually encompassed by
the term ‘cyberspace’.
Cyberspace, as experienced through the facility of user interfaces, is an interactive
and immersive virtual electronic space. The internet is a primary environment within it,
MEDIA/CULTURE STUDIES AND SEMIOTICS 59

in the McLuhanesque sense (above). It now has its own communities and its own set of
conventions for communicating and interacting. Movement and interaction in cyberspace
are ‘hyperreal’, to use Jean Baudrillard’s (1983) apt term. The modern human being lives
in two universes, that of physical reality and that of hyperreality, and the distinction
between the two is becoming more and more indistinguishable. Baudrillard introduced
the term simulacrum to describe this form of ‘amalgamated consciousness’, whereby
reality and fantasy mesh in an entrenched way.
Baudrillard’s concept of the simulacrum can be considered the starting point for new
media semiotics to emerge a decade or so later, since it prefigured how meaning unfolds
in cyberspace – as an unconscious blend of reality and hyperreality. While still based on
fundamental semiotic notions, new media semiotics has expanded its range of application
to encompass such phenomena as multimodality, algorithmic design, psychometrics,
micro-targeting, digital profiling, memes and memetic codes, and other specific aspects
of cyberspace (e.g. Mazzali-Lurati 2007; Warschauer and Grimes 2007; Albertazzi and
Cobley 2010; Canizzaro 2016; Poulsen, Kvåle and van Leeuwen 2018; Danesi 2019).
In addition to simulacrum theory, new media semiotics often incorporates McLuhan’s
Four Laws of Media into its modus operandi, since they seem to be particularly relevant
to the study of how media, technology and meaning-making codes intersect in the digital
universe (Danesi 2018). The laws are called amplification, obsolescence, reversal and
retrieval (McLuhan and McLuhan 1988). As discussed above, a new technological mass
communications medium will at first amplify some sensory, intellectual or other human
faculty. While one faculty is amplified, another is lessened or rendered obsolete – it is
amputated – until it is used to maximum capacity whence it reverses itself and is retrieved
in another medium.
An important target of new media semiotics is how AI is shaping meaning systems and
how we now read or experience texts (Walsh-Matthews and Danesi 2019). For instance,
augmented reality (AR), which is the technology that superimposes a digital image on
someone’s view of the real world, providing a composite augmented view of reality,
manifests how Baudrillard’s simulacrum concept is an applicable one to the world of
AI. AR projects elements of the consciousness that accrues in the virtual world onto the
perceptual mechanisms involved in the real world, by amplifying hearing and seeing. In
consequence of such technology, texts such as novels are being created in ‘augmented’ ways
so that the author is no longer the key element in the text’s interpretation or resolution – it
is the reader, who becomes enmeshed in the simulacrum. This has led to reconsiderations
of semiotic theories about the ways in which authors and readers are intertwined in a text,
such as those by Bakhtin (above) and the late Umberto Eco (1979, 1992).
We have clearly moved away from the Gutenberg Galaxy, based on print media,
into the ‘Matrix Galaxy’, as it can be called, in homage to the movie. Today, the ability
to decipher, create, understand and interpret digital texts, platforms and social media
venues meaningfully involves understanding the relation between reality and hyperreality
more than ever before. Digital literacy, moreover, is an extension of print literacy – again,
to use McLuhan’s key notion. Without such literacy today, people’s ability to function
in modern society diminishes considerably. It now encompasses a host of subliteracies,
including computer literacy, or how to use the computer to read and write, mobile
literacy, which involves the ability to create and understand texts on mobile devices,
virtual worlds literacy, such as Twitter, Instagram and Facebook communities, which
have developed new forms of language and textuality within them, and others (for an
overview, see Wong 2019).
60 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

A few thematic areas of media semiotic research include local and global discourse
structures, which reveal shifts in linguistic behaviours, including sentence typologies,
speech acts, turn-taking, text messaging, the use of hashtags, the virality of memes and
the contagious spread of content online. Studying local discourse structures involves
examining how the internal structures of linguistic and communicative competence
are responsive to traditional, real-world, social contexts, reflecting and encoding them
simultaneously. Studying global discourse involves examining the themes and topics of
conversations as they occur in digital contexts and how they are organized structurally and
deliver meaning through the new technological channels. Global discourse also involves
the kinds of speech genres that have developed online. For example, the Facebook
homepage, which allows people to update their activities and leave relevant messages
for visitors, follows specific stylistic conventions across the globe that identify it as such.
Instant messaging or text messaging reveals a compressed and multimodal style of
writing, including visual supports (emoji, GIFs) and abbreviated writing styles (lol, brb).
Social media discourse patterns also fall into the global domain, but they vary according
to social media platform – Facebook, Instagram, Snapchat, Twitter, etc. – each of which
has developed their own textual genres with a specific style (see Poulsen, Vigild, Kvåle
and Leeuwen 2018). Hashtag (i.e. #thisisahashtag) literacy is also becoming a primary
area of interest of new media semiotics within the global discourse frame. The hashtag
emerged in 2007 as a system for categorizing and tagging tweets. It slowly gained
traction, migrating to various social media platforms. Adding hashtags to a post enables
multiple posts to become threaded into a common network of related subject matter.
Hashtags can be used as well for finding users who are looking for topics on social
media. Hashtags also have a ‘commentary’ function. For example, a linguistic marker
referencing the target of evaluation in a tweet not only renders the content searchable,
but also constitutes a means to affiliate with values expressed in the tweet. This has led
some linguists to refer to hashtags as part of ‘searchable talk’, whose subtext is ‘Search
for me and affiliate with my values’ (Zappavigna 2019). It also implies that hashtags can
become ideological and politically subtle codes. In effect, hashtags are now intrinsic to
a subtle discourse strategy, constituting a cue for expressing meaning and tone (irony,
sarcasm, surprise, etc.).
Memes are signs or texts of short duration that are transmitted throughout the internet
virally (Canizzaro 2016; Danesi 2019). Meme theory provides a useful lexicon for
describing the ‘contagious’ spread of something through the internet, even though it fails
to explain why memes have a short life, whereas certain important ideas and movements
do not. As is well known, the term was coined by sociobiologist Richard Dawkins in
The Selfish Gene (1976) to explain how information gets passed on from generation to
generation in a specific culture in order to enhance survivability and promote progress
and further evolution by replicating the functions of genes. Culturally speaking, memes
are structures of information (ideas, laws, clothing fashions, artworks, tunes and so on)
and of behaviour (marriage rites, courtship rituals, religious ceremonies and so on) that
people inherit directly from their cultural environments. An internet meme may be an
image or a video, a hyperlink, a hashtag, a website or even just a word or phrase that
spread ‘virally’ via social networks, blogs and email. This implies that internet culture is
based on mimicry (memetic culture).
Some see memes as a new form of gossip or a bizarre word-of-mouth speech in a
global digital context, known as secondary orality. Primary orality, as Walter Ong (1982)
explained at the threshold of the digital age, refers to a culture in which information
MEDIA/CULTURE STUDIES AND SEMIOTICS 61

and traditions are passed on from one generation to the next orally. Some forms of
oral culture include poetry, folktales, and proverbs as well as magical spells, religious
incantations, and stories of the past. Orality involves a face-to-face form of contact.
Cyberspace has led to the emergence of a secondary form of orality, since social
media interactions occur as if they were oral and their content is similar, with memes
constituting a form of gossip or, often, a way of joking. As McLuhan (1964) argued, the
primary form of orality is the fundamental one and still exists in many societies or groups,
but in the age of electronic communications, a secondary form has emerged which has
blended, strangely, with writing, transforming writing into a simulative-oral medium – it
is elliptical and compressed (lacking many punctuation marks and correct grammar), in
ways that parallel vocal speech; it is synchronous (occurs in real time) like vocal speech,
while it is also editable, like writing; it is informal and highly conversational in style and
mode of delivery; it encourages immediate replies from one’s interlocutor, as in oral
communication.

CONCLUDING REMARKS
Media semiotics, in both its classic and current versions, is about how meaning structures
are embedded in different media and how they evolve over time through technological and
social changes. It seeks to flesh out meanings from media texts, venues and performances
through a series of specific notions, such as opposition theory, code, text, iconicity,
simulacra, a-signification, deconstruction and a few others. It has provided a lexicon that
allows scholars from various disciplines to extract from media semiotic analysis relevant
findings and theories for their own utilization. The semiotic study of media culture is
fundamentally an exercise in unravelling the psychological reasons why we use media
to extend ourselves, thus aligning itself with McLuhan’s framework of the effects of our
media on our evolution.
Media semiotics started with Barthes, but it could be claimed, in fact, that the true
founder was Marshall McLuhan, as intimated throughout this chapter. His views of
how media extend the human body into the world, allowing it to do more than would
be possible by biology, are actually a fundamental principle of semiotic science as a
whole. McLuhan never used the word semiotics in his writings. But there is little doubt
that many, perhaps all, of his ideas are fundamentally semiotic. A McLuhanesque media
semiotics would put into its core approach a study of the ways in which new media
change people and societies. It would constitute a form of semiotic investigation that
goes beyond explorations of ways in which our individual intelligence is affected by
new media. In addition to this, it would also explore ways in which the global village we
inhabit, based on the internet, has led to a connected form of intelligence, as Toronto
communications scholar Derrick De Kerckhove (1997) has argued, which means that
the sum total of the ideas of people will be vastly more important than those of any
one individual. He speculates that we are undergoing one of the greatest evolutionary
leaps in the history of our species. The architecture of connected intelligence, as
represented by the World Wide Web, resembles that of a vast brain whose cells and
synapses are encoded in software and hardware that facilitate the assemblage of minds.
Because of this, individual brains are able to see more, hear more and feel more. In this
environment, experts are just one part of the collective mind, whose ideas are carried
by software and hardware systems that overlap with them and with relevant data and
information.
62 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

As the media for knowledge-making and communicating change, we develop a different

sense of ourselves. Semiotics has always studied this sense in terms of the body-mind-
environment connection. It is thus well-adapted to study how this connection is unfolding
in cyberspace. As McLuhan put it (1970: 180), ‘When the evolutionary process shifts
from biology to software technology the body becomes the old hardware environment.
The human body is now a probe, a laboratory for experiments.’

REFERENCES
Adorno, T. W. (1941), The Culture Industry, London: Routledge.
­Albertazzi, D. and P. Cobley (2010), The Media: An Introduction, New York: Routledge.
Arnold, M. (1869), Culture and Anarchy, London: Smith, Elder, & Co.
Bakhtin, M. M. (1981), The Dialogic Imagination: Four Essays by M. M. Bakhtin, trans.
C. Emerson and M. Holquist, Austin: Texas University Press.
Barthes, R. (1957), Mythologies, Paris: Éditions du Seuil.
Barthes, R. ([1964] 2004), ‘Rhetoric of the Image’, in C. Handa (ed.), Visual Rhetoric in a
Visual World: A Critical Sourcebook, New York: Bedford/St. Martin’s.
Barthes, R. (1975), The Pleasure of the Text, New York: Hill and Wang.
Baudrillard, J. ([1981] 1994), Simulacra and Simulation, Ann Arbor: University of Michigan Press.
Baudrillard, J. (1983), Simulations, New York: Semiotext(e).
Bell, D. (1976), The Coming of Post-industrial Society, New York: Harper Colophon Books.
Bignell, J. (2002), Media Semiotics: An Introduction, 2nd edn, Manchester: Manchester
University Press.
Canizzaro, S. (2016), ‘Internet Memes as Internet Signs: A Semiotic View of Digital Culture’,
Sign Systems Studies, 44: 562–86.
Cantril, H. (1940), The Invasion from Mars: A Study in the Psychology of Panic, Edison, NJ:
Transaction Publishers.
Carr, N. (2011), The Shallows: What the Internet Is Doing To Our Brains, New York: Norton.
Carr, N. (2015), The Glass Cage: How Our Computers Are Changing Us, New York: Norton.
Castells, M. (2011), The Rise of the Network Society, Hoboken: Wiley-Blackwell.
Cheney-Lippold, J. (2017), We Are Data: Algorithms and the Making of our Digital Selves, New
York: New York University Press.
Danesi, M. (2015), ‘The Semiotics of the Mass Media’, in Peter Pericles Trifonas (ed.),
International Handbook of Semiotics, New York: Springer.
Danesi, M. (2016), The Semiotics of Emoji, London: Bloomsbury.
Danesi, M. (2018), Understanding Media Semiotics, 2nd edn, London: Bloomsbury.
Danesi, M. (2019), Memes and the Future of Popular Culture, Leiden: Brill.
Dawkins, R. (1976), The Selfish Gene, Oxford: Oxford University Press.
De Kerckhove, D. (1997), Connected Intelligence: The Arrival of the Web Society, Toronto:
Somerville.
Deleuze, G. and F. Guattari (1987), A Thousand Plateaus, Minneapolis: University of
Minnesota Press.
Derrida, J. (1976), Of Grammatology, trans. G. C. Spivak, Baltimore: Johns Hopkins Press.
Derrida, J. (1978), Writing and Difference, London: Routledge & Kegan Paul.
Donald, M. (2010), ‘The Exographic Revolution: Neuropsychological Sequelae’, in
L. Malafouris and C. Renfrew (eds), The Cognitive Life of Things: Recasting the
Boundaries of the Mind, 71–9, Cambridge: McDonald Institute Monographs.
MEDIA/CULTURE STUDIES AND SEMIOTICS 63

Eco, U. (1979), The Role of the Reader. Explorations in the Semiotics of Texts, Bloomington:
Indiana University Press.
Eco, U. (1992), Interpretation and Overinterpretation, Cambridge: Cambridge University Press.
Fauconnier, G. and M. Turner (2002), The Way We Think: Conceptual Blending and the Mind’s
Hidden Complexities, New York: Basic.
Foucault, M. (1972), The Archeology of Knowledge, trans. A. M. Sheridan Smith, New York:
Pantheon.
Genosko, G. (2008), ‘A-signifying Semiotics’, The Public Journal of Semiotics, 11 (1): 11–21.
Gerbner, G. (1956), ‘Toward a General Model of Communication’, AV Communication
Review, 4: 171–99.
­Gerbner, G. and L. Gross (1976), ‘Living with Television: The Violence Profile’, Journal of
Communication, 26 (2): 173–99.
Gramsci, A. (1937), Letters from Prison, New York: Columbia University Press.
Greenfield, S. (2015), Mind Change, New York: Random House.
Group µ (1970), A General Rhetoric, Paris: Larousse.
Group µ (1992), Traité du signe visuel: Pour une rhétorique de l’image, Paris: Seuil.
Guattri, F. (1995), ‘Machinic Junkies’, in Soft Subversions, New York: Semiotext(e).
Hadlington, L. (2017), Cybercognition, Los Angeles: Sage.
Hall, S. (1978), Policing the Crisis, New York: Palgrave.
Hawreliak, J. (2018), Multimodal Semiotics and Rhetoric in Videogames, New York: Routledge.
Hayles, K. (2017), Unthought: The Power of the Cognitive Nonconscious, Chicago: The
University of Chicago Press.
Hebdige, D. (1979), Subculture: The Meaning of Style, London: Methuen.
Heidegger, M. ([1953] 2010), Being and Time, trans. Joan Stambaugh, New York: State
University of New York Press.
Herman, E. and N. Chomsky (1988), Manufacturing Consent: The Political Economy of the
Mass Media, New York: Pantheon.
Hoggart, R. (1957), The Uses of Literacy, London: Chatto & Windus.
Horkheimer, M. (1947), Eclipse of Reason, New York: Oxford University Press.
Hosanagar, K. (2019), A Human’s Guide to Machine Intelligence: How Algorithms Are Shaping
Our Lives and How We Can Stay in Control, New York: Viking.
Jenkins, H. (2006), Convergence Culture: Where Old and New Media Collide, New York: New
York University Press.
Jensen, K. B. (1995), The Social Semiotics of Mass Communication, London: Sage.
Johnson-Laird, P. N. (1983), Mental Models, Cambridge: Harvard University Press.
Johnson, M. (1987), The Body in the Mind: The Bodily Basis of Meaning, Imagination and
Reason, Chicago: University of Chicago Press.
Katz, E. (1959), ‘Mass Communication Research and the Study of Culture’, Studies in Public
Communication, 2: 1–6.
Katz, E. and P. F. Lazarsfeld (1955), Personal Influence: The Part Played by People in the Flow
of Mass Communications, Glencoe: The Free Press.
Lakoff, G. (1987), Women, Fire, and Dangerous Things: What Categories Reveal about the
Mind, Chicago: University of Chicago Press.
Lakoff, G. and M. Johnson (1980), Metaphors We Live By, Chicago: Chicago University Press.
Lakoff, G. and M. Johnson (1999), Philosophy in the Flesh: The Embodied Mind and Its
Challenge to Western Thought, New York: Basic.
Langlois, G. (2014), Meaning in the Age of Social Media, New York: Palgrave.
Lasswell, H. D. (1927), Propaganda Techniques in World War I, Cambridge: MIT Press.
64 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Lazarsfeld, P. F., B. Berelson and H. Gaudet (1948), The People’s Choice, New York: Columbia
University Press.
Lazzarato, M. (2014), Signs and Machines: Capitalism and the Production of Subjectivity, South
Pasadena: Semiotext(e).
Latour, B. (2007), Reassembling the Social: An Introduction to Actor-Network-Theory, Oxford:
Oxford University Press.
Law, J. (2008), ‘Actor Network Theory and Material Semiotics’, in B. S. Turner (ed.), The New
Blackwell Companion to Social Theory, 141–58, Oxford: John Wiley & Sons.
Leavis, F. R. (1952), The Common Pursuit, London: Chatto & Windus.
Lévi-Strauss, C. (1987), Introduction to Marcel Mauss, London: Routledge.
­Levy, A. (2005), Female Chauvinist Pigs: Women and the Rise of Raunch Culture, New York:
Free Press.
Lippmann, W. (1922), Public Opinion, New York: Macmillan.
Lowenthal, L. (1949), Triumph of Mass Idols, New Brunswick: Transaction Books.
Manovich, L. (2002), The Language of New Media, Cambridge: MIT Press.
Marcuse, H. (1972), An Essay on Liberation, Harmondsworth: Penguin.
Mazzali-Lurati, S. (2007), ‘Here Is the Author! Hyperlinks as Constitutive Rules of
Hypertextual Communication’, Semiotica, 167 (2): 119–34.
McCombs, M. and D. Shaw (1972), ‘The Agenda-Setting Function of Mass Media’, Public
Opinion Quarterly, 36: 176–87.
McLuhan, M. (1951), The Mechanical Bride: Folklore of Industrial Man, New York: Vanguard.
McLuhan, M. (1962), The Gutenberg Galaxy, Toronto: University of Toronto Press.
McLuhan, M. (1964), Understanding Media: The Extensions of Man, New York: McGraw Hill.
McLuhan, M. (1970), Culture Is Our Business, New York: Wipf & Stock.
McLuhan, M. and E. McLuhan (1988), Laws of Media: The New Science, Toronto: University
of Toronto Press.
McLuhan, M. and Q. Fiore (1967), The Medium is the Massage: An Inventory of Effects, New
York: Random House.
McRobbie, A. (1991), Feminism and Youth Culture: From ‘Jackie’ to ‘Just Seventeen’, Boston:
Unwin Hyman.
Minsky, M. and S. Papert (1969), Perceptrons: An Introduction to Computational Geometry,
Cambridge, MA: MIT Press.
Negroponte, N. (1995), Being Digital, New York: Knopf.
Nesselroth, P. W. (2007), ‘Playing Doubles: Derrida’s Writing’, Semiotica, 166 (3): 427–32.
Noelle-Neumann, E. (1984), The Spiral of Silence: Public Opinion – Our Social Skin, Chicago:
University of Chicago Press.
Nöth, W. (1997), Semiotics of the Media: State of the Art, Projects, and Perspectives, Berlin:
Mouton de Gruyter.
Ong, W. (1982), Orality and Literacy, New York: Methuen.
Pasquinelli, M. (2017), ‘Machines that Morph Logic: Neural Networks and the Distorted
Automation of Intelligence as Statistical Inference’, Glass Bead: Logic Gate, The Politics of
the Artifactual Mind. Site 1, www.glass-bead.org/wp-content/uploads/GB_Site-1_Matteo-
Pasquinelli_Eng.pdf
Peirce, C. S. ([1866–1913] 1931–58), The Collected Papers of Charles Sanders Peirce, vols. 1–6,
C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University
Press. Cited as CP.
Peirce, C. S. (1868 [1992]), ‘Some Consequences of Four Incapacities’, in Nathan Houser
and Christian Kloesel (eds), The Essential Peirce 1 (1867–93), Bloomington: Indiana
University Press.
MEDIA/CULTURE STUDIES AND SEMIOTICS 65

Phoca, S. and R. Wright (1999), Introducing Postfeminism, Cambridge: Icon Books.

Pollio, H. R., J. M. Barlow, H. J. Fine and M. R. Pollio (1977), The Poetics of Growth:
Figurative Language in Psychology, Psychotherapy, and Education, Hillsdale, NJ: Lawrence
Erlbaum Associates.
Poulsen, V., G. Kvåle and T. van Leeuwen, eds (2018), Social Media as Semiotic Technology,
Special Issue of Social Semiotics, London: Routledge.
de Saussure, F. (1916), Cours de linguistique générale, ed. C. Bally and A. Sechehaye, Paris:
Payot.
­Scheufele, D. A. and P. Moy (2000), ‘Twenty-five Years of the Spiral of Silence: A Conceptual
Review and Empirical Outlook’, International Journal of Public Opinion Research, 12: 3–28.
Simonson, P. and J. Durham Peters (2008), ‘Communication and Media Studies, History to
1968’, in W. Donsbach (ed.), International Encyclopedia of Communication, 764–71,
Oxford: Blackwell.
Thompson, E. P. (1963), The Making of the Working Class, London: Vintage.
Turner, Graeme (2002), British Cultural Studies: An Introduction, 3rd edn, London: Routledge.
Unsworth, N., B. D. McMillan, G. A. Brewer and G. J. Spillers (2012), ‘Everyday Attention
Failures: An Individual Differences Investigation’, Journal of Experimental Psychology
Learning, Memory, and Cognition, 38 (6): 1765–72.
Van Zoonen, L. (1994), Feminist Media Studies, London: Sage.
Walsh-Matthews, S. and M. Danesi (2019), ‘AI: A Semiotic Perspective’, Chinese Semiotic
Studies, 15 (2): 199–216.
Warschauer, M. and D. Grimes (2007), ‘Audience, Authorship, and Artifact: The Emergent
Semiotics of Web 2.0.’, Annual Review of Applied Linguistics, 1: 1–23.
Wiener, N. (1948), Cybernetics, or Control and Communication in the Animal and the
Machine, Cambridge: MIT Press.
Williams, R. (1950), Reading and Criticism, London: Frederick Muller.
Williams, R. (1977), Marxism and Literature, London: New Left Books.
Wong, M. (2019), Multimodal Communication: A Social Semiotic Approach to Text and Image
in Print and Digital Media, New York: Palgrave.
Zappavigna, M. (2019), Searchable Talk: Hashtags and Social Media Metadiscourse, London:
Bloomsbury.
66
 ­C HAPTER THREE

Digital Humanities and

Semiotics
ALIN OLTEANU AND ARIANNA CIULA

EXISTING APPROACHES: A MODELLING FOCUS

The prominent role of modelling activities in digital humanities (McCarty 2005), hereafter
DH, was recently addressed from a semiotic perspective (Ciula and Eide 2017; Ciula
and Marras 2019), as a way to foster a modelling theory fitted for this new and quickly
developing area of research. This approach is not unexpected, as semiotic theories can
justify in many regards the use of digital methods in humanities research, which they can
help develop. These two areas of scholarship share modelling, understood as schematic
(diagrammatic), as a key interest. By schematic we refer to structures of meaning that are
more accurately represented through diagrams than in verbal language.
Given the swift uptake and advancement of digital technology in recent decades,
converting (non-digital) objects into digital forms, processable by computers, a process
termed digitization, and the analysis of those digitized objects via computational methods
have acquired momentum in art and humanities research as well as in the cultural heritage
sector. This has been occurring in parallel with the restructuring and expansion of human
societies onto digital media, in general, a process termed digitalization (Brennen and Kreiss
2016). Digitalization produces a social networking effect: it merges human communication
channels into vast networks, on a global scale, where nodes can be regarded as signal
emitters. Thus, in social sciences and humanities research on digitalization, a schematic
(network-like) notion of model can be observed. In this direction, we remark, among
others, boyd’s (2014) approach to (cultural) identity vis-à-vis social networks and Hartley’s
(2015) notion of urban semiosis. boyd explains that the structures of communication of
social media networks have consequences for the mental modelling of audiences, which
further has implications for the identities of individuals and, hence, social organization
(see also Marwick and boyd 2010). The engineering of digital communication networks is
not distinct or inconsequential for cultural modelling: ‘increasingly, software engineers are
designing and building algorithms to observe people’s practices and interests in order to
model who they are within a broader system. [. . .] What becomes visible – either through
people or through algorithms – can affect how people understand social media and the world
around them’ (boyd 2014: 160–1). Relying on Lotman’s semiotic theory of modelling,
Hartley (2015: 80) explains that cities, as sites of creativity where systems ‘clash’, are not
so much geographical locations as they are networks, irrespective of geographic proximity.
Also, inspired by Lotman, as well as other major semiotic scholars, such as Umberto
Eco, Bankov’s recent book (2022) addresses digital culture from a semiotic perspective.
68 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

While Bankov does not focus on modeling and digital humanities methods, the book
demonstrates the intricacies of semiotic theories and digital transformations.
Construing models as networks is commonplace in semiotics. The diagram-centred
semiotics of Peirce (see Pietarinen 2006; Stjernfelt 2007) has been given renewed interest
recently because of its compatibility with the revolution in linguistics and philosophy
of mind produced by the emergence of the notion of schema in cognitive linguistics
(Johnson 1987; Lakoff 1987). In general, this schematic trend in linguistics and semiotics
can be seen as aligned to, if not a major part of the iconic turn in the humanities (Moxey
2008; Boehm and Mitchell 2009). Here, we explain why it is relevant for DH that the
primary consequence of the iconic or schematic turn has been a surge in the popularity of
certain schools of semiotics. These are, to begin with,
1. Peirce’s semiotics, given the centrality of iconicity in this theory
2. Cognitive semiotics, where the cognitive linguistic notion of schema has been
adopted with a semiotic scope
3. The multimodality framework, which considers meaning (expressed in text) as
irreducible to a double articulation
4. Biosemiotics, given its focus on modelling and embodiment, and
5. Ecosemiotics, the sub-branch of biosemiotics that focuses on networks
This chapter explains the synergy between semiotics and DH, particularly in relation
to modelling, and suggests potential directions for further theoretical development, which
could contribute to overcoming what has been called the crisis of the humanities (see
Nussbaum 2010; Jay 2014), in itself of particular interest to semiotics (Martinelli 2016;
Cobley 2017) as well as DH.1 The remediation processes connected to digitization of the
objects of study of the humanities and their modelling are a critical matter in overcoming
this impasse. Digitization – when dealing with conversion across media – and digitilization
are forms of intermedial translation, hence of relevance to translation studies that found
its theoretical grounding in semiotics (e.g. Elleström 2014; Pérez-González 2014; Kourdis
2015; O’Halloran et al. 2016).
Observing that semiotic theories fit this purpose, O’Halloran (2015) established
multimodal digital semiotics as a lens for multimodal digital humanities. As this approach
is focused on the comprehensive analysis of multimodal texts, primarily relying on the
traditional theoretical tools of systemic functional linguistics (Halliday 1978) and social
semiotics (Kress and van Leeuwen [1996] 2006, 2010; O’Toole 2011), it does not present
modelling as a theory-framing notion. It does, however, rely on modelling as an analytical
method (O’Halloran 2015: 390), as common for DH, albeit within the limited context
of cultural analytics.
The adoption of semiotic modelling in the DH scholarship (Ciula and Eide 2017)
followed one of the extrapolations of Charles Peirce’s semiotics as modelling theory,
particularly as developed in a philosophy of science view (Kralemann and Lattmann 2011).
Models and modelling are a common, central concern in both DH, in its research products
and process, and in semiotics, as a theoretical cornerstone. Computers are modelling
machines (see McCarty 2005: 22; Mahoney 2000, 2005) and, more often than not,
semiotic theories are modelling theories. The work of signs, according to many accounts,
is modelling (Lotman 1977, 1991; Myrdene and Merrell 1991; Sebeok 1991; Sebeok
and Danesi 2000; see also Nöth 2018). To make sense of things, or, putting it differently,
to discover meaning in experiencing the factual world is to make useful models of those
things or facts. A commonly accepted hypothesis of semiotics is that meaning is a relation
DIGITAL HUMANITIES AND SEMIOTICS 69

(and models are relational), and thus to discover meaning is to establish (or observe) a
relation. As such, the models by which semiotics describes the meaningful environments
of living organisms have the structure of networks – interconnected sums of relations. For
this reason, semiotic modelling theory particularly fits the work of and with computers.
This semiotic perspective is of immediate value for DH (Ciula and Marras 2016, 2019).
Connections between DH in its earliest manifestations (humanities computing) and
semiotic theories have been more or less explicit. Pending a systematic review of these
intersections, they can tentatively be clustered in three non-exclusive categories:
1. More or less explicit overarching references to historical analysis and philosophical
underpinnings around the meaning of information, data, systems and infrastructures
converging cyclically towards reflections and critiques around the agency of the
machine and platforms in processing symbols in relation to human intelligence,
interpretation and bias (in this respect Smithies (2017) offers an insightful analysis
of DH seen from the socio-technical lens of the digital modern)
2. Practical approaches inspired by semiotic concepts to inform the creation of
computable models and standards as well as interactive interfaces
3. Attempts to connect the two levels (theory and practice) in particular via a semiotic
theorization of modelling
Approaches that we subsume under 2 vary extensively. For example, they include
mainly text-focused approaches grounded in philosophy of language and attempting to
provide formal models of the linguistic sign in areas akin to Natural Language Processing
methods and computational linguistics. This category can also account for user design-
oriented methods partially anchored to media studies (see, e.g. Drucker 2013), as well as to
the field of human-computer interaction. Further, under the second category we can also
group more inclusive analysis of cultural forms and objects leading for example towards
the multimodal turn discussed further below. For example, the project of a semiotic
branch of DH was addressed explicitly in multimodality studies, particularly within a
project that Kay L. O’Halloran started at the University of Singapore (O’Halloran 2015:
389–90; see also O’Halloran et al. 2013, 2014).
The third category is an area with potential that calls for further research. Indeed,
while semiotics concepts have informed and continue to inspire DH research, only rarely
a semiotic approach has been explicitly adopted to underpin data models (see for example
Boot 2006). Particularly, the semiotic proposal in DH found inspiration in Peirce’s
pragmatic and context-aware notion of meaning-making (semiosis) and in his notion
of iconicity, as exploited in recent scholarship on modelling (Kralemann and Lattmann
2013) and medial iconicity (Elleström 2013).
It is noteworthy that the proposal for a semiotic approach to DH modelling came
from DH scholarship. So far, there seem to be only suggested implications coming from
semiotic scholarship, given its ramifications into areas of research such as information
theory, design and, in a very general sense, textual studies. Ciula and Eide’s (2017) initial
proposal reveals numerous applications and can find support in semiotic theories broadly,
beyond the sources explicitly explored so far. Semiotics can offer a robust philosophical
and theoretical underpinning of DH for various reasons that have not been explicated yet
in semiotic scholarship but are implicit in semiotic research on modelling, multimodality,
translation, iconicity, embodiment, literacy and industry. Recent semiotic scholarship
richly explores these topics of relevance for the DH, but there seems to be no remarkable
attempt from this end at developing a semiotic theoretical underpinning for the DH.
70 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Also, while non-discursive languages are part of the epistemological foundations of DH

(e.g. for visual languages see Drucker 2011), its most recognized landmarks are founded
on (McCarty 2005; Berry 2012), or tend to be pulled towards, discursive theory,2 arguably
a language-centred semiotic theory (Cobley 2016: 18, 95). Here we draw attention
to a mismatch: as digit(al)ization boosts the modal plurality of social representations,
thus bringing to the fore the (cross-)iconicity of semiotic systems, its language-centred
construal is prone to miss many of its important characteristics. Model-focused semiotics,
as mostly explored in light of the Peircean school, tends to distance from language-,
discourse- and text-based conceptions in favour of more comprehensive schematism
(e.g. Pietarinen 2006; Stjernfelt 2007, 2014; Cobley 2016). A semiotic exploration of
possible contributions to DH should account for non-language-centred modelling tools,
that is, develop a theory that does not conceive of meaning as necessarily dependent
upon a linguistic representation. Language-centrism, as Kress and van Leeuwen point
out (2001: 4, see also Olteanu 2019: 124, 2020), relies on a rigid notion of meaning as
fixed form-content articulation. The limitations of this traditional linguistic axiom have
been observed in DH, for instance in discussions on layered models for digitizing and
interpreting handwritten objects, in digital palaeography (Ciula 2017).
Arguably, recent trends in semiotics suggest that semiotic theory is particularly fit
to apprehend the way in which digit(al)ization models human cultures and societies of
the past, present and futures, in ‘data worlds’ (Gray 2018), structures of networks and
infrastructures.3 Studying and informing the construction of these data worlds, networks
and infrastructures falls within the scope of DH and digital social sciences more widely.
As such, accounting for the entanglement of the technical and ethical dimensions of the
modelling process in digital media and its development in light of semiotic theory could
offer an opportunity to address the ongoing crisis of the humanities. The decline of
humanities studies in universities and the relatively low financing that research funders
direct to the humanities are due, among other things, to the uninvolvement of the
humanities in engineering digital societies (see Smithies, ffrench and Ciula, forthcoming).
A proper inter- (or trans-) disciplinary dialogue between research on meaning (semiotics)
and infrastructure studies could open up opportunities in this respect.

MULTIMODALITY
In the past two decades, discussions in social semiotics have been dominated by the
concept of multimodality (Kress and van Leeuwen 2001), which has become a framework
for communication studies altogether (Kress 2010). This term refers to the articulation of
meaning through the use of the plurality of sense perception channels and representation
modalities available. It is a development of and a critique of the traditional humanistic,
literary notion of text (see, e.g., Lotman 1991), particularly as developed in systemic
functional linguistics (see Halliday 1978; Halliday and Webster 2009). The need for
such an approach to meaning and communication was inspired by the diversification
of media technologies in recent decades. Particularly, this awareness was accelerated by
digit(al)ization, as digital media afford and make use of designs which, to be produced
and interpreted, require the bringing together of various modalities. As traditional
theories that regard meaning as a double articulation of form and content miss essential
aspects of multimodally constructed discourses, social semiotics drew its attention to the
multimodal articulation of meaning. While the multimodality framework has become
DIGITAL HUMANITIES AND SEMIOTICS 71

highly popular, arguably becoming the default perspective of social semiotics, currently
used in the study of many social phenomena, it has been seldom invoked in a full-fledged
critique of digitalization (suggestions in this direction might be found, arguably, in some
works such as Kress Selander 2012) or in regard to its potential relevance for DH (as
gestured by Clivaz 2017).

TRANSLATION
Translation appears an insightful area of interest at the intersection of semiotics and
DH because it is a form of modelling. Any model-driven engineering activity requires an
operation of translation across multiple dimensions (Génova et al. 2009). As a poietic
creation, conceptualizing and writing software in code is a matter of translation, from
human models of reality and systems into computable or computational models. As
noted above, digiti(li)zation is an intermedial and, also, intersemiotic translation which,
following Jakobson (1959), stands for a translation from a linguistic to a non-verbal
system. Digit(al)ization brings to the fore a panoply of examples of translation processes
that do not involve language at all. As O’Halloran et al. (2016: 199) observe,
For Jakobson, intersemiotic translation involved language. He does not discuss
translation from one non-verbal semiotic system to another non-verbal semiotic system
or the translation of multisemiotic texts.
Because of its language-centrism tainting, Jakobson’s classic notion of intersemiotic
translation is not a suitable theoretical lens for understanding digitization and modelling
in DH. Moreover, modelling in DH reveals not only that some translation processes,
such as in human-machine interaction, do not necessarily involve language but that
monolingualism is a flawed doctrine altogether because translation and modelling
themselves are always inter- or even poly-semiotic. As such, digitalization produced
critical changes in translation theory and practice (see Olteanu 2020). A revealing example
that theoretical work on digitization offers, which we discuss below, is the translation of
existential graphs, as naturalistic representations of logical operations, into conceptual
graphs, embodied in software design and development (Sowa 2000, 2014).
Jakobson’s semiotic theory, given his view on language, could nevertheless provide
another solid pillar for the semiotic take on DH. He was an avant-garde promoter of
Peirce’s semiotics, observing that, as Peirce’s semiotics implies, iconicity is essential to
language, much more so than pure convention, which allows for the articulation of
opposites (Jakobson [1965] 1987). This was at least controversial, in the structuralist
ambiance of the (semiotics of the) 1960s. His advocacy for iconicity in language implies
what in contemporary terminology can be defined as the multimodal quality of language.
Thus, we note the relevance of pursuing Jakobson’s iconicity-based semiotics, with critical
revisions (e.g. with respect to translation) in the scope of DH.
For the same reasons that led to the switch from double articulation to multimodality
in social semiotics, in translation studies a semiotic turn can be noticed of recently. In
a society that increasingly relies on intermedial and multimodal designs, traditional
linguistic theories fall short in grasping the meaning effects of translating multimodal
texts, particularly when such translations require cross-modal operations. Texts in
contemporary advertising, marketing, education and art, to name just a few social practices,
explicitly and purposefully require an awareness of their inherent multimodality in order
72 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

to be created and interpreted. Thus, the relevance of non-verbality and of schematicism in

meaning-making has drawn the attention of translation studies (Bezemer and Kress 2008;
Torresi [1998] 2009; Pérez-González 2014). For this reason, contemporary construals
of translation tend to coincide with medial adaptation (Elleström 2014, 2017), or what
in classic Greek terminology was expressed as ekphrasis. This indicates the necessity of
an overarching theory, simultaneously comprehensive of both semiotics and mediality.
The media semiotic theories of Danesi (2002) and Elleström (2014, 2018) present
such perspectives. However, despite its critical innovations, Danesi’s theory holds an
anchor in language-centred theories of (post)structural inheritance. A semiotic approach
to DH should seek to clear away language-centred concepts of meaning. For example,
the study of translation and multilingual corpora, which are traditionally approached
from language-centred perspective, has provided opportunities in DH research for non-
language-centred analysis via data curation, processing and visualizations.4 Elleström’s
theory (2013, 2014), as already illustrated by its adoption by Ciula and Eide (2016),
constitutes a good starting point for a semiotic theory of digit(al)ization and of digital
modelling. As its purpose is the thorough analysis of meaning effects of media, it can be
(and has been at least in Eide 2015) employed as one core aspect of DH modelling, from
a semiotic perspective.
Following Peirce, translation ‘is semiotic mediation’ (Nöth 2012: 279), which leads
to construing translation as a type of intermedial modelling. This opens up potentially
new integrated approaches informed by semiotics to interactions between humans
and computers and among humans within and across digital media. From a pragmatic
perspective, models in DH are artefacts produced during modelling activities in iconic
relation with other objects or phenomena they aim to represent, based on specific interests
and biases, contexts and purpose. They are experienced, manipulated, reasoned upon
artefacts of different nature. They are not only computational models instantiated into
formal code but also sketches and diagrams, for example, often documented in stages of
design and analysis as part of model-driven engineering processes of non-linear translations
(for a recent contribution to the discussion on translation across models in DH from a
research software engineering perspective, see Ciula et al. 2020 and forthcoming). As any
translation is an interpretation and vice versa (e.g. Pajević 2014) and models are non-
language centred interpretations with specific interests, the theoretical construal of digital
modelling can rely on (some accounts in) the vast semiotic scholarship on translation (e.g.
Gorlée 1994; Stecconi 1994, 2004; Nöth 2012; Kourdis 2015). In turn, by addressing
modelling as a translation process, translation studies would develop towards a more
finely tuned understanding of the specific meaning effects of computational methods,
digital media and platforms.

ICONICITY
Elleström’s medium-centred theory of meaning and communication relies on the concept
of iconicity, particularly as developed in light of Peirce’s semiotics. Peirce termed icon
the sign type that signifies due to similarity. Iconicity refers to how, or on what basis
does one phenomenon signify another by means of similarity (e.g. Stjernfelt 2007: 49).
In a Peircean view, iconicity is present in any instance of meaning-making and, also,
can constitute meaning on its own, without reliance on other types of representation.
Given this aspect of Peirce’s semiotics, his work is one of the sources that inspired
DIGITAL HUMANITIES AND SEMIOTICS 73

the recent iconic turn (Moxey 2008; Boehm and Mitchell 2009), which started in
the visual arts. While at first a matter of (visual) art research, the iconic turn has been
gaining ground in the humanities and social sciences. It consists in the switch from
verbal language to schematic structures as the basic and general vehicle of knowledge,
understood as both a further development of the linguistic turn and a rejection of
some of the latter’s central claims. In brief, the iconic turn regards operations with
image schemata as basic for modelling. In philosophy of science, the widespread use
of models and the iconic turn, away from analytical methods in scientific research,
have been attributed to the prominence of computational models and the widespread
adoption of visualization tools (Godfrey-Smith 2009: 108). As Boehm and Mitchell
remark in their landmark conversation (2009), digitalization played an important role
in stimulating the iconic turn. After this realization in visual art research, more fields
of the humanities have followed suit, such as cognitive linguistics, phenomenology,
theory of embodiment and semiotics. Also, the iconic turn is intimately related to the
multimodality turn, as image schemata afford various simultaneous modalities, unlike
the modelling at which linear, verbal language, particularly in the common perception
of print mediality,5 aims.

EXISTENTIAL GRAPHS
A central thesis in Peirce’s semiotics is that icons, that is, signifying relations of similarity,
are the only sign types that can be used as predicates, thus having a critical role in logic.
This position has been instrumental and of much interest particularly in the iconic turn
in semiotics (Pietarinen 2006; Stjernfelt 2007), which is focused on Peirce’s notions
of hypoicons (image, diagram, metaphor), as semiotic counterparts to the cognitive
linguistic notion of schema. Peirce introduced and explained the notion of iconicity to
justify the rationale of his system of graphic representations of logic (Stjernfelt 2015),
termed existential graphs. Existential graphs have already been adopted in information
systems development, as translated by Sowa (see 2000, 2014) into linearly represented
conceptual graphs. Conceptual graphs have been included as a formal knowledge
representation system in the ISO standard for common logic (see Sowa 2008). While
Peirce found an advantage in representing logical operations graphically, his system of
existential graphs proves useful to represent knowledge for computational processing
because, while its translation into a linear syntax and semantics hides the schematic into
a symbolic representation, it retains the iconic ground of logic.
The purpose of existential graphs is ‘to afford a method (1) as simple as possible (that
is to say, with as small a number of arbitrary conventions as possible), for representing
propositions (2) as iconically, or diagrammatically and (3) as analytically as possible’ (Peirce
1905: CP 4.561). This is also the rationale within which Elleström develops his media
communication theory, by considering that ‘[a] model should be understood as a clearly
outlined cognitive scheme that is both described with the aid of language, and depicted
as a diagram’ (2018: 270). Thus, as stated previously, modelling must be approached as
a cross-modal process of translation. Following this logic, a conceptual graph is not a
different model than its existential graph equivalent, but simply an analytic translation
of its diagrammatic counterpart. The work of modelling consists in conceptualizing the
graph and perceiving the correlation between its schematic and analytic representations.
It deals with a process of cross-modal translation.
74 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Even if multiple approaches are possible, DH modelling has to take into consideration
the underpinning justification of existential graphs, namely the iconicity of logic, and
practised in the emerging adoption of graph databases and conceptual models in particular
in relation to the Linked Open Data paradigm.6

EMBODIMENT
The iconic and multimodal turns are related to yet another turn, namely an embodiment
turn. Or, rather, these could be considered three aspects of one and the same turn
currently occurring in the humanities. The idea that the vehicle of knowledge consists
in schemata, some of which may be non-verbalizable, is connected with the idea that
the mind is embodied (e.g. Lakoff and Johnson 1999; Merleau-Ponty 1995; Stjernfelt
2007, 2014; Mittelberg 2013, 2019). The multimodality framework brought
awareness to the fact that humans have always made sense of their environment
multimodally. Some media technologies, among which especially the conventions of
print (with which we are so familiar in our daily encounters to the point of printing
cultures codes being overlooked by most but bibliographic experts), have inculcated
an illusion of modelling, and particularly of scholarly modelling disseminated in linear
publications, as monomodal. The discovery that social semiotics arrived at, thus, is
not multimodality itself but that monomodality is impossible, yet taken for granted
in modern philosophy and science.7 Monomodality is an abstraction. Conceiving a
monomodal model is not phenomenally possible for human beings or any animals
that have semiotic competencies comprising a plurality of representation modalities.
This is of some importance for thinking through digitalization and modelling in the
digital humanities for at least two reasons. First, as observed by Danesi (2002), unlike
previous technological media, digital media structure social awareness in ways that
reintegrate body and mind, undoing the dichotomizing effect of print, which shaped
modern dualism (analogically, for instance, consider the emergence of nationalism in
modernity, as a social consequence of print-capitalism, according to Anderson [1983]
2006). Second, as observed by Elleström (2018), mediality must be understood as
initially evoked by corporeality. Mediality consists in a relation between the body
and the environment, which implies an affordance (Gibson 1979) perspective for
media and representation (see Majchrzak et al. 2013; Treem and Leonardi 2013).
As we discuss below, the implications for ecology of digitalization should also be
considered in light of modelling. Hence, as digital modelling unfolds – for example,
in the translations across verbal and non-verbal languages, natural and programming
languages, human and machine interfaces – it has to be construed in light of the
human body’s basic semiotic competencies, as discussed in biosemiotics (Stjernfelt
2006; Campbell et al. 2020).

LITERACY
A main concern of the humanities tradition, literacy is also a central concern for research
on digitalization. Moreover, digitalization produced a turn in (new) literacy studies
(Mills 2010), as well. A critical aspect of this turn, yet again, consists in a reorientation
towards semiotic theories (Mills 2016: 65–6). The new literacies and the lifestyles
that they (will) support (see boyd 2014: 176–97) are essential to modelling society
DIGITAL HUMANITIES AND SEMIOTICS 75

and culture. Since each medium presents different affordances to a given species or
individual for representation – and, hence, for modelling – digital media have their own
specific effects on humans’ modelling of their world. For instance, Anderson (1983)
claimed that the spreading of literacy that was caused by print led to specific models of
organizing society. Without wanting to embrace over-deterministic views nor to elect
one culture and context as representative of global phenomena, the spreading of any
medium-specific literacy skills is deemed to affect social restructuring (McLuhan 1995:
177–8; Olteanu 2020). As such, since digital models depend on models of the (so-
called) empirical and social world of humans, there is no reason not to believe that the
future development of DH modelling will be shaped by the prospects brought about by
what is now termed digital literacy (Lankshear and Knobel 2008). Hartley (2011: 20–1)
explains this notion as an extension ‘beyond the defensive notion of “critical reading”
and “media literacy” as taught in schools, towards [. . .] a form of hands-on productive
expression, taught by and within a milieu in which it is deployed, using multiplatform
devices to “write” as well as “read” electronic media’. Because the theoretical notion
of digital literacy has had to keep up with the fast development of digital technology
lately, it is unlikely a long-term comprehensive definition will consolidate. In addition,
the plural digital literacies (Lankshear and Knobel 2008: 2) better reflects the plurality
of forms of literacy emerging in a dynamic digital media context.
Literacy is essentially understood as consisting in those competences required
for surviving and thriving in a certain human society.8 Human competences for
modelling are intimately related to literacy. For reasons similar to the iconic turn
and the semiotic turn in translation studies, the study of the specific literacies of
digital media has found inspiration in semiotic theories (e.g. Scolari et al. 2018).
Also, the multimodality framework found literacy as a particularly rich area of
application (e.g. Jewitt 2006; Bezemer and Kress 2008; Kress and Selander 2012).
In light of the same limitation that literary theory encounters in tackling multimodal
representations, Stables (2012: 76–7) proposes replacing the term literacy, which
suggests competences for operating with fixed symbolic signs, namely letters, with
a more comprehensive semiosy. This more holistic concept is also better suited to
the DH context, where objects of study are surely verbal texts but also span visual
objects, sculptures, maps, cultural artefacts of different kind, and derivative and
intermediate products of modelling from diagrams, formulas and snippet of code
to three-dimensional models. Natasa Lacković’s (2018, 2020) recent, impactful and
distinctively Peircean perspective on critical thinking and multimodal literacy can
provide insights on the mutuality between digital literacies and DH modelling; for,
even the more software-intensive or mathematically informed branches of DH, it has
been argued, critical and historical understanding of cultural objects cannot rely on
statistical analysis alone, but need ‘critically analysing and historicizing of data set’
(Bode 2020).
Typical of modernity, the traditional concept of literacy centres on competences
of coding and decoding texts. Arguably, this construal of literacy is still applicable,
but digitalization critically changed the concept of text. Thus, understanding the new
literacies of the digital age depends on construals of digital textuality.
Since digital modelling, in a very wide sense, is the result of the interaction between
humans and computers, digital literacies are instrumental as to how digital modelling
techniques and methodologies will develop. In light of the many social and cultural
changes it entails, digitalization supposes different educational programmes and curricula
76 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

than developed, so far, in view of modern ideas of science and social organization. A main
difference comes from the reality that digital technologies afford so many possibilities
for industry, many of which unpredictable, that digital literacies cannot be exhausted
by fixed lists (Lankshear and Knobel 2008). While this is particularly so in digitalized
societies, it is a valuable insight that digitalization provides for educational philosophy
and theory. Digitalization could well inspire an anti-deterministic philosophy, in contrast
to mechanistic mindsets which see modernity unidirectionally shaped by technology.
Nevertheless, the modelling work of digitalization can offer some intuitions as to what
could be useful skills, as deliverable through education, be it formal or informal (see Scolari
et al. 2018) and how professional carriers of the future could be affected by alternative
paths suited to a maker-culture, beyond or to complement traditional academic research
for the humanities.9
What is at stake in conceiving digital literacies and their apprehension by educational
systems is, of course, the (future) competences of current students on job markets, as
well as their social integration and mobility. Without ignoring the unpredictability of
future digital technologies and markets, there must also be a large number of learnable
and transferable predetermined skills pertaining to digital modelling, from high-quality
coding in various programming languages to data analysis and visualization techniques.
However, arguably, it is the process-oriented character of modelling that represents the
most interesting pedagogical challenge. Processes of making and negotiating meaning
collaboratively and translating across verbal and non-verbal languages are educationally
more valuable than supposed educational outputs or goals, as recent semiotic approaches
in philosophy of education also argue (e.g. Pesce 2010, 2020; Stables 2012, 2016;
Stables et al. 2018). Being literate, or semiate (following Stables 2012: 80), in the
digital age can equate to any version of combinations of practical skills and knowledge
that allow an individual (or community) to thrive, which means to be able to learn
more. These skills and knowledge belong to a broad spectrum going from the basics of
computational skills and modelling competences to advanced methods as required for
a research software engineering career (see recent recommendations on training and
education at the intersection between DH and data science suggested by McGillivray
et al. 2020). In any instance, it is important not to lose out of sight that knowledge of
how and when to use non-digital media is an essential part of digital literacies (Bawden
2008: 28).
Thus, digital literacies require further investigation in light of the modelling processes,
both by which they were established and by which they will likely foster further. Most
important, semiotic investigation should aim at revealing the underpinning meaning
patterns in digital modelling, which could be used to conceive general principles of digital
literacies.
Arguably, modelling, as a logical operation, should be taught as a curricular subject.
Existential graphs could be one school subject, for instance. Existential graphs or other
illustration of schematic logic, logical principles of DH modelling could be introduced
in pre-university curricula. This argument posits that such a curricular use of semiotics
would serve as a scaffolding for students’ acquisition of specific digital skills, such as
translating across styles of reasoning and expressions including coding (see Scolari et al.
2018), and it is, therefore, pedagogically more valuable, from the perspective of liberal
education, than the teaching of predetermined skills. This would mean the cultivation in
schooling of a semiotic (networked, relational) manner of thinking.
DIGITAL HUMANITIES AND SEMIOTICS 77

INDUSTRY
According to Floridi (2014), a current philosophy of information ought to account for
the way Information and Communication Technologies have transformed ourselves and
our environment (the infosphere) where, rephrasing Hegel, ‘what is real is informational
and what is informational is real’ (Floridi 2014: 41):

The most obvious way in which ICTs are transforming the world into an infosphere
concerns the transition from analogue to digital and then the ever-increasing
growth of the informational spaces within which we spend more and more of our
time. [. . .] This radical transformation is also due to the fundamental convergence
between digital tools and digital resources. The intrinsic nature of the tools (software,
algorithms, databases, communication channels and protocols, etc.) is now the same
as, and therefore fully compatible with, the intrinsic nature of their resources, the raw
data being manipulated. [. . .]. In the infosphere, populated by entities and agents all
equally informational, where there is no physical difference between processors and
processed, interactions become equally informational.
(Floridi 2014: 41)

In the infosphere, industry is of course digital, which means digital modelled and
engaged in digitally modelling. Hartley (2015) considers the Lotman’s celebrated
concept of semiosphere is explicative for the cultural dynamics of creative industries.
Similarly, O’Halloran (2013: 412) observes the need for a concept of ‘digital
semiosphere for multimodal digital humanities research’ that ‘involves the development
of computational tools and visualization techniques for analysing semiotic data’.
As mentioned above, many areas of industry (cultural, advertising, marketing) are
working with designs that require multimodal semiotic operations. As suggested in
the previous section, for the newly emerging creative industries (e.g. Hartley 2005),
learning a fixed list of skills and following of a predetermined school curriculum
is deemed to be inadequate for a market geared towards digital creativity which
favours, for example, collaborative thinking and making via rapid prototyping and
participatory design. Traditional notions of literacy and text are unsuited and foreign
to the digital job market. In addition to the acquisition of skills, students must cultivate
creativity and imagination as semiotic resources that allow a high adaptability in a
society where change is the status quo. Moreover, digitalization offers an opportunity,
which humanity does not afford to miss, for an equitable and sustainable development.
Arguably, the ongoing third industrial revolution (Rifkin 2011) consists in the merging
of digital media networks with renewable energy grids. The resulting global network,
which should distribute resources as necessary, exhibits a diagrammatic structure, akin
to semiotic modelling theories. Recent scholarship alludes to the inherently semiotic
structure of digitally modelled societies, which made the creative industries possible
(Hartley 2015). Thus, a main concern of the semiotic approach to DH, not immune
from the ethics in the infosphere, consists in modelling the social networks resulting
from the digitalization of industry. Indeed, the integration of digital technology in the
labour market come with benefits in the form of new behavioural affordances (Treem
and Leonardi 2013: 178) as well as risks (which the DH community is increasingly
aware of),10 if roles are not adequately professionalised.
78 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­A BIOSEMIOTIC APPROACH: FULL-BLOWN

MODELLING THEORY
Unlike previous technological media, the modelling work channeled via digital media
is not uni-directionally linear or nucleic but spreads the flow of meaningful content in
network structures. Semantic network analysis is one of several mainstream modelling
techniques in DH. It is akin to the main motivation of the biosemiotic approach to
culture and ecology, given the focus on schematic representation here (Kull 2003). The
turn from ladder-like or tree-like to web-like models that ecology produced in the natural
sciences is similar to the effect of (Peirce’s) semiotics in the humanities: it highlights the
network structure of systems. Further, this reveals the insightfulness of a modelling
approach, in contrast to the textual one, to culture and society. The semiotic modelling
theory per excellence, which has not been considered yet in regard to the digital, is
biosemiotics. From its conception, Thomas Sebeok (e.g. 1991) defined biosemiotics
as a modelling theory. Sebeok was inspired by Lotman’s cultural semiotic theory, but
also criticized it. He adopted Lotman’s idea that the work of signs is modelling, that
language is a modelling system and that human culture and society are pregnant with
supralinguistic, secondary modelling systems, such as art, science, technology and so on,
that culture itself is such a top-layer modelling system. In accordance with contemporary
evolution theories (e.g. Gould and Vrba 1982), Sebeok argued that if meaning is to
be conceived in terms of modelling systems, two levelled-modelling does not suffice.
Since most of communication in the biological world is non-verbal, then the primary
modelling system, it has to be assumed, is non-verbal. Language, then, is a secondary
modelling system, dependent upon the primary system, and supralinguistic modelling is
yet a third system. This has major consequences for conceptualizing meaning, language,
culture and modelling itself.
From Sebeok’s perspective, the primary, non-verbal modelling system is schematic,
roughly corresponding to Peirce’s iconic representations (see also Sebeok and Danesi
2000). It first and foremost shapes an organism’s intimate, inner world. This undermines
the rigid and traditional view in the humanities that modelling is textual (challenged by
Bod 2018), an idea that otherwise runs through Lotman’s theory (1991). Notions such as
culture text are deemed unnecessary, thus.
Biosemiotics implies, as more recently explained by Cobley (2010, 2016), that cultural
phenomena should be understood from the perspective of modelling. In the scope of the present
paper, this implies that DH modelling processes must rely on the embodied, schematic human
competences for modelling, as well as on language and, possibly, other medial modelling
systems including those embodied by information agents and platforms (indeed both human
and machine materiality affect our reasoning apparatus and its potential – e.g. see Dourish
2014). While Sebeok’s version of biosemiotics is foundational and still innovative, more
recent developments on the modelling work of organisms must be considered. Particularly,
holding on to a Peirce-faithful notion of iconicity, as the justification of existential graphs,
reveals more finely grained modelling activities. From this perspective, perceptual judgements
in the form of propositions have a pivotal role in modelling (Stjernfelt 2014). Perceptual
judgements are the models, not necessarily cognitive, that an organism primarily experiences,
as a reaction to its environmental situated presence. Their propositional structure affords, in
view of the organism’s semiotic competences, both their decomposition into more basic sign
types (predicates, icons, indexes) and the further using of these discovered signs as models or
tools for modelling and their copulation with other signs, towards the construction of more
DIGITAL HUMANITIES AND SEMIOTICS 79

complex models, such as arguments (Stjernfelt 2014; Olteanu et al. 2020). In the scope of
DH, such modelling operations can therefore also be multi-layered and distributed across
human and technical objects (see Smithies, ffrench and Ciula, forthcoming), and can be
communicated in the form of existential (or conceptual) graphs, or any other system of logical
representation based on the principle of iconicity.
Another more recent concept that biosemiotics can provide for such an analysis is
that of semiotic scaffolding (e.g. Hoffmeyer 2007). From this perspective, modelling in
a digital and computational environment must be understood as supported (scaffolded)
by pre-existing human environmental and technological affordances. The future-
oriented question, in turn, is what can and does modelling for-and-with digital tools and
computational methods support further on. Such an approach is inherent of the positions
of the iconic, multimodal and embodiment turns and can be further employed in the
conceptualization of modelling in the digital humanities.
Furthermore, for his semiotic modelling theory, Sebeok had another foundational
source besides Peirce and Lotman, namely Jakob von Uexküll’s (1926) theoretical
biology. One of the great merits of Uexküll’s biology, which Sebeok eagerly adopted,
is the merging of the notions of modelling and environment in the celebrated Umwelt
(arguably, a more obvious connection for a German-speaking scholar, such as Uexküll,
than for a non-German one). Sebeok’s biosemiotics is rooted in the analysis of how cycles
of perceptions and actions form an organism’s model(s) of the world (i.e. Umwelt) in
terms of Peircean categories and sign types. Discussing both human apprehension and the
work with and of computers in terms of modelling allows for a comparison of the two,
which brings a compelling answer to McCarty’s (2005: 21) foundational idea for the
DH that computer systems must have non-virtual corresponding models. Contradictions
(aniconicity) between human semiotic competences and given digital modelling methods
are thus cause for rethinking digital methods or questioning human understanding. The
friction, when it occurs and is noticed, has epistemological potential.

DIGITAL MODELLING AND

THE NATURAL ENVIRONMENT
Further, this leads to interrogating the environmental aspects and implication of DH
modelling. Biosemiotics produced an ecological semiotics theory, named ecosemiotics,
on the ground that organisms have inner cognitive models of the environment (Nöth
1998), coinciding with the Uexküllean Innenwelt. Since these inner cognitive models
are based on primary modelling, they are schematic and, thus, the scope of ecosemiotics
is that of investigating ‘the impact of maps on the mapped’ (Maran and Kull 2014:
41). The assumption here is that organisms’ mapping of the environment impacts on
the environment – a rather recent realization, missed by modern philosophy and its
corresponding medialities. Thus, the question that all of this points to is the following:
What do digital models instantiated in artefacts imply for the environment? In this
regard, there is an unmissable hope, which scholarship has only scratched the surface
of, and which semiotics could equip DH to exploit. According to Rifkin (2011), since
digital media newtoks and renewable energy grids can be merged, then digital media
can bring about a ‘biosphere consciousness’ (2011). Kull’s argument that semiotics and
ecology are easily compatible because of the similarity of their underpinning models,
as networks, endorses this (2003, see above). Also, inspired by Lotman’s cultural
80 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

theory, Hartley (2015) argues that the digitally modelled creative industries require
a theoretical approach that takes the biosphere as a starting point. These are research
avenues that semiotics, as a modelling theory, can help DH and data science more
generally explore, providing both the theoretical justification and methodological
means for doing so.

CONCLUSIONS
This overview points to the vast possibilities of semiotic research methods in the
scope of DH. While these two disciplines have many common interests and (at times,
apparently accidental) intersections, we consider that a solid theoretical framing for their
interdisciplinary merging revolves around the notions of model and modelling. These
reveal common research avenues around key concepts such as mediality, multimodality,
literacy, embodiment and language, to name some. Literature in this regard started to
develop recently, but this new common philosophical direction of research remains
largely unexplored. The digital turn in the humanities gives way to a semiotic approach
by its consideration of the meaning effects that modelling unavoidably has. The reason
for which DH finds modelling a central and framing concept has to do with modelling
processes in general, often considered independently of mediality (see Bod 2018). By
foregrounding the modelling work of humans in general, the digital turn in the humanities
gives new scope to the schools of semiotics which have been traditionally model-centred.
Processes of digit(al)ization reveal that meaning and, therefore, knowledge are always
situated and embodied. This also highlights that media are semiotically non-neutral. In
one of his last contributions to biosemiotics, Jesper Hoffmeyer (2018: 5) confirmed that
digitization illustrates that knowledge is by default embodied:
Knowledge is a bodily phenomenon, which is why we can’t obtain it – as people
often seem to believe – simply by summing up information. The only reason why we
can so easily retrieve any thinkable piece of information, is that Google already has
digitised it. But in the very process of digitizing it, the information necessarily loses
its natural connection to the rest of the world. Information is converted to fact. But
this implies that the information must be given a new context – or otherwise remain
incomprehensible to the human mind. Most of the time we manage this without giving
it another thought, and experience the information as meaningful. This means that the
information has now – unknowingly perhaps – become re-contextualised. Without
this recontextualisation, we would have only the ‘naked information’, and no matter
how many pieces of such ‘naked information’ we add, we will never reach any kind of
understandable knowledge.

NOTES
1 As initiatives such as https://4humanities.org/ demonstrate.
2 See for example van Zundert et al. (2017) for code criticism in relation to the performative
dimension of programming or Galey and Ruecker (2010) for prototypying as a form of
argument in design practices.
3 See https://cistudies.org/
4 See the experiments in https://dral.kdl.kcl.ac.uk/
DIGITAL HUMANITIES AND SEMIOTICS 81

­5 Note that philological, editorial and bibliographic studies informed by material culture
approaches have studied the complex structure and layout of visual signs in print material
and informed document and interface theories in humanities computing and DH (see, e.g.,
McGann 2014).
6 See, e.g. http://dataforhistory.org/ and https://lincsproject.ca/
7 With important exceptions, unsurprisingly, in the study of the role of modelling and models
in the natural, physical and social sciences (see Nersessian 2008 and Morgan 2012).
8 See the multi-dimensional definition of cultural literacy proposed at http://cleurope.eu/about/
key-concepts/
9 See, for example, the Research Software Engineering career pathways emerging in laboratories
such as King’s Digital Lab: https://www.kdl.kcl.ac.uk/blog/rse-career-development/; Smithies
and Ciula (2020).
10 See, for example, https://dh2018.adho.org/en/precarious-labor-in-the-digital-humanities/

REFERENCES
Anderson, B. ([1983] 2006), Imagined Communities: Reflections on the Origin and Spread of
Nationalism, revised edn, London: Verso.
Bankov, K. (2022). ‘The Digital Mind: Semiotic Explorations in Digital Culture’, Cham: Springer.
Bawden, D. (2008), ‘Origins and Concepts of Digital Literacy’, in C. Knobel and M. Lankshear
(eds), Digital Literacies, 17–32, New York: Peter Lang.
Berry, D. M., ed. (2012), Understanding Digital Humanities, New York: Palgrave Macmillan.
Bezemer, J. and G. Kress (2008), ‘Writing in Multimodal Texts: A Social Semiotic Account of
Designs for Learning’, Written Communication, 25 (2): 166–95.
Bod, R. (2018), ‘Modelling in the Humanities: Linking Patterns to Principles’, Historical Social
Research, Supplement, 31: 87–95.
Bode, K. (2020), ‘Why You Can’t Model Away Bias’, Modern Language Quarterly, 81 (1):
95–124.
Boehm, G. and W. J. T. Mitchell (2009), ‘Pictorial Versus Iconic Turn: Two Letters’, Culture,
Theory and Critique, 50 (2–3): 103–21.
boyd, d. (2014), It’s Complicated: the Social Lives of Networked Teens, New Haven: Yale
University Press.
Boot, P. (2006), ‘Decoding Emblem Semantics’, Literary and Linguistic Computing, 21 (suppl.
1): 15–27.
Brennen, J. S. and D. Kreiss (2016), ‘Digitalization’, in K. Jensen, C. T. Robert, J. D. Pooley
and E. W. Rothenbuhler (eds), The International Encyclopedia of Communication Theory
and Philosophy, Oxford: Wiley Blackwell.
Campbell, C., A. Olteanu and K. Kull (2020), ‘Learning and Knowing as Semiosis: Extending
the Conceptual Apparatus of Semiotics’, Sign Systems Studies, 47 (3–4): 352–81.
Ciula, A. (2017), ‘Digital Palaeography: What Is Digital about It?’ Digital Scholarship in the
Humanities, 32 (2): ii89–ii105.
Ciula, A. and C. Marras (2016), ‘Circling around Text and Language: Towards “Pragmatic
Modelling” in Digital Humanities’, Digital Humanities Quarterly, 10 (3): 17–29.
Ciula, A. and C. Marras (2019), ‘Exploring a Semiotic Conceptualisation of Modelling in Digital
Humanities Practices’, in A. Olteanu, A. Stables and B. Dumitru (eds), Meanings & Co.: The
Interdisciplinarity of Communication, Semiotics and Multimodality, 33–52, Cham: Springer.
­Ciula, A. and Ø. Eide (2017), ‘Modelling in Digital Humanities: Signs in Context’, Digital
Scholarship in the Humanities, 32 (1): i33–i46.
82 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Ciula, A., G. Noël, P. Caton, G. Ferraro, B. Maher, and M. Vieira. ‘The Place of Models
and Modelling in Digital Humanities: Intersections with a Research Software Engineering
Perspective’. In Book of Abstracts of Digital Humanities 2020. The Alliance of Digital
Humanities Organizations. http://dx.doi.org/10.17613/789z-9p59.
Ciula, A., G. Noël, P. Caton, G. Ferraro, T. Ong, J. Smithies, and M. Vieira. ‘The Place of
Models and Modelling in Digital Humanities: Some Reflections from a Research Software
Engineering Perspective’, in J. Burkhardt, E. Arestau, N. Herold, R. Sierig and M. Annisius
(eds). Elisabeth Burr Festschrift, Leipzig: Universitätsbibliothek, Forthcoming.
Clivaz, C. (2017), ‘Multimodal Literacies and Continuous Data Publishing: Une Question de
Rythme’, in P. Boot, A. Cappellotto, D. Woot, F. Fischer, A. Kelly, A. Mertgens, A. M.
Sichani, E. Spadini and H. van Hulle (eds), Advances in Digital Scholarly Editing, Papers
presented at the DiXiT conferences in The Hague, 99–104, Leiden: Sidestone Press.
Cobley, P. (2010), ‘The Cultural Implications of Biosemiotics’, Biosemiotics, 3: 225–44.
Cobley, P. (2016), Cultural Implications of Biosemiotics, Dordrecht: Springer.
Cobley, P. (2017), ‘What the Humanities Are for: A Semiotic Perspective’, in K. Bankov and P.
Cobley (eds), Semiotics and Its Masters, 3–23, Berlin: de Gruyter Mouton.
Danesi, M. (2002), Understanding Media Semiotics, London: Arnold.
Dourish, P. (2014), ‘No SQL: The Shifting Materialities of Database Technology’,
Computational Culture, 4, http://computationalculture.net/no-sql-the-shifting-materialities-
ofdatabase-technology/
Drucker, J. (2011), ‘Humanities Approaches to Graphical Display’, Digital Humanities
Quarterly, 5 (1), http://digitalhumanities.org/dhq/vol/5/1/000091/000091.html
Drucker, J. (2013), ‘Performative Materiality and Theoretical Approaches to Interface’, Digital
Humanities Quarterly, 7 (1), http://www.digitalhumanities.org/dhq/vol/7/1/000143/000143.html
Eide, Ø. (2015), Media Boundaries and Conceptual Modelling: Between Texts and Maps,
London: Palgrave Macmillan.
Elleström, L. (2013), ‘Spatiotemporal Aspects of Iconicity’, in L. Elleström, O. Fischer and C.
Ljungberg (eds), Iconic Investigations, 95–117, Amsterdam: John Benjamins.
Elleström, L. (2014), Media Transformation: The Transfer of Media Characteristics among
Media, London: Palgrave Macmillan.
Elleström, L. (2017), ‘Transfer of Media Characteristics among Dissimilar Media’, Palabra
Clave, 20 (3): 663–85.
Elleström, L. (2018), ‘A Medium-centered Model of Communication’, Semiotica, 224: 269–93.
Floridi, L. (2014), The Fourth Revolution: How the Infosphere Is Reshaping Human Reality,
Oxford: Oxford University Press.
Galey, A. and S. Ruecker (2010), ‘How a Prototype Argues’, Literary and Linguistics
Computing, 25 (4): 405–24.
Gibson, J. J. (1979), The Ecological Approach to Visual Perception, Boston: Houghton Mifflin.
Génova, G., M. C. Valiente and M. Marrero (2009), ‘On the Difference between Analysis
and Design, and Why It Is Relevant for the Interpretation of Models in Model Driven
Engineering’, The Journal of Object Technology, 8 (1): 107–27.
Godfrey-Smith, P. (2009), ‘Models and Fictions in Science’, Philosophical Studies, 143 (1): 101–16.
Gorlée, D. L. (1994), Semiotics and the Problem of Translation: With Special Reference to the
Semiotics of Charles S. Peirce, Amsterdam: Rodopi.
Gould, S. J. and E. S. Vrba (1982), ‘Exaptation – A Missing Term in the Science of Form’,
Paleobiology, 8 (1): 4–15.
­Gray, J. (2018), ‘Three Aspects of Data Worlds’, Krisis: Journal for Contemporary Philosophy,
37 (1): 4–17, https://archive.krisis.eu/three-aspects-of-data-worlds/
DIGITAL HUMANITIES AND SEMIOTICS 83

Halliday, M. A. K. (1978), Language as Social Semiotic: The Social Interpretation of Language

and Meaning, London: Edward Arnold.
Halliday, M.A.K. and J. J. Webster, eds (2009), Continuum Companion to Systemic Functional
Linguistics, New York: Continuum.
Hartley, J., ed. (2005), Creative Industries, Oxford: Blackwell Publishing.
Hartley, J. (2011), The Uses of Digital Literacy, London: Transaction Publishers.
Hartley, J. (2015), ‘Urban Semiosis: Creative Industries and the Clash of Systems’, International
Journal of Cultural Studies, 18 (1): 79–101.
Hoffmeyer, J. (2007), ‘Semiotic Scaffolding of Living Systems’, in M. Barbieri (ed.),
Introduction to Biosemiotics: The New Biological Synthesis, 49–166, Dordrecht: Springer.
Hoffmeyer, J. (2018), ‘Knowledge Is Never Just There’, Biosemiotics, 11 (1): 1–5.
Jakobson, R. (1959), ‘On Linguistic Aspects of Translation’, in R. A. Brower (ed.), On
Translation, 232–9, Cambridge: Harvard University Press.
Jakobson, R. ([1965] 1987), ‘Quest for the Essence of Language’, in K. Pomorska and S. Rudy
(eds), Language in Literature, 413–27, Cambridge: Belknap Harvard.
Jay, P. (2014), The Humanities ‘Crisis’ and the Future of Literary Studies, New York: Palgrave
Macmillan.
Jewitt, C. (2006), Technology, Literacy and Learning: A Multimodal Approach, Abingdon:
Routledge.
Johnson, M. (1987), The Body in the Mind: The Bodily Basis of Meaning, Imagination and
Reason, Chicago and London: The University of Chicago Press.
Kourdis, E. (2015), ‘Semiotics of Translation: An Interdisciplinary Approach to Translation’, in
P. P. Trifonas (ed.), International Handbook of Semiotics, 303–20, Dordrecht: Springer.
Kralemann, B. and C. Lattmann (2011), ‘The Semantics of Models: A Semiotic Philosophy
of Science Approach’, in K. D. Schewe and B. Thalheim (eds), Semantics in Data and
Knowledge Bases, 50–69, Dordrecht: Springer.
Kralemann, B. and C. Lattmann (2013), ‘Models as Icons: Modelling Models in the Semiotic
Framework of Peirce’s Theory of Signs’, Synthese, 190: 3397–420.
Kress, G. (2010), Multimodality: A Social Semiotic Approach to Contemporary Communication,
London: Routledge.
Kress, G. and T. van Leeuwen (2001), Multimodal Discourse: The Modes and Media of
Contemporary Communication, London: Arnold.
Kress, G. and T. van Leeuwen ([1996] 2006), Reading Images: The Grammar of Visual Design,
2nd edn, London: Routledge.
Kress, G. and S. Selander (2012), ‘Multimodal Design, Learning and Cultures of Recognition’,
Internet and Higher Education, 15: 265–8.
Kull, K. (2003), ‘Ladder, Tree, Web: The Ages of Biological Understanding’, Sign Systems
Studies, 31 (2): 589–603.
Lacković, N. (2018), ‘Analysing Videos in Educational Research: An “Inquiry Graphics”
Approach for Multimodal, Peircean Semiotic Coding of Video Data’, Video Journal of
Education and Pedagogy, 3 (6): 1–23.
Lacković, N. (2020), ‘Thinking with Digital Images in the Post-truth Era: A Method in Critical
Media Literacy’, Postdigital Science and Education, 2: 442–62.
Lacković, N. and A. Olteanu (2020), ‘Rethinking Educational Theory and Practice in Times of
Visual Media: Learning as Image-concept Integration’, Educational Philosophy and Theory,
53 (6): 597–612. DOI: 10.1080/00131857.2020.1799783.
­Lakoff, G. (1987), Women, Fire, and Dangerous Things: What Categories Reveal about the
Mind. Chicago, London: University of Chicago Press.
84 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Lakoff, G. and M. Johnson (1999), Philosophy in the Flesh: The Embodied Mind and Its
Challenge to Western Thought, New York: Basic Books.
Lankshear, C. and M. Knobel, eds (2008), Digital Literacies: Concepts, Policies and Practices,
New York: Peter Lang.
Lotman, J. (1977), ‘Primary and Secondary Communication-modelling Systems’, in D. P. Lucid
(ed.), Soviet Semiotics: An Anthology, Baltimore: Johns Hopkins University.
Lotman, J. (1991), Universe of the Mind: A Semiotic Theory of Culture, trans. A. Shukman,
London: I.B. Tauris.
Mahoney, M. (2000), ‘Historical Perspectives on Models and Modelling’, Zürich: DHS–
DLMPS Joint Conference 13, on ‘Scientific Models: Their Historical and Philosophical
Relevance’, www.princeton.edu/~hos/Mahoney/articles/models/models.html
Mahoney, M. (2005), ‘The Histories of Computing(s)’, Interdisciplinary Science Reviews, 30
(2): 119–35.
Majchrzak, A., S. Faraj, G. C. Kane and B. Azad (2013), ‘The Contradictory Influence of
Social Media Affordances on Online Communal Knowledge Sharing’, Journal of Computer-
Mediated Communication, 19 (1): 38–55.
Martinelli, D. (2016), Arts and Humanities in Progress: A Manifesto of Numanities, Cham:
Springer.
Marwick, A. E. and d. boyd (2010), ‘I Tweet Honestly, I Tweet Passionately: Twitter Users,
Context Collapse, and the Imagined Audience’, New Media & Society, 13 (1): 114–33.
Maran, T. and K. Kull (2014), ‘Ecosemiotics: Main Principles and Current Developments’,
Geografiska Annaler: Series B, Human Geography, 96 (1): 41–50.
McCarty, W. (2005), Humanities Computing, New York: Palgrave Macmillan.
McGann, J. (2014), A New Republic of Letters: Memory and Scholarship in the Age of Digital
Reproduction, Cambridge: Harvard University Press.
McGillivray, B., B. Alex, S. Ames, G. Armstrong, D. Beavan, A. Ciula, G. Colavizza, J.
Cummings, D. De Roure, A. Farquhar, S. Hengchen, A. Lang, J. Loxley, E. Goudarouli,
F. Nanni, A. Nini, J. Nyhan, N. Osborne, T. Poibeau, M. Ridge, S. Ranade, J. Smithies,
M. Terras, A. Vlachidis and P. Willcox (2020), The Challenges and Prospects of the
Intersection of Humanities and Data Science: A White Paper from The Alan Turing Institute,
Figshare, dx.doi.org/10.6084/m9.figshare.12732164
McLuhan, M. (1995), Essential McLuhan, ed. E. McLuhan and F. Zingrone, London:
Routledge.
Merleau-Ponty, M. (1995), La Nature. Notes. Cours de Collège de France, Paris: Seuil.
Mills, K. A. (2010), ‘A Review of the “Digital Turn” in the New Literacy Studies’, Review of
Educational Research, 80 (2): 246–71.
Mills, K. A. (2016), Digital Literacy Theories for the Digital Age: Social, Critical, Multimodal,
Spatial, Material and Sensory Lenses, Bristol, Buffalo, Toronto: Multilingual Matters.
Mittelberg, I. (2013), ‘The Exbodied Mind: Cognitive-semiotic Principles as Motivating
Forces in Gesture’, in C. Müller, A. Cienki, E. Fricke, S. Ladewig, D. McNeil and S.
Teßendorf (eds), Body – Language – Communication: An International Handbook in Human
Interaction, vol. 1, 755–84, Berlin: De Gruyter Mouton.
­Mittelberg, I. (2019), ‘Visuo-kinetic Signs Are Inherently Metonymic: How Embodied
Metonymy Motivates Forms, Functions, and Schematic Patterns in Gesture’, Frontiers in
Psychology, 10: Article 254.
Morgan, M. (2012), The World in the Model, Cambridge: Cambridge University Press.
Moxey, K. (2008), ‘Visual Studies and the Iconic Turn’, Journal of Visual Culture, 7 (2):
131–46.
DIGITAL HUMANITIES AND SEMIOTICS 85

Myrdene, A. and F. Merrell, eds (1991), On Semiotic Modelling, Berlin: Mouton de Gruyter.
Nersessian, N. J. (2008), Creating Scientific Concepts, Cambridge: MIT Press.
Nöth, W. (1998), ‘Ecosemiotics’, Sign Systems Studies, 26: 332–43.
Nöth, W. (2012), ‘Translation and Semiotic Mediation’, Sign Systems Studies, 40 (3–4):
279–98.
Nöth, W. (2018), ‘The Semiotics of Models’, Sign Systems Studies, 46 (1): 7–43.
Nussbaum, M. C. (2010), Not for Profit: Why Democracy Needs the Humanities, Princeton:
Princeton University Press.
O’Halloran, K. (2015), ‘Multimodal Digital Humanities’, in P. P. Trifonas (ed.), International
Handbook of Semiotics, 389–415, Dordrecht: Springer.
O’Halloran, K. L., M. K. L. E., A. Podlasov and S. Tan (2013), ‘Multimodal Digital Semiotics:
The Interaction of Language with Other Resources’, Text and Talk: Special Edition for
Michael Halliday, 33 (4–5): 665–90.
O’Halloran, K. L., A. Chua and A. Podlasov (2014), ‘The Role of Images in Social Media
Analytics: A Multimodal Digital Humanities Approach’, in D. Machin (ed.), Visual
Communication, 565–88, Berlin: de Gruyter.
O’Halloran, K. L., S. Tan and P. Wignell (2016), ‘Intersemiotic Translation as Resemiotization:
A Multimodal Perspective’, Signata: Annals des sémiotiques/Annals of Semiotics, 7: 199–229.
O’Toole, M. ([2010] 2011), The Language of Displayed Art, 2nd edn, London: Routledge.
Olteanu, A. (2019), Multiculturalism as Multimodal Communication: A Semiotic Perspective,
Cham: Springer.
Olteanu, A. (2020), ‘Translation from a Contemporary Media Perspective: Avoiding
Culturalism and Monolingualism’, Social Semiotics, aop: doi.org/10.1080/10350330.2020.
1714204.
Olteanu, A., C. Campbell and F. Sebastian (2020), ‘Naturalizing Models: New Perspectives in a
Peircean key’, Biosemiotics, 13: 179–97.
Pajević, M. (2014), ‘Translation and Poetic Thinking’, German Life and Letters, 67 (1): 6–21.
Peirce, C. S. ([1866–1913] 1931–58), The Collected Papers of Charles Sanders Peirce, vols. 1–6,
C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University
Press. Cited as CP.
Pesce, S. (2010), ‘From Semiotics of Teaching to Educational Semiotics’, in I. Semetsky (ed.),
Semiotics, Education, Experience, 99–113, Rotterdam: Sense Publishers.
Pesce, S. (2020), ‘From Analogical to Analytical Thinking and Back: The Adaptation of
Teacher’s Reasoning to Complex Situations’, in H. Yama and V. Salvano-Pardieu (eds),
Adapting Human Thinking and Moral Reasoning in Contemporary Society, 223–44. IGI
Global.
Pérez-González, L. (2014), ‘Multimodality and Translation and Interpreting Studies:
Theoretical and Methodological Perspectives’, in S. Bermann and C. Porter (eds), A
Companion to Translation Studies, 119–31, Oxford: Wiley Blackwell.
Petrilli, S. (2003), ‘Translation and Semiosis: Introduction’, in S. Petrilli (ed.), Translation,
Translation, 17–37, Amsterdam: Rodopi.
Pietarinen, A. V. (2006), Signs of Logic: Peircean Themes on the Philosophy of Language,
Games, and Communication, Dordrecht: Springer.
Rifkin, J. (2011), The Third Industrial Revolution: How Lateral Power Is Transforming Energy,
the Economy, and the World, London: PalgraveMacmillan.
­Scolari, C. A., M. J. Masanet, M. Guerrero-Pico and M. J. Establés (2018), ‘Transmedia
Literacy in the New Media Ecology: Teens’ Transmedia Skills and Informal Learning
Strategies’, El Profesional de la Informacion, 27 (4): 801–12.
86 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Sebeok, T. (1991), A Sign Is Just a Sign: Advances in Semiotics, Bloomington: Indiana University
Press.
Sebeok, T. A. and M. Danesi (2000), The Forms of Meaning: Modelling Systems Theory and
Semiotic Analysis, Berlin: Mouton de Gruyter.
Smithies, J. (2017), The Digital Humanities and the Digital Modern, London: Palgrave
Macmillan.
Smithies, J. and A. Ciula (2020), ‘Humans in the Loop: Epistemology & Method in King’s
Digital Lab’, in K. Schuster and S. Dunn (eds), Routledge International Handbook of
Research Methods in Digital Humanities, 155–72, London: Routledge.
Smithies, J., P. ffrench, and A. Ciula. ‘Droit de Cité: Towards a Socialized Model for
Technology Development in the Arts & Humanities’, in U. Pawlicka-Deger and C. Thomson
(eds.) Digital Humanities and Laboratories: Perspectives on Knowledge, Infrastructure and
Culture, edited. Routledge, Forthcoming.
Stables, A. (2012), Be(com)ing Human: Semiosis and the Myth of Reason, Rotterdam: Sense
Publishers.
Stables, A. (2016), ‘Education as Process Semiotics: Towards a New Model of Semiosis for
Teaching and Learning’, Semiotica, 212: 45–58.
Stables, A., W. Nöth, A. Olteanu, S. Pesce and E. Pikkarainen (2018). Semiotic Theory of
Learning: New Perspectives in the Philosophy of Education, London: Routledge.
Stecconi, U. (1994), ‘Peirce’s Semiotics for Translation’, Koiné, 4: 161–80.
Stecconi, U. (2004), ‘Interpretive Semiotics and Translation Theory: The Semiotic Conditions
to Translation’, Semiotica, 150: 471–89.
Stjernfelt, F. (2006), ‘The Semiotic Body. A Semiotic Concept of Embodiment?’ in W. Nöth
(ed.), Semiotic Bodies, Aesthetic Embodiments and Cyberbodies, 13–48, Kassel: Kassel
University Press.
Stjernfelt, F. (2007), Diagrammatology: And Investigation on the Borderlines of Phenomenology,
Ontology and Semiotics, Dordrecht: Springer.
Stjernfelt, F. (2014), Natural Propositions: The Actuality of Peirce’s Doctrine of Dicisigns,
Boston: Docent Press.
Stjernfelt, F. (2015), ‘Iconicity of Logic and the Roots of the “Iconicity” Concept’, in M. K.
Hiraga, W. J. Herlofsky, K. Shinohara and K. Akita (eds), Iconicity: East meets West, 35–53,
Amsterdam: John Benjamins.
Sowa, J. F. ([1999] 2000), Knowledge Representation: Logical, Philosophical, and
Computational Foundations, Pacific Grove: Brooke Cole Publishing Co.
Sowa, J. F. (2008), ‘Conceptual Graphs’, Foundations of Artificial Intelligence, 3: 213–37.
Sowa, J. F. (2014), ‘From Existential Graphs to Conceptual Graphs’, in L. Johnston, K. Wolfe,
J. Yoder and A. DeMarco (eds), Computational Linguistics: Concepts, Methodologies, Tools,
and Applications, 439–72, Hershey: IGI Global.
Torresi, I. ([1998] 2009), ‘Advertising’, in M. Baker and G. Saldanha (eds), Routledge
Encyclopedia of Translation Studies, 2nd edn, 6–10, Oxford: Routledge.
Treem, J. W. and P. M. Leonardi (2013), ‘Social Media Use in Organizations: Exploring
Affordances of Visibility, Editability, Persistence, and Association’, Annals of the
International Communication Association, 36 (1): 143–89.
Uexküll, J. von. (1926), Theoretical Biology, London: Kegan Paul, Trench, Trubner & Co.
van Zundert, J. J. and R. Haentjens Dekker (2017), ‘Code, Scholarship, and Criticism: When
Is Coding Scholarship and When Is It Not?’ Digital Scholarship in the Humanities, 32
(suppl. 1): i121–i123.
 ­C HAPTER FOUR

Systems Theory and

Semiotics
RICARDO GUDWIN AND JOÃO QUEIROZ

INTRODUCTION
Here, we explore the idea that a systems theory-based approach can help us to define
more accurately many of the properties of semiosis for C. S. Peirce. For the sake of
our arguments, an approach to semiosis (from Peirce’s mature semiotics) requires the
notion of an open self-organized emergent and anticipatory system evolving in time.
This theoretical position has many implications. A number of investigators maintain that
semiosis (meaning process), and its metaphysical counterpart, must be considered in terms
of complex adaptive systems (see Merrell and Queiroz 2009; Loula and Queiroz 2011;
Bickhard 2007; Steels 1999; Steels and Kaplan 1999; Wagner et al. 2003; Christiansen
and Kirby 2003; Cangelosi 2002; Cangelosi and Parisi 1998; Cangelosi and Turner 2002;
Vogt 2007; Vogt and Coumans 2003; Jung and Zelinsky 2000; Briscoe 1998).
We can say that, from a Peircean perspective, there are reactive and semiotic systems.
For the cognitive scientists Nolfi and Floreano (2002: 93), reactive systems are
systems in which sensors and motors are directly linked and which always react to the
same sensory state with the same motor action. In these systems, internal states, if any,
play a comparatively limited role in determining the motor output of the system.
By contrast, in semiotic systems, some (internal or external) states are crucial for the
interpretation of other states perceived through sensory apparatuses, and, as sign
interpretation plays a fundamental role in the processes through which the system
answers to environmental cues, different actions can result from the same sensory state,
depending on the relationships established within the system and between the system
and its semiotic niche. A ‘semiotic system’ is a system that produces, transmits, receives,
processes and interprets signs of different types. As Fetzer (1997: 358) writes: ‘What
makes a system “semiotic” thus becomes that the behaviour of the system is causally
affected by the presence of a sign because that sign stands for something else iconically,
indexically, or symbolically, for that system.’
Signs can be found in many domains and levels of descriptions – physico-chemical
(physiosemiosis), biological (biosemiosis), animal (zoosemiosis), ecological (ecosemiosis),
computational (artificial semiosis), cognitive, social, etc. By interpreting signs, semiotic
systems show self-corrective behaviour (Ransdell 1977: 162). Such self-corrective
behaviour depends on the capability of semiotic systems of using signs as media for
88 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

the communication of habits (regular patterns of semiotic activity) from objects to

interpretants so as to constrain their own behaviour (Queiroz and El-Hani 2006).

PEIRCE’S PROCESS SEMIOTICS: SOME BASIC NOTIONS

Peirce’s process semiotics is grounded on a list of logical-phenomenological
categories – Firstness, Secondness, Thirdness – which corresponds to an exhaustive
system of hierarchically organized classes of relations (Houser 1997). This system makes
up the formal foundation of his model of semiosis as a process and of his classifications
of signs (Murphey 1993: 303–6). Firstness as a mode of being is related to the modality
of possibility. It is the category of vagueness and novelty – ‘the mode of being which
consists in its subject’s being positively such as it is regardless of anything else. That can
only be a possibility’ (Peirce 1903: CP 1.25). Secondness is the mode of being ‘which
is as it is relatively to a Second but regardless of any Third’ (1898: CP 6.200). It is a
kind of reaction. Like Firstness, Secondness can be related to a modality, namely, the
modality of actuality (Peirce 1908: CP 6.455; Parker 1998). The actuality of a thing
is simply its occurrence. Rephrased, actuality is the realization of a possibility, without
thereby making reference to something larger, be that a general law or an interpretation.
Peirce considered ‘the idea of any dyadic relation not involving any third as an idea of
secondness’ (1904: CP 8.330). Thirdness is the category of mediation, habit, generality
and conceptualization (Peirce 1895: CP 1.340). The example par excellence is Peirce’s
semiotic process (semiosis) in which a sign is related to an object by mediation through
an interpretant.
Semiosis implies process (Queiroz and Merrell 2008). Indeed, one of the most
remarkable characteristics of Peirce’s theory of signs is its processual nature. As a genuine
process thinker, it was quite natural that Peirce conceived semiosis as a process in which
triads (sign-object-interpretant) are systematically linked to one another so as to form a
web (Gomes et al. 2007). In this regard, we follow Rescher in his definition of a process
as ‘a coordinated group of changes in the complexion of reality, an organized family of
occurrences that are systematically linked to one another either causally or functionally’
(1996: 38).
Peirce developed, between 1867 and 1908–11, a model of sign as process, and
designed elaborated divisions of signs in order to describe these processes. According to
Peirce, semiosis (sign in action) entails an irreducible triadic relation between a sign, its
object and its interpretant (1902: CP 2.171; 1903: 2.274). That is, according to Peirce,
any description of semiosis involves a relation constituted by three irreducibly connected
terms, which are its minimal constitutive elements (1907: MS 318.81; 1903: CP 2.242).
In Peirce’s words:

My definition of a sign is: A Sign is a Cognizable that, on the one hand, is so determined
(i.e., specialized, bestimmt) by something other than itself, called its Object, while, on
the other hand, it so determines some actual or potential Mind, the determination
whereof I term the Interpretant created by the Sign, that that Interpreting Mind is
therein determined mediately by the Object.
(n.d.: CP 8.177; emphasis in the original)

­In his ‘most fundamental division of signs’ (1903: CP 2.275), Peirce characterized
icons, indexes and symbols as matching, respectively, relations of similarity, contiguity and
SYSTEMS THEORY AND SEMIOTICS 89

law between S and O (sign-object relation) in the triad S-O-I. Icons are signs which stand
for their objects through similarity or resemblance (Peirce 1903: CP 2.276), irrespective
of any spatio-temporal physical correlation that S may have with an existent O (Peirce
1893: CP 2.299). If a determinative relation of the sign (S) by the object (O) is a relation
of analogy, that is, if S is a sign of O in virtue of a certain quality that S and O share, then
S is an icon of O. If S is an icon, then S communicates to I a quality of O: ‘An Icon is a
sign which refers to the Object that it denotes merely by virtue of characters of its own,
and which it possesses, just the same, whether any such Object actually exists or not’
(1903: CP 2.247). In contrast, if S is a sign of O by reason of ‘a direct physical connection’
between them (Peirce 1885: CP 1.372), then S is said to be an index of O. In that case, S
is really determined by O, and both must exist as events: ‘An Index is a sign which refers
to the Object that it denotes by virtue of being really affected by that Object’ (Peirce
1903: CP 2.248). The notion of spatio-temporal co-variation is the most characteristic
property of indexical processes. The examples range from a pronoun, demonstrative or
relative, which ‘forces the attention to the particular object intended without describing it’
(Peirce 1885: CP 1.369), to physical symptoms of diseases, photographs, weathercocks,
thermometers. In a Peircean analysis, small red spots in a child’s skin, for instance, can
be treated as a sign (S) which stands for a disease, say, measles, its object (O), so as to
constrain its interpretant, the effect the red spots have on an interpreter, say, a doctor
performing a diagnosis. The sign processes involved are indexical. Finally, in a symbolic
relation, the interpretant stands for ‘the object through the sign’ by a determinative
relation of law, rule or convention (Peirce 1903: CP 2.276). According to Peirce (1901:
CP 2.307), a symbol is ‘a Sign (q.v.) which is constituted a sign merely or mainly by the
fact that it is used and understood as such, whether the habit is natural or conventional,
and without regard to the motives which originally governed its selection’.
Our effort here is, by approaching semiosis from a perspective of a process and a
dynamical system, to make the understanding of Peirce’s notion more precise. But before
that, we need to introduce some theoretical background on systems theory.

SYSTEMS THEORY
According to von Bertalanffy (1972), the notion of system is as old as European philosophy.
The term system was not born yesterday, out of current questions of mathematics,
science, philosophy and technology. Rather, it is a fruit of a methodological investigation
of perennial problems which have been recognized for centuries and discussed in the
language available at the time. Two principal ideas were advanced in order to deal with
the problem of order and organization, which is subjacent to the idea of a system (von
Bertalanffy 1972). One was the comparison with man-made machines. The other was
to conceive of order as a product of chance (von Bertalanffy 1972). The theory of the
living organism as a machine was constructed by the combination of these two, and was a
natural derivation, which apparently could be explained by the evolution of science (von
Bertalanffy 1972).
Notwithstanding the singular success achieved in the explanation of ever more and finer
life processes, basic questions remained unanswered, at least for a while. The evolution
of machines by events at random appears to be, at the beginning, self-contradictory
(von Bertalanffy 1972). Mitochondrial ‘machines’ of enzymatic organization in even the
simplest cell or nucleoprotein molecules are incomparably more complex than a watch or
other man-made artefacts.
90 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

The evolutionary concept of ‘survival of the fittest’ seems to lead to a circuitous

argument. Self-maintaining systems must first exist before they can enter into competition,
which leaves systems with higher selective value predominant. Self-maintenance,
however, is not provided by the ordinary laws of physics. Rather, the second law of
thermodynamics prescribes that ordered systems in which irreversible processes take
place tend towards most probable states and, hence, towards destruction of existing order
and ultimate decay. Thus, how to explain biological systems as being instances of the
same kind of general principles ruling man-made machines? How to explain the finalistic
behaviour exhibited by living organisms? Since the fundamental character of the living
things is its organization, the customary investigation of the single parts and processes
cannot provide a complete explanation of the vital phenomena. This investigation gives
us no information about the coordination of parts and processes. These questions led
to the creation of a whole research programme in biology which was later called ‘the
organismic programme’ and was the germ of what later became known as general systems
theory (von Bertalanffy 1972).
The old Aristotelian dictum of the whole being more than its parts, which was
originally neglected by the mechanistic conception, and which later led to a vitalistic
demonology, was evolved to a more satisfactory explanation: ‘the properties and modes
of action of higher levels are not explicable by the summation of the properties and modes
of action of their components taken in isolation. If, however, we know the ensemble of
the components and relations existing between them, then the higher levels are derivable
from the components’ (von Bertalanffy 1972: 411). In order to understand an organized
whole, we must know both the parts and relations between them.
Later, the notion of a general systems theory was formulated by von Bertalanffy (1972:
411), orally in the 1930s:
There exist models, principles and laws that apply to generalized systems or their
subclasses irrespective of their particular kind, the nature of the component elements
and the relations or forces between them. We postulate a new discipline called General
System Theory. General System Theory is a logico-mathematical field whose task is the
formulation and derivation of those general principles that are applicable to systems
in general. In this way, exact formulations of terms such as wholeness and sum,
differentiation, progressive mechanization, centralization, hierarchical order, finality
and equifinality, etc, become possible, terms which occur in all sciences dealing with
systems and imply their logical homology.
General systems theory, then, consists of the scientific exploration of ‘wholes’ and
‘wholeness’ which, not so long ago, were considered to be metaphysical notions
transcending the boundaries of science. Novel concepts, methods and mathematical
fields have developed to deal with them. At the same time, the interdisciplinary nature
of concepts, models and principles applying to systems provides a possible approach
towards the unification of science.
Now, with a general unified theory, physical, biological, social, psychological and
higher-order systems might share the same background and conceptual framework.
Even though it was an outstanding progress, things were not that easy. There were
different situations in which General Systems Theory might be employed by the different
sciences. Its formulation might be acceptable and satisfactory for analytical sciences,
like biology (excluded synthetic biology), psychology and social sciences, but it might
become a problem for synthetic sciences, like engineering and computer science. The
SYSTEMS THEORY AND SEMIOTICS 91

finalistic behaviour exhibited by living beings and investigated since Aristotle’s first
proposition of final cause, as the tendency to a final state, was still a challenge for
synthetic computational approaches. As already mentioned, the idea of a final cause, a
purposive, intentional or goal-based behaviour, was criticized for a long time as being a
non-scientific kind of mysticism introduced by Aristotle’s ideas and further explored by
the catholic church (Mayr 1992). General systems theory apparently solved the mystery
by proposing that goal-based behaviour might be just a matter of coordinating both
random events (chance) and mechanical deterministic processes in specific ways. But
how? How to create a mechanistic explanation aligned with general systems theory for
purposive or intentional actions?
A system is usually defined as a set of elements that maintain relations with one another.
‘Elements’ are conceived as primitive entities which are found at each instant in one among
several possible states. Elements establish ‘relations’ when the state of an element depends
on the state of another one. Bunge (1979) defines a system as possessing composition
(the set of its components), structure (the set of relations between its components) and
environment (the set of other systems with which a system establishes relations).
According to Bresciani and D’Ottaviano (2018), and aligned with General Systems
Theory (von Bertalanffy 1950, 1972), a system is a unity, constituted by a non-empty
set of active elements, which maintain to each other, in an organized nature, a given set
of relations with time-invariant characteristics, giving them, as a whole, an identity. In
other words, a system is a whole, composed by a set of interacting parts. Each part (or
element) might have a set of properties which might be fixed or vary through time. These
properties might be observable (which means they are measurable) or non-observable
(which means that despite having some value, this value cannot be measured by a physical
device). Properties might assume different values through time. This set of elements
forms a structure with a given functionality, which gains an identity due to providing this
functionality. Systems can be natural, as in the case of living systems, or artificial, as in
the case of technological artefacts built by men. The organization1 of a system pertains
to the structure connecting the many different system parts, meaning that each part is in
some kind of relation to one or more other parts of the system. This connection might be
physical, or simply logic, implying that there is some relation between two specific parts
within the system. In a man-made technical system, the idea of organization might not
be really related to true organs, as in a living body, but have to do with the existence of
subsystems of the overall system, carrying on some sort of functionality, and having on
themselves some sort of identity as a composite of parts.
These definitions are definitely quite abstract, but can be applied to, since collections
of physical objects (particles, atoms, molecules, physical bodies, etc), up to living beings,
technical artefacts or social entities like business organizations/groups of people, and
either human minds or consciousnesses. But it is important to point out that the term
organization holds a double meaning. It either might designate a fixed structure defining
the system (an atemporal structure or, in a temporal system, its particular structure in
a given instance of time), which we will be calling here an s-organization (meaning an
organization as a structure) or can be related to a dynamical process, where this structure
is modified along time, which we will be calling here a p-organization (meaning an
organization as a process). In other words, the term organization might be related either
to the structure configuring a system in a given instant of time (as e.g. when we talk
about the current organization of a system, an s-organization), or to the process where
this structure might be changing over time (as e.g. when we talk about the process of
92 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

organization of a business company department along history, a p-organization). This

double meaning can be quite misleading, especially if we are talking about a specific
phenomenon: self-organization.
Before touching the issue of self-organization we need first to address a more elementary
phenomenon: dynamical systems.

DYNAMICAL SYSTEMS
A system might be static or dynamical, depending on its relation to time. A static system
does not change through time. A dynamical system, on the contrary, has its structure or
their component properties changing through time. Dynamical systems theory conceives
systems as sets of interdependent variables. By ‘variable’ we mean some entity that can
change, i.e. that can be in different states at different times – similarly to the concept of
element. The state of a system is simply the state or value of all its variables at a given time
t. The behaviour of a system, in turn, consists of transitions between states (Van Gelder
and Port 1995: 5). General Systems Theory describes (represents) a dynamical system in
terms of two distinct features:
1. Structure
2. Behaviour
Static systems don’t have in fact a behaviour, which is a characteristic only of dynamical
systems. A behaviour describes how the system structure or its properties change across
time. Both structure and behaviour can be described (and defined) in different ways. One
popular and useful way is through mathematics. This means that systems components are
referred to as variables, and their relations are described as formulas and equations. This
form of description is very popular in engineering and physical sciences. In computer
sciences, other means for describing systems might be used, such as data structures and
computer programs. Other sciences might prefer other ways of describing systems, as
e.g. a collection of statements (observations, facts) and rules connecting them. These are
more popular in sciences like economy, social sciences and psychology. Additionally,
other means might be used, such as diagrams and figures. In some situations, a mixture of
mathematical/ linguistic/diagrammatic descriptions might be used.
One important concept within general systems theory is the notion of state. The
state of dynamic system is the smallest set of variables (called state variables) such that
knowledge of these variables at t = t0, together with knowledge of the input for t ≥ t0,
completely determines the behaviour of the system for any time t ≥ t0.
Dynamical systems allow the description of a system by means of a process. This
system-as-a-process aspect is crucial for our understanding of semiosis (action of sign),
from a system perspective.

­CLOSED VERSUS OPEN SYSTEMS

A system can be either a closed or an open system (Veeke et al. 2008). A closed system is
a system that does not interact with neither another system nor the environment. It means
that all possible behaviour is determined by only the system itself. An open system, on
the contrary, has an interface, where it might receive inputs from either the environment
or other systems, and also determine outputs, which might affect other systems or the
environment.
SYSTEMS THEORY AND SEMIOTICS 93

DETERMINISTIC VERSUS
NON-DETERMINISTIC SYSTEMS
A system might be deterministic, if its behaviour through time can be precisely defined by
its state, and a given set of rules governing its state transitions. It might be non-deterministic
(Bartlett 1978) if there is any sort of randomness or chance affecting its behaviour,
and as such preventing a precise determination of its current state. Nevertheless, non-
deterministic systems might be described and represented under some sort of imprecision,
as e.g. through states defined under intervals and/or fuzzy variables. Also, many non-
deterministic systems can be represented by means of statistical laws, governing its
behaviour, which is described by means of it following some sort of probabilities.

COMPLEX SYSTEMS
Systems theory can be used in many different scopes involving sciences and humanities.
A system can be since a physical system, which can be investigated and studied using
standard methods from physics, passing through biological organisms in biology, up to
human societies or social groups, which might be investigated and studied by methods
from psychology, anthropology and social sciences. Amongst this multitude of phenomena
which might be addressed by the name of system, an important concept arises – complexity.
The concept of what is complexity is though, at a minimum, quite complicated. For many
researchers, a system can be said to exhibit complexity if its behaviour cannot be easily
inferred from its component properties.
From one side, part of the scientific community believes that complexity is a matter
of compositionality and hierarchy (Salthe 1985). In this sense, a complex system might
be simply a system which can be decomposed into subsystems. This belief comes mainly
from fields of research like physics, chemistry or engineering, where systems of an
arguable complexity (or complicatedness) can be derived by means of compositionality
and hierarchy. But another part of the scientific community (mainly coming from biology)
alerts us that just compositionality and hierarchy might not be enough (Mikulecky 2001).
Particularly, what has been the source of uncountable disputes in the history of science
is what since Aristotle is known as final cause, or teleological behaviour, the tendency of
a system to a final state. Despite its common use in biology to refer to many kinds of an
organism’s behaviour, this notion was criticized for a long time as being a non-scientific
mysticism (Mayr 1992). In biology, a completely new word, teleonomy, was created to
designate an apparent purposefulness or goal-directedness of structures and functions in
living organisms.
It was just after Cybernetics (Rosenblueth et al. 1943; Christensen 1996) that the
real nature of teleological/teleonomic behaviour became clear in science. Nowadays, we
know that this tendency to a final state is just a collateral effect caused by feedback.
Feedback provides an ingredient which is essential to complexity: closure. Connecting
some of the system’s outputs to some of its inputs, these feedbacks create closures, which
provide the missing piece, together with compositionality and hierarchy, in order to
create complexity. In this phenomenon, some system property acts as a reference set
point, becoming a parameter which, entering the feedback loop, drives an update into
another system property value, forcing successive updates in these values until a final
value will lead this property to assume the same value prescribed by the original reference
property. Nowadays, this is a common knowledge widely used by control engineers to
94 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

set up control systems. To summarize, a property which a true complex system has,
and which a simple system (despite its complicatedness, in the sense of composition and
hierarchy) does not exhibit, is the property to anticipate its future states. Among others,
one who studied the phenomena of such anticipatory systems was Robert Rosen (1985).

SELF-ORGANIZED SYSTEMS
According to Debrun (2018a, 2018b), a system can be hetero-organized, if the system
structure is imposed by an external source, or self-organized, if it produces itself,
i.e. if it evolves by itself, as an offspring of an interaction of a set of parts, which
were independent before they gain together its entity status (its identity as a system)
due to providing a functionality. Debrun also differentiates between a primary self-
organization, if the system gains its identity during the initial interaction of its many
parts, before they can be seen as a system, and secondary self-organization, if the system,
after its identity is already determined, is able to preserve its structure, if this structure
is disturbed in any form.
Using our terminology, self-organization is achieved if a system might reach a stable
s-organization due to its p-organization, or similarly, if it reaches its s-organization due
to its p-organization. If this stable s-organization is achieved during the p-organization
process, primary self-organization is achieved. If the system is originally hetero-organized,
but is able to maintain its stable s-organization due to its p-organization, if its structure
is disturbed in some way, we might say that the system reaches just secondary self-
organization.
Usually, machines and other man-made systems are hetero-organized, as their
structure don’t change along time. This is the most common case for man-made systems.
Conversely, natural systems like living beings can be seen as (primary) self-organized
systems. It is important to point out, though, that a system not necessarily must have
material parts. They might be just logical.

EVOLUTIONARY SYSTEMS
Another kind of system which is worth mentioning here is the class of evolutionary
systems (Csányi 1989; Fogel 2006). Evolutionary systems are a kind of intelligent
systems where many aspects of biological evolution are simulated in a computational
environment. Despite its many variations, evolutionary systems usually comprise a
population of elements, which are processed in an iterative way, using combinatorial
operators like crossover, mutation and others, generating new populations along time.
Each element in the population represents the solution of a mathematical problem, in
the form of a structured collection of parameters, which are important for characterizing
the mathematical problem. This element is usually called a chromosome, in a straight
analogy to biological evolution, which can be evaluated by a fitness function, providing a
measuring for the quality of the solution brought by a particular element. With this, in a
given population many different possible solutions can be evaluated, and usually the best
solution is considered as an output from the evolutionary system. The initial population
is usually randomly generated, and as soon as new populations are generated through
this evolutionary process, solutions of a better quality are derived (or ‘evolved’), making
the evolutionary system a kind of optimization process. Each evolutionary step implies
SYSTEMS THEORY AND SEMIOTICS 95

into a first step, where the population first grows, using the many combinatorial process
(like crossover and mutation), and later contracts, when a selection process maintains
in the final population, only the elements considered more apt, i.e. those with a greater
fitness value. With this expansion/contraction movement, the size of a population can be
maintained constant along the generation of successive populations.
Evolutionary systems, as in the case of neural networks, can be classified as self-organized
systems. We might identify either primary or secondary aspect of self-organization in their
functioning. As populations are initially randomly generated (using, of course, pseudo-
random number generators in their implementation in computers), they might require
many steps of evolution before reaching their final configuration, where a good solution
for a mathematical problem is generated. This characterizes its primary self-organization
properties. But then, after good solutions are available in the population, the evolutionary
system preserves those of a better quality in the next populations, meeting the criteria for
being classified also as a secondary self-organized process.

ANTICIPATORY SYSTEMS
An anticipatory system is a system whose current state is determined by a future state,
or, according to Robert Rosen (1985), ‘A system containing a predictive model of itself
and/or its environment, which allows it to change state at an instant in accordance with
the model’s predictions pertaining to a latter instant’. These predictions can be goals,
plans or simply estimations of future states. According to Mihai Nadin (1999; 2003),
‘Anticipation is a recursive process described through the functioning of a mechanism
whose past, present, and future states allow it to evolve from an initial to a final state that
is implicitly embedded in the mechanism’. Anticipatory systems are somewhat different
from the usual kind of systems we generally find in engineering and systems sciences and
have many interesting properties that make them more than pure mechanical deterministic
systems. Rosen (1991) argues that their behaviour is what makes living systems different
from non-living systems. Living systems would be anticipatory systems. Anticipatory
systems may also provide the kind of finalistic behaviour that is particularly related to the
property we use to call ‘intelligence’ in human beings. This finalistic behaviour situates
anticipatory systems as a natural candidate to instantiate the Peircean notion of ‘thirdness’.
Peirce argues that all that can be known must fit into three different categories: Firstness,
Secondness and Thirdness. This is in the kernel of Peircean philosophy. But how to
understand Peirce’s claims if we assume very simple systems as our reality?
A simple system like

1. y(t+1) = Random()

will be a system of pure Firstness.

Supposing that t+1 is equivalent to the present time, we have a system where the present
is completely random.
A system like

2. y(t+1) = f (y(t))

will be a purely deterministic, reactive or mechanical system. This is a system where there
is only Secondness. It is a system where the present is completely determined by the past.
96 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

A system like

3. y(t+1) = f (Random() + y(t))

will be a system where there is Firstness and Secondness.

A candidate for a system with Thirdness will be something like:

4. y(t+1) = f (y(t+τ))

or, in other words, a system where the present depends on the future. But, if time passes
from the past to present, this seems to be impossible. How can this be possible? To
understand that, we need to make a change in the equation:

5. y(t+1) = f (E(y(t+τ),t))

In this case, E(.) is the estimation in time t of a future state y(t+τ). This system is perfectly
feasible. In fact, this estimation is possible, because an anticipatory system carries within
itself a model of its environment and uses this model to anticipate whatever is going to
happen, before it really happens. This predictive capability enables an anticipatory system
to generate future estimations, simulating possible futures and using these estimations
to base its current behaviour. These estimations are as good as the system’s model is
tuned to its environment, but are not perfect. It is important to notice that, despite the
quality of a model, there is always a level of uncorrelation between a model and what it
is supposed to model. Nevertheless, anticipatory systems have an advantage over purely
reactive systems, because they are able to include possible future states in their decision-
making, and with that analyse multiple possible courses of action before deciding for a
specific one, which will force the future to be as previously specified. This ability to forge
the future onto specific goals allows anticipatory systems to exhibit a final-like behaviour,
which is indistinguishable from a finalistic behaviour. This is generally called a goal-based
behaviour.
Several implications of anticipatory systems approach are explored in the context of
AL and AI (artificial life and artificial intelligence) experimental protocols.

ARTIFICIAL SEMIOTIC SYSTEMS

Computational-based methodologies have been used to design virtual/embodied
experimental protocols, where it is possible to simulate the predictions derived from
theoretical models (Parisi 2001), in particular those describing semiotic processes in
artificial semiotic systems. Computer simulations can be used to study different levels of
the organization of semiotic processes (Cangelosi and Turner 2002; Parisi and Cangelosi
2002; Perfors 2002). These levels include the simulation of syntactic structures (Batali
1994, 1998; Kirby 1999), morpho-syntactic compositionality (Christiansen and Ellefson
2002), lexicalization phenomena (Steels 1999; Cangelosi and Parisi 1998; Steels et al.
2002), symbolic competence (Cangelosi 2001), communication (Steels 1997; Steels and
Kaplan 1999) and meaning creation in communication.
Computational approaches have been used to model and simulate semiosis (meaning
processes) from many different perspectives, including Cognitive and Evolutionary
SYSTEMS THEORY AND SEMIOTICS 97

Robotics, Artificial Life, and Synthetic Ethology. Among the advantages of the ‘building-
up synthetic’ approaches is that one can create and modify the parameters defining the
patterns of events, the cognitive architectures of creatures, the environment and the laws
controlling the behaviour of creatures; one can alter each parameter separately, couple
selected parameters, temporally combine their variations in cascade, and observe the
consequences resulting from one or from several of these proceedings; one can replicate
all these proceedings, introduce new parameters and remove old ones; one can see again
the interactive history of particular organisms with conspecifics, with their environments,
and from this information calculate ‘probabilities of meetings of organisms’ or predict
the emergence of certain activities, when some conditions are fixed. What is of particular
importance regarding the premises underlying this chapter, when semiotic process is the
target of simulation, the approaches have worked on different levels including syntactic
properties (Batali 1994; Kirby 1999), morpho-syntactic compositionality (Ellefson and
Christiansen 2000), emergence of grammaticality (Batali 1998), lexicalization phenomena
(Steels et al. 2002; Vogt and Coumans 2003) and communication (Steels and Kaplan
1999; MacLennan 2002).
In spite of these new developments, and in light of recent, undeniably important
technological discoveries and innovations among members of diverse communities of
investigators involved in modelling and simulating projects, there is little regard for the
nature of the genuine, ongoing process of semiosis making emerging from the senses. This
is based on considerable evidence pointing towards the notion that models modelling the
meaning process are lacking a sufficiently strong sense of flexibility of the sort found in
complex semiotic systems.

CONCLUSION
A semiotic system is causally affected by signs in action (semiosis). As we argued, semiotic
systems can be regarded as the embodiment of semiotic processes (Peirce 1868: CP
5.314). Surely, this blurs a distinction between agents and processes. Peirce was a process
thinker, i.e. a representative of a philosophical tendency of treating processes as being
more fundamental than entities as metaphysical and epistemic categories. A process
philosophy can address things and entities as stable bunches of processes. A semiotic
system can be understood in these terms as a stable (both spatially and temporally) cluster
of semiotic processes, or semiosis.
An approach based on the theory of dynamical systems provides a new perspective on
the main properties that characterize semiosis associated with a theoretical framework
not yet explored by Peirce’s scholarship. What are the advantages? Advantages for
whom? One of the advantages related to the use of systems theory is that it is based on
a very formal structure, which can lead to a necessary terminological disambiguation of
Peirce’s semiotics, a mind-tool in understanding such a complex and abstract theory. At
the same time, Peirce’s philosophy of sign provides a powerful collection of theoretical
constraints to model and simulation of semiosis. Such constraints are crucial, for example,
when we design computational simulation protocols for evolutionary semiotic systems.
Once computer simulation becomes tantamount to theoretical simulation, involving
epistemological metaphors of ontological and semiotics versions (see Parisi 2001), the
selection and choice of models can dramatically affect the nature of all work involving
simulation. This is to say, the goal of computer simulation is to pattern theoretical
98 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

simulation through modelling – simulation that is created out of, and supports, the
theory, insofar as that theory is constructed in an effort to pattern some aspect of the
physical world (Hartmann 1996; Humphreys 2004).

NOTE
1 The term organization can be etymologically traced back to the process leading to the
formation of organs in a living body, i.e. a collection of parts of a physical body exhibiting
collectively some sort of functionality, and gaining together the status of an organ due to
performing this functionality (the Greek word organon means ‘tool’).

REFERENCES
Bartlett, M. S. (1978), An Introduction to Stochastic Processes: With Special Reference to
Methods and Applications, Cambridge: Cambridge University Press.
Batali, J. (1994), ‘Innate Biases and Critical Periods: Combining Evolution and Learning in the
Acquisition of Syntax’, in R. Brooks and P. Maes (eds), Proceedings of the Fourth Artificial
Life Workshop, Cambridge, 160–71, MA: MIT Press.
Batali, J. (1998), ‘24 Computational Simulations of the Emergence of Grammar’, Approaches
to the Evolution of Language: Social and Cognitive Bases, 405–27, Cambridge: Cambridge
University Press.
Bickhard, M. (2007), ‘Modern Approaches to Language’, New Ideas in Psychology, 25: 67–9.
Bickhard, M. (2011), ‘Some Consequences (and Enablings) of Process Metaphysics’,
Axiomathes, 21 (1): 3–32.
Bresciani, F. E. and I. M. L. D’Ottaviano (2018), ‘Basic Concepts of Systemics’, in A. Pereira
Jr., R. Gudwin and W. A Pickering (eds), Systems, Self-Organisation and Information: An
Interdisciplinary Perspective, 47–63, London: Routledge.
Briscoe, T. (1998), ‘Language as a Complex Adaptive System: Coevolution of Language and of
the Language Acquisition Device’, in S. Verberne, H. Van Halteren and P. A. J. M. Coppen
(eds), Proceedings of the 8th Meeting of Computational Linguistics in the Netherlands
Conference, 3–40, Nijmegan: Rodopi Amsterdam.
Bunge, M. (1979), Treatise on Basic Philosophy: Ontology II: A World of Systems, vol. 4,
Berlin: Springer Science & Business Media.
Cangelosi, A. (2001), ‘Evolution of Communication and Language Using Signals, Symbols, and
Words’, IEEE Transactions on Evolutionary Computation, 5 (2): 93–101.
Cangelosi, A. and D. Parisi (1998), ‘The Emergence of a Language in an Evolving Population of
Neural Networks’, Connection Science, 10 (2): 83–97.
Cangelosi, A. and D. Parisi (2012), Simulating the Evolution of Language, London: Springer.
Cangelosi, A. and H. Turner (2002), ‘L’emergere del Linguaggio’, in A. M. Borghi and
T. Iachini (eds), Scienze della Mente, 227–44, Bologna: Il Mulino.
Christensen, W. (1996), ‘A Complex Systems Theory of Teleology’, Biology and Philosophy,
11 (3): 301–20.
­Christiansen, M. H. and M. R. Ellefson (2002), ‘Linguistic Adaptation without Linguistic
Constraints: The Role of Sequential Learning in Language Evolution’, in A. Wray (ed.), The
Transition to Language, 335–58, Oxford: Oxford University Press.
Christiansen, M. H. and S. Kirby (2003), ‘Language Evolution: Consensus and Controversies’,
Trends in Cognitive Sciences, 7 (7): 300–7.
SYSTEMS THEORY AND SEMIOTICS 99

Csányi, V. (1989), Evolutionary Systems and Society: A General Theory of Life, Mind, and
Culture, Durham: Duke University Press.
Debrun, M. (2018a), ‘The Idea of Self-Organization’, in A. Pereira Junior, R. Ribeiro Gudwin
and W. A Pickering (eds), Systems, Self-Organisation and Information: An Interdisciplinary
Perspective, 7–22, London: Routledge.
Debrun, M. (2018b), ‘The Idea of Self-Organization’, in A. Pereira Junior, R. Ribeiro Gudwin
and W. A Pickering (eds), Systems, Self-Organisation and Information: An Interdisciplinary
Perspective, 23–46, London: Routledge.
Fogel, D. B. (2006), Evolutionary Computation: Toward a New Philosophy of Machine
Intelligence, vol. 1, Hoboken: John Wiley & Sons.
Fetzer, J. H. (1988), ‘Signs and Minds: An Introduction to the Theory of Semiotic Systems’,
in J. Fetzer (ed.), Aspects of Artificial Intelligence, 133–61, Dordrecht: Kluwer Academic
Publishers.
Fetzer, J. H. (1997), ‘Thinking and Computing: Computers as Special Kinds of Signs’, Minds
and Machines, 7: 345–64.
Gomes, A., C. El-Hani, Gudwin R. and J. Queiroz (2007), ‘Towards the Emergence of Meaning
Processes in Computers from Peircean Semiotics’, Mind & Society-Cognitive Studies in
Economics and Social Sciences, 6: 173–87.
Hartmann, S. (1996), ‘The World as a Process: Simulations in the Natural and Social Sciences’,
in K. G. Troitzsch, R. Hegselmann and U. Mueller (eds), Modelling and Simulation in
the Social Sciences from the Philosophy of Science Point of View, 77–100, Dordrecht:
Kluwer.
Humphreys, P. (2004), Extending Ourselves: Computational Science, Empiricism, and Scientific
Method, Oxford: Oxford University Press.
Kirby, S. (1999), ‘Learning, Bottlenecks and Infinity: a Working Model of the Evolution of
Syntactic Communication’, in K. Dautenhahn and C. Nehaniv (eds), Proceedings of the
AISB’99 Symposium on Imitation in Animals and Artifacts, 55–63, Cambridge: Cambridge
University Press.
Jung, D. and A. Zelinsky (2000), ‘Grounded Symbolic Communication between Heterogeneous
Cooperating Robots’, Autonomous Robots Journal, 8 (3): 269–92.
Loula, Â. and J. Queiroz (2011), ‘Modeling the Emergence and Evolutionary History
of Semiotic Systems and Processes’, International Journal of Signs and Semiotic Systems,
1: i–iv.
MacLennan, B. J. (2002), ‘Synthetic Ethology: A New Tool for Investigating Animal
Cognition’, in Magnani and N. J. Nersessian (eds), Model-based Reasoning. Science,
Technology, Values, New York: Kluwer Academic and Plenum Publishers.
Mayr, E. (1992), ‘The Idea of Teleology’, Journal of the History of Ideas, 53 (1): 117–35.
Mikulecky, D. C. (2001), ‘The Emergence of Complexity: Science Coming of Age or Science
Growing Old?’ Computers & Chemistry, 25 (4): 341–8.
Murphey, M. G. (1993), The Development of Peirce’s Philosophy, Indianapolis: Hackett.
Nadin, M. (1999), ‘Anticipation – A Spooky Computation’, Third International Conference on
Computing Anticipatory Systems (CASYS 99), Haute Etudes Commerciales, Liege, Belgium,
9–14 August.
Nadin, M. (2003), Anticipation – The End Is Where We Start from (English-German-French
text), Baden: Lars Müller Publishers.
­Nolfi, S. (2002), ‘Power and Limits of Reactive Agents’, Neurocomputing, 49: 119–45.
Parisi, D. (2001), Simulazioni – Larealta`rifatta nel Computer, Bologna: Il Mulino.
Parker, K. (1998), The Continuity of Peirce’s Thought, Nashville: Vanderbilt University Press.
100 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Peirce, C. S. ([1857–1914] 1787–1951), The Charles S. Peirce Papers Harvard University,

Houghton Library, MS Am 1632. Individual papers are referenced by manuscript number, in
R. Robin (ed.), Annotated Catalogue of the Papers of Charles S. Peirce, Amherst: University of
Massachusetts Press, 1967, and in Robin, ‘The Peirce Papers: A Supplementary Catalogue’,
Transactions of the Charles S. Peirce Society 7, 1971: 37–57. Cited as MS.
Peirce, C. S. ([1866–1913] 1931–58) The Collected Papers of Charles Sanders Peirce, vols. 1–6,
C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University
Press. Cited as CP.
Perfors, A. (2002), ‘Simulated Evolution of Language: A Review of the Field’, Journal of
Artificial Societies and Social Simulation, 5 (2): 1–46.
Queiroz, J. and S. Ribeiro (2002), ‘The Biological Substrate of Icons, Indexes, and Symbols in
Animal Communication’, in M. Shapiro (ed.), The Peirce Seminar Papers – The State of the
Art, vol. 5, 69–78, Oxford: Berghan Books.
Queiroz, J. and C. El-Hani (2006), ‘Semiosis as an Emergent Process’, Transaction of C. S.
Peirce Society, 42 (1): 78–116.
Queiroz, J. and F. Merrell (2006), ‘Semiosis and Pragmatism: Toward a Dynamic Concept of
Meaning’, Sign System Studies, 34 (1): 37–66.
Queiroz, J. and F. Merrell (2009), ‘On Peirce’s Pragmatic Notion of Semiosis – A Contribution
for the Design of Meaning Machines’, Minds & Machines, 19: 129–43.
Queiroz, J. and F. Merrell (2008), ‘Some Remarks on Bickard and Process Metaphysics’,
Cybernetics and Human Knowing: A Journal of Second Order Cybernetics, Autopoiesis &
Semiotics, 15: 64–70.
Ransdell, J. (1977), ‘Some Leading Ideas of Peirce’s Semiotic’, Semiotica, 19 (3–4): 157–78.
Rescher, N. (1996), Process Metaphysics: An Introduction to Process Philosophy, New York:
State University of New York Press.
Rosen, R. (1985), Anticipatory Systems: Philosophical, Mathematical, and Methodological
Foundations, (vol. 1 of IFSR International Series on Systems Science and Engineering),
Amsterdam: Elsevier Science & Technology Books.
Rosen, R. (1991), Life Itself: A Comprehensive Inquiry into the Nature, Origin and Fabrication
of Life, New York: Columbia University Press.
Rosenblueth, A., N. Wiener and J. Bigelow (1943), ‘Behaviour, Purpose and Teleology’,
Philosophy of Science, 10 (1): 18–24.
Salthe, S. N. (1985), Evolving Hierarchical Systems: Their Structure and Representation, New
York: Columbia University Press.
Steels, L. (1997), ‘The Synthetic Modeling of Language Origins’, Evolution of Communication,
1 (1): 1–34.
Steels, L. (1999), ‘The Talking Heads Experiment: Vol I. Words and Meanings, Pre-Edition’,
VUB Artificial Intelligence Laboratory, LABORATORIUM, Antwerpen.
Steels, L. and F. Kaplan (1999), ‘Situated Grounded Word Semantics’, in T. Dean (ed.),
IJCAI’99 Proceedings of the 16th International Joint Conference on Artificial Intelligence,
862–7, San Francisco: Morgan Kaufmann Publishers.
Steels, L., F. Kaplan, A. Mcintyre and J. Van Looveren (2002), ‘Crucial Factors in the Origins
of Word-Meaning’, in A. Wray (ed.), The Transition to Language, 252–71, Oxford: Oxford
Press.
­Van Gelder, T. J. and R. Port (1995), ‘It’s about Time: An Overview of the Dynamical
Approach to Cognition’, in R. Port and T. van Gelder (eds), Mind as Motion: Explorations
in the Dynamics of Cognition, 1–43, Cambridge: MIT Press.
SYSTEMS THEORY AND SEMIOTICS 101

Von Bertalanffy, L. (1950), ‘An Outline of General System Theory’, British Journal for the
Philosophy of Science, 1 (2): 134–65.
Von Bertalanffy, L. (1972), ‘The History and Status of General Systems Theory’, Academy of
Management Journal, 15 (4): 407–26.
Veeke, H. P., J. A. Ottjes and G. Lodewijks (2008), The Delft Systems Approach: Analysis and
Design of Industrial Systems, Berlin: Springer Science & Business Media.
Vogt, P. (2007), ‘Language Evolution and Robotics: Issues on Symbol Grounding and Language
Acquisition’, in A. Loula, R. Gudwin and J. Queiroz (eds), Artificial Cognition Systems,
176–209, Hershey: IGI Publishing.
Vogt, P. and H. Coumans (2003), ‘Investigating Social Interaction Strategies for Bootstrapping
Lexicon Development’, Journal of Artificial Societies and Social Simulation, 6 (1): 1–25.
Wagner, K., J. A. Reggia, J. Uriagereka and G. S. Wilkinson (2003), ‘Progress in the Simulation
of Emergent Communication and Language’, Adaptive Behaviour, 11 (1): 37–69.
102
 ­C HAPTER FIVE

Phenomenology
and Semiotics
PEER F. BUNDGAARD

PREAMBLE
If one considers two of the historical beacons of semiotics on either side of
the Atlantic – Algirdas J. Greimas and his Paris School of Semiotics in Europe and Husserl’s
almost contemporary Charles Sanders Peirce’s work in America – it would be tempting
to say that phenomenology has only occasionally, perhaps even incidentally, influenced
the development of a theory of signs and a theory of meaning. However, if one broadens
the perspective, it is clear that aspects of Husserl’s phenomenology in particular and the
phenomenological tradition in general have decisively, albeit indirectly, both shaped the
epistemological foundation of what could be called classical continental (Greimasian)
semiotics (Greimas 1966, 1970, 1982) and prepared the ground for a non-axiomatic,
non-formalist semiotic theory of human making-making – all this while at the same time
addressing issues and proposing hypotheses that are cognate with some of those Peirce
elaborated in parallel with and independently of Husserl.
The crucial import of Husserl’s third and fourth Logical Investigations ([1900–1] 1984)1
on the development of structural linguistics has long since been laid down by Roman
Jakobson (1990), and subsequently by Elmar Holenstein (1976, 1992; see also Bundgaard
2003). Its constitution of the notion of ‘structure’ in terms of dependency relations is
indeed at the core of this important branch of linguistics developing from the 1920s and
on, but it is also found verbatim in Louis Hjelmslev’s Prolegomena (1943) – yet without
any mention of Husserl2 – which itself constitutes the epistemological and methodological
framework that Greimas intended to transpose from the domain of language proper to
the domain of meaning. Similarly, but on the other side of the Atlantic, Husserl’s idea that
there exists a universal, formal and ideal scaffolding of language as such, underpinning
and constraining any empirical language formation, and thus from which any empirical
language (use) can be generated is obviously a predecessor (most likely mediated by
Jakobson) of Chomsky’s generative grammar (the generative hypothesis is by the way
also at the core of Greimas’s semiotics).
Now, if we look at more recent developments of semiotics, Husserl has played
a pivotal role, along with Kant’s transcendentalism, in Petitot’s morphodynamic
refoundation of and subsequent breakaway from Greimas’s axiomatic semiotics (Petitot
1982, 1992); Petitot was himself, along with Per Aage Brandt (1992), the first to
introduce Cognitive Linguistics (Lakoff and Johnson 1980, 1999; Langacker 1987–91;
104 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Talmy 2000) in continental semiotics (in the mid-80s), many of whose key theoretical
assumptions have roots directly back to Husserl3: e.g. the embodiment of cognition,
the import of intersubjectivity on cognition and human meaning-making, the idea that
language expresses a ‘construal’ of a referent scene or a situation, i.e. the situation such
as it has been experienced or intended, and that language possesses a design structure
capable of specifying such construals.4 Husserlian phenomenology and what we could
call Merleau-Ponty’s post-phenomenology are also recurrent references in the work of
cognitive semioticians such as Jordan Zlatev (2018) and Göran Sonesson (2015). Finally,
Frederik Stjernfelt has in a series of pathbreaking works, notably Diagrammatology
(2007), established the profound, and before his book rather neglected, affinities between
Husserl and Charles Sanders Peirce.
In this chapter, I shall not unfold the history of how (Husserlian) phenomenology has
influenced semiotics writ large, since his import can be grasped directly at the source, that
is in the works mentioned above (see also Bundgaard 2010). Rather, I would introduce
to and elaborate on an aspect of Husserl’s work which has been less discussed by semiotic
scholars, at least recently, even though it is of core relevance for this field; I am thinking of
his theory of signs: a work he initiated in 1890 in a manuscript entitled ‘On the Logic of
Signs (Semiotics)’ (Husserl [1890] 1970), that is to say ten years before the publication
of the LI in 1900–1: it is indeed an attempt to determine the concept of sign as well
as to minutely classify different sign types by means of carefully drawn distinctions, and
it constitutes the foundation of the 1st Logical Investigation, ‘Expression and meaning’,
itself devoted to:
1. The definition of the concept of sign
2. The essential distinction of different sign types
3. The determination of the essential properties of genuine (symbolic) signs
4. The constitution of the relation between mind and signs and
5. The relation between sign, meaning and the intended objective states of affairs.
As Thomas Byrne (2018) has observed, Husserl returned to his theory of signs and
meaning in the manuscripts preceding the planned revisions of the sixth LI in 1913–14
(Husserl [1893–1921] 2005). In the following, I shall address Husserl’s theory of signs
and meaning, according to the abovementioned dimensions of it, both as they are
developed in the first LI and according to the amendments he makes in the manuscripts
from 1913 to 1914. I shall do so, however, with a specific emphasis on the cognitive acts
he integrates into the essence of genuine signs in their communicative use. This implies
some walking on the knife’s edge, as it were, between the (formal) logic of signs and the
(pragmatic) psychology of sign use. Husserl was a methodological anti-psychologist (just
like Peirce), and the LI were intended both to lay the ground for such a philosophical anti-
psychologism and to determine essential properties of the human mind: consciousness,
intentionality, meaning (as experienced directly in perception or imagination, or indirectly
via linguistic and other sign types) within such an anti-psychological framework. This
implies that the object of study – e.g. ‘sign’ or ‘meaning’ – will be defined in its objective
essence independently of the vagaries of the individual consciousness, its psychology, its
particular history, its concrete situation of use and so on. So, for example, the content
of a given sign, the meaning it signifies and is intended to signify, is to Husserl an ideal,
objective essence which by no means can be reduced to its situation of use. Likewise, a
sign – as we shall see – has formal properties that are independent of any individuals’
PHENOMENOLOGY AND SEMIOTICS 105

use of them. Now, on the other side, Husserl is, of course, perfectly aware of the fact
that signs are indeed used in given contexts, involving sign users and specific mental acts
engaged in meaning production and meaning processing. This, then, implies that his
descriptions also reach outside the realm of the purely objective properties and essences
and into the psychology of sign use with a view to characterizing important aspects of
this psychology.
My approach in this chapter is double. I shall first and foremost try to lay down the
components of Husserl’s semiotics, both the essential, logical ones and the pragmatic,
psychological ones. But I shall also, while doing this, try to illustrate how Husserl’s theory
of signs (its questions and his answers) still informs linguistic and semiotic research in
signs, sign-processing and meaning; as well as how recent research in turn can be used
to illustrate and in a sense further strengthen some of Husserl’s claims; even if these
cognitive-pragmatic extensions of Husserl’s theory come with a rather considerable prize:
a much more psychological, semiotic theory than Husserl himself would have endorsed,
but perhaps also a descriptively more adequate or powerful theory.

WHAT IS A SIGN? INDICATIONS AND EXPRESSIONS

Husserl operates with a somewhat broad and flatfooted general notion of signs, which
simply reads that a sign (outside its use in solitary life or private thinking) has a physical
appearance given to the senses that is associated with something else, which is not
necessarily sensed. However, ‘not every sign has meaning’ (Husserl 1900–1: 183); that is
to say, not every sign expresses a meaning. ‘Indications’, which are to be understood as
Peircean indices, are such signs that elicit the representation of something else, without
expressing any meaning. So, smoke indicates the simultaneous reality of fire, but it is not
a sign that expresses any meaning beside that indication; likewise, a footstep indicates
the past presence of a human being, and fossil vertebrae are signs of prediluvian animals
(Husserl 1900–1: 184). The relation between the indication and what it is a sign for is
considered by Husserl to be a motivational unity in that the existence of ‘certain objects
or states of affairs of whose reality someone has actual knowledge indicate to him the
reality of certain other objects or states of affairs, in the sense that his belief in the reality
of them is experienced [. . .] as motivating a belief or surmise in the reality of the other’
(Husserl 1900–1: 184; his italics).
In short, the experience of indications is essentially one of a ‘descriptive unity’ of the
sign proper (the phenomenon given to the senses) and the object that it is a sign for, and
which must exist or have existed since the former exists.

Expressions and meaning-giving acts

Now, how about signs used in communication, that is to say here in the LI, linguistic
signs, which Husserl calls ‘expressions’? Just as was the case for indications, expressions
are also given to the senses, but contrary to indications, they carry meaning, and they
carry meaning not in and of themselves, or simply by virtue of associations established in
the past, but because the addresser in his use of them bestows an intended meaning upon
them. Expressions are signs to the extent that someone produces them with the intention
of expressing himself about something to someone. In other words, expressions are sound
formations (Wortlaute) that have been animated by certain sense-giving acts, without
which they would remain simple sound gestalts.5 We shall return to the sense-giving
106 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

acts in due course, but let us first observe that Husserl integrates the cognitive meaning-
giving acts into the essence of the word sign, which has a couple of consequences that are
worthwhile stressing. One is that word signs do not simply mean what their definition in
the dictionary reads, but what I, within certain limits, make them mean in a given context,
relative to a given state of affairs. Another consequence is that if, for example, we take
a walk at the beach, and the vagaries of waves, stones and sand produce marks on the
ground which perfectly resemble words aligned in a well-formed sentence, these marks
would be just that: marks that resemble words, but which are not expressions, since they
have not been endowed with meaning by someone who had the intention of expressing
himself.

Intimation
From this follows another consequence or another essential property of word signs, which
logically precedes the meaning-giving acts. This property is that expressions manifest a
meaning intention, i.e. an intention to communicate certain mental states or thoughts,
Thus, expressions are also indications, namely of the fact that speaker has thoughts,
means to say something to someone: they show forth the existence of such mental states
or meaning intentions (just like smoke shows forth the existence of fire). This indicating
function of the expression is dubbed ‘intimation’ (Kundgabe) by Husserl, and is the
function that sets off the whole sign-processing dynamics, that is to say, that which makes
physical marks or sound formations appear as genuine signs. It is of course because the
otherwise well-formed sentences produced by a parrot do not display any intention to
intimate mental states that they do not pass the semiotic threshold.
As regards the logic of signs, expressions are (conventionally constituted) sound
patterns (1) which intimate the hearer to believe that the speaker has an intention to
express something (they are indications of a communicative intention); (2) which have
been endowed with meaning by the speakers meaning-conferring acts; it is by virtue of
such acts that they mean something, and it is by virtue of what they mean that they relate
‘to what is objective’ (Husserl:1900–1: 192). From this strictly logical perspective there
is nothing more to it.

Meaning-fulfilling acts
Now, expressions appear in speech or discourse, and discourse and speech are by and large
communicative: we use signs to express something about something to someone. Thus,
even though, logically speaking, the essence of the sign is exhaustively established by the
intimating function and the meaning-giving acts, we still need to characterize signs as they
are used in social life, we need to shed light on their communicative use. To this end,
Husserl introduces another essential distinction which leads him to the essential notion
of the meaning-fulfilling acts. The distinction is the one between full and empty meaning
intentions. So, a meaning intention confers meaning to a word which, in accordance with
that meaning, relates to something objective, the object, state of affairs or event that is
being spoken about. In some cases, the object, which the sign is related to, is given in
front of us, directly in perception. You can be speaking about something we both have
perceptual access to. This is, for example, the case in all cooking shows, YouTube recipes
and live comments on sports events, or when we exchange judgements about the beauty
or ugliness of a landscape, a painting or a piece of music (cf. Zwaan 2014). In such cases,
PHENOMENOLOGY AND SEMIOTICS 107

we are talking about what we are witnessing, or what we are actually doing here and
now, and Husserl would say that the relation to the object established by the sign (and
the meaning-conferring acts) is given in ‘intuitive fulness’ (Husserl 1900–1: 192). The
meaning intention is said to be full or fulfilled.6
Without being exotic, the above cases are nevertheless rather rare or not standard
cases of communication. Typically, we use word signs, and speech, to talk about what is
not there, which is of course one of the major advantages of language: it enables us to
represent the absent. In such situations, the meaning intentions are said to be ‘empty’, and
the relation to the object is not immediately realized. Meaning is all there is, as Husserl
perhaps a bit enigmatically puts it:

A name, e.g., names its object whatever the circumstances, in so far as it means that
object. But if the object is not intuitively before one, and so not before one as a named
or meant object, mere meaning is all there is to it. If the originally empty meaning-
intention is now fulfilled, the relation to an object is realized, the naming becomes an
actual conscious relation between name and object named.
(Husserl 1900–1: 192)

What happens, then, in a communicative situation where there isn’t any perception to
fulfil the meaning and realize the relation to the object? Here the addressee must produce
acts that fulfil the meaning of the sign. Language-processing consists in performing
meaning-fulfilling acts. These acts, Husserl stresses this time and again, are not essential
to the expression proper (only intimation and sense-conferring acts or meaning intentions
are), but they are essential to the communicative situation as such, because they are the
ones by virtue of which the hearer actualizes the expression’s relation to the object.
But how are these acts to be described? One should tread carefully here because, even
though I will indeed assign particular importance to the considerable subset of them
that could be dubbed quasi-perceptual or quasi-experiential acts, they are by necessity of
different sorts. If I say ‘the three perpendiculars of a triangle intersect in one point’ or
‘the square root of sixteen is four’, the meaning fulfilling acts that relate the propositions
to their object cannot be characterized as being quasi-perceptual, i.e. acts that establish
this relation by means of imagery or intuitive contents. The same may hold true for
propositions involving abstract concepts, such as ‘Freedom of speech is one of the pillars
of the democratic society’. Yet, I will nevertheless propose that an important subset of
meaning-fulfilling acts could be aptly considered as quasi-perceptual acts or acts involving
intuitive or experiential contents, which would entail that to process signs is to ‘simulate’
or cognitively re-enact the experience of the situation the expression is about.

Meaning-fulfilling acts as quasi-perceptual simulations

In the cognitive psychology of text-processing, Rolf Zwaan has for years developed
a theory that considers what he calls ‘simulation’ (Zwaan 2004, 2016) as the pivotal
element of language comprehension. I here simply propose that an important subset of
meaning-fulfilling acts could be adequately described as ‘simulations’. What is, then, a
simulation? It is a representation that is a quasi-experience of the situation (writ large)
described or expressed by the linguistic input. Meaning-fulfilling acts thus realize the
relation of the sign(s) to the object by simulating the experience of the object (or the way
it has been intended) such as it is encoded in the sign(s). Here is how Zwaan succinctly
108 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

puts it: ‘The basic premise is that language is a set of cues to the comprehender to
construct an experiential (perception plus action) simulation of the situation’ (2004: 36).
In other words, the comprehension of an expression (which in Peircean terms could be
the interpretant of the sign) is the ‘reconstitution’ of the experience expressed by the sign
(Zwaan 2004: 38).
Zwaan, and others (notably Barsalou [1999]), have provided abundant evidence for
the quasi-perceptual or quasi-experiential nature of language comprehension, that is of
the meaning that is the endpoint of meaning-fulfilling acts and the communicative finality
of the meaning-giving acts. A classical experiment consists in having participants read
either of the following sentences (Zwaan 2004):

(1a) The ranger saw the eagle in the sky.

(1b) The ranger saw the eagle in the nest.

They are subsequently shown a picture of an eagle and asked if it was mentioned in
the sentence. Interestingly, response time is faster when there is a match between the
shape of the eagle in the picture and the one implied by the sentence; response time is
thus faster when people after having read (1a) see a picture of an eagle with its wings
outstretched. In other words, the phenomenal shape of items referred to by a complex
of signs is routinely integrated in the comprehension of those signs. This is part of what I
mean when I say that an important subset of meaning-fulfilling acts can aptly be described
as quasi-experiential simulations.7
Other kinds of examples, here adapted from Zwaan (2004), show the quasi-perceptual
nature of linguistic contents:

(2a) The red squirrel approaches the fence.

(2b) The red tractor approaches the fence.

Here, obviously the understanding of the sentence, which relates word expressions and
their meanings to a given objective scene, implies rather different construals of both word
meanings and the referred relations between objects. As Zwaan remarks, ‘red’ will not be
represented in the same way in both sentences, nor will ‘approach’, since in the squirrel
scene ‘approach’ requires much more proximity than in the ‘tractor’ scene; and finally,
the implied perspectives in each scene are essentially different (proximal vs. distal). These
interpretations, construals or simulated quasi-perceptual experiences are elicited by the
meaning-giving acts.
Now, consider a somewhat different case since it does not imply a straightforward
perceptual simulation of a scene (for the simple reason that the information does not
concern the perceptual properties of the scene), which, of course, implies that meaning-
fulfilling acts and simulations can be more than proxies for genuine perceptual meaning-
fulfilling acts. Consider someone informing someone else of the state of affairs in
example (3):
(3) This computer is not child safe.
For reasons pertaining to the fact that language essentially underdetermines its
referent, both syntactically and semantically, the meaning of the sentence cannot be
computed following compositional principles. The sentence can mean one thing and its
opposite, namely that it is not safe for children to use the computer, since I haven’t
PHENOMENOLOGY AND SEMIOTICS 109

installed any parental control software on it; or that it may not resist children’s playing
with it. The experienced meaning of the sentence, which is of course dependent on
contextual information, will be the outcome of the meaning-fulfilling acts. Now, these
acts are perhaps not as perceptual as the ones triggered off by examples [1a through 2b],
but they can still be said to be experiential, and not simply abstract representations, since
they elicit different simulations of the things you can and cannot do with the computer,
for example its fragility vis-à-vis children’s playing with it.
Here’s then the complete model for the word sign and its use in communication:
1. An expression is shaped so as to intimate to the hearer that speaker has a
communicative intention; an expression indicates a meaning intention.
2. Meaning is conferred to the sign in meaning-giving acts.
3. In cases where the meaning intention is not immediately fulfilled by perception,
meaning-fulfilling acts realize the sign’s relation to the object, according to its
meaning.
As we can see, this semiotic setup by far exceeds a simple signifier/signified relation of
a Saussurean sort. Moreover, even though the logical basis of the word sign is limited to
its intimating function and the meaning-giving acts, it is impossible in practise to avoid
the essential interference from the situation of sign use and the crucial role played by
the meaning-fulfilling acts elicited by the word sign such as it has been used. This has a
consequence which Husserl acutely observes and then attempts to restrict the validity
of. The consequence is that the meaning of a sign is not ‘simply’ the meaning bestowed
upon it by a sign user, but rather the meaning-fulfilling acts that realize the meaning
intention’s relation to the object; that is, those acts that represent (or simulate) the way
the sign intends a given object. In other words, when meaning is conferred to signs, then
the ultimate intention of this act is to elicit meaning-fulfilling acts that correspond to the
original meaning intention. The meaning of a sign is, in a sense, the meaning-fulfilling act
and its content. Here’s how Husserl puts it:

The word ‘expression’ is normally understood [. . .] as the sense-animated expression.

One should not, therefore, properly say (as one often does) that an expression
expresses its meaning (its intention). One might more properly adopt the alternative
way of speaking according to which the fulfilling act appears as the act expressed by
the complete expression: we may, e.g., say that a statement ‘gives expression’ to an act
of perceiving or imagining.
(Husserl 1900–1: 192)

As I have already mentioned, there is a sort of double approach to word signs in the
LI. On the one hand, Husserl attempts to lay down the essential properties of such signs,
their logical foundation; on the other hand, he intends to describe signs in communicative
use, which makes him recruit acts, namely the meaning-fulling acts, that are essential to
the communicative situation, but not essential properties of the sign as such. In short, we
seem to have both a logic of signs and a pragmatics of sign use. This distinction between
elements that are essential to the sign and elements that are essential to its use, but not to
what it really is, may seem artificial, and as we shall see Husserl himself may have come to
that conclusion at a later point in time. Suffice here to say that he maintains the restricted
definition and the difference in the LI because he wants a definition of the sign that
110 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

seamlessly covers both its dialogical (communicative) use and its monological use (what
he calls ‘expressions in solitary life’), and here, obviously, the meaning-fulfilling acts are
not an essential element of the way the sign intends an object through its meaning.
The question is now whether this distinction can be upheld, and whether Husserl
himself by preparing the ground for a pragmatic semiotics has not, in a sense, undermined
the restricted definition of the sign, not in the sense that it cannot be logically maintained,
but in the sense that this restricted definition of the sign is so compressed that it cannot
be used to adequately describe the communicative use of signs. Let’s consider the case of
the communicative use of signs again, which complexifies things considerably: so we have
one speaker who intends to use signs to express something (a meaning) about an object
(writ large) to a receiver, we therefore have the speaker’s meaning-giving acts, we have
the situation of communication, we have an object related to and a manner of intending
the object expressed by the signs, and we have the receivers meaning-fulfilling acts that
are representations of the way in which speaker intends the object in case. Now, and this
is the crux of the question, if the speaker’s meaning intention is to elicit in the receiver,
by means of a combination of signs, a representation of the manner in which he relates to
or intends a given object, then, of course, it seems strange, if not infelicitous, to relegate
the meaning-fulfilling acts – which, again, realize that relation – to something extraneous
to the sign proper, something beyond or somehow secondary to the essence of the sign.

THE REDEFINITION OF SIGNS IN THE

MANUSCRIPTS FROM 1913 TO 1914
Husserl himself may have reached that conclusion because in his, posthumously
published, revisions to the second edition of the Logical Investigations, he proposed
a series of interesting modifications of his original theory of signs, which assigns
particular importance to the communicative use of signs proper. In these manuscripts,
Husserl proceeds by making a more fine-grained division of signs into inauthentic signs
(indications) and authentic signs, which again are divided into authentic categorial signs
and authentic signal signs; only the latter two are expressive in the sense that they carry
or express an intention to communicate something to someone, but only the authentic
categorial signs are linguistic. We shall now see how Husserl, by virtue of this crucial
distinction between inauthentic and authentic signs, so to speak puts the communicative
function of the sign back into its essence.
As we have already seen, indications elicit the representation of, and the belief in,
the existence of another object; they do so compellingly by virtue of a past association
that motivates the link between the two objects. So, dark clouds elicit the representation
of (possible) rain, smoke elicits the representation of (and thus signifies) fire and so on.
However, the motivated association between sign (indication) and signified object may
wane, as it were; that is to say, the compelling link between the two objects may come
in degrees of compulsion; and the essential reason for this can be expressed in terms of
Husserl’s semiotics. Indications or indicative signs do not have any intimating function;
therefore they do not express any meaning intention or communicative intention;
and therefore the degree to which indications (smoke, footsteps) compellingly elicit a
representation of their signified object (fire, human beings) varies in function of their
relevance in the situation where they are perceived. Footsteps may strongly indicate
human presence for Robinson Crusoe for reasons pertaining to his situation; not so or
PHENOMENOLOGY AND SEMIOTICS 111

hardly so for me if I am taking a stroll on the beach; smoke compellingly signifies fire
or combustion of some sort if it is in my kitchen; less so or hardly so if I see it above a
cityscape, perhaps not at all if I am more interested in knowing the force of the wind
or its direction.8 The situation of use is inarguably essential for the comprehension of
any sign, including, of course, linguistic signs (Bundgaard 2019) the meaning of which
is tributary to the context. However, the relevance of the situation where an indication
appears is even more crucial in that it depends on its situational relevance whether or
not the indication will even start signifying, that is, whether it will, and to what degree,
elicit the representation of its associated object. This is not the case for linguistic
signs which compulsively will elicit a representation of their default meaning and its
relation to an object, even in the absence of a situation of use: in such – probably purely
fictive – situations, the meaning-fulfilling acts will construct a default situation where the
word signs will take on their default meaning.

Genuine signs come with the demand that they should be acted upon
Here is then the new distinction Husserl proposes in his revisions to the sixth (and also
the first) LI: the one between authentic non-categorial signs and authentic categorial signs
(the latter being linguistic signs). A non-categorial sign is any non-linguistic sign which
has a communicative purpose and thus expresses the intention to communicate something
to someone. It could be a flag raised by pirates communicating that no prisoners will be
taken, or a flag communicating that there are contaminated people on this ship which
should therefore not be approached; or it could be Husserl’s own main example, namely
a siren signalling an incoming storm from north-east. We shall shortly see what, according
to Husserl, makes such signals different from categorial signs, but before that, we shall
consider what makes signal sign authentic signs on a par with linguistic signs. That is of
course the fact that they are communicative, express a state of affair and are used for
this very purpose (for example by the harbour authorities). Yet, there is more to it since
Husserl now introduces a property of genuine signs which he didn’t mention in the first
LI and which appears as an essential refinement of his notion of ‘intimation’ (see also
Byrne 2018 for this). Any genuine signs, whether storm signals, road signs or linguistic
signs, are accompanied with or manifest the addresser’s demand (Zumutung) that the
addressee should (sollen) pay attention to what is being communicated, that is to say, he
‘should know this’. So, any genuine sign is bestowed not only with a meaning, but also
with what scholars in a continental tradition of semiotics9 have called an ‘enunciative’
demand and a should: ‘Alle echten Zeichen haben den Ursprung aus solchem zumutenden,
von Subjekten ausgehenden Sollen’ (Husserl [1893–1921] 2005: 97); ‘All genuine signs
originate in this demand, in this Should that emanates from the subject.’
Husserl himself doesn’t develop this ‘Sollen’ in much detail, but we can try here to
derive some consequences for a theory of signs that may exceed the limits of what Husserl
actually proposes here, but which is nevertheless still in accordance with the main tenets
in it. The first consequence is that genuine signs are genuinely communicative, that is
to say, if the communicative sign comes with such a demand, the addressee and, along
with him, the meaning-fulfilling acts are essential moments of the sign in use, they are
the instance towards which the sign tends. This is an important extension, and also an
improvement, of the restricted definition from the first LI.
The next consequence is that this extension implies a pragmatic definition of the sign
that, of course, goes beyond the relation between sign and signified. The sign user not
112 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

only expresses himself in such and such a way, for such and such reason, but also demands
of the receiver that he act upon what is being communicated to him; in Husserl’s more
restricted understanding of this ‘Sollen’, it could simply mean ‘you should pay attention
to this’, ‘it is important that you know this’ and so on. Now, the pragmatic character of
this new definition of the sign becomes evident if we consider its dual form, as it were,
that is, if we inverse the perspective. If it is the case that what the receiver is being told
is something that by definition comes with the demand ‘you should know this’, then the
receiver must consequently also expect that what he is being told is relevant in some
respect. The pragmatic notion of relevance is therefore built into the key notions of
‘demand’ and ‘Should’ (Zumutung and Sollen).
I suggested above that the notion of ‘should’ implies that the sign comes with the
demand that the receiver act upon what is being said (and that indexical signs do not
come with that demand). It should be mentioned that Husserl may stop short from
fully developing the sense ‘should’ and the full, pragmatic, meaning of ‘acting upon’ a
communicative sign. Let’s return to Husserl’s own example: a storm siren is signalling an
incoming storm from north-east; by hearing it, I know I should pay attention to it, and
paying attention to it, I understand what it intends to express. An analogous situation
would be that I am in a theatre and a man yells ‘Fire!’, which elicits the representation that
the theatre is on fire. Could we in either of these situations exhaustively and adequately
describe the semiotic situation, and thus the sign, in terms of it eliciting a representation
of an incoming storm from north-east or that the theatre is on fire? Hardly so, since the
meaning of the storm siren is crucially not simply that there is a storm coming in, but
to make you act accordingly, that is to say: you should stay in the harbour. In Peircean
terms, we could say that the final interpretant of the storm signal is the adequate way
of acting on it; the conclusion ‘so I should stay in the harbour’ is not an inference that
is drawn beyond the realm of the sign, something extraneous to its formal essence, it is
indeed the genuine meaning of that sign. This, trivially, explains why it is prohibited to
yell ‘Fire!’ in a theatre if there isn’t any fire in it. In short, genuine signs come with an
extended demand, which is part of their intended meaning, namely that a corresponding
act should be executed (physically, intellectually or perceptually).

The difference between categorial (linguistic signs) and signal signs

The above explains why signals, contrary to indicative signs, are genuine signs: they
express a meaning relative to a state of affair; they come with an intersubjective demand
and they call for an appropriate action. Now, what makes them different from categorial
signs, that is to say, linguistic signs? The difference rests on the fact that ‘in the realm
of signals, there doesn’t exist any grammar’ (Husserl [1893–1921] 2005: 53). This may
at first glance seem a bit astonishing since they do seem to display some sort of minimal
compositionality in that we can have articulated signals like [A storm is coming] + [from
north-east], and we can easily transpose that meaning into words. Now, according to
Husserl, what signals do is that they simply indicate the state of affairs consisting of
a storm and where it comes from; contrary to linguistic signs they do not articulate
meanings and their categorial forms so as to constitute them as specifically intended
states of affairs. They point at a fact [Storm coming] + [from north-east] and leave it to
us to combine these elements and shape them into given categorial expressions: as in,
e.g., ‘There is a storm; the storm is from north-east’; or ‘There is a north-east storm’; or
‘A north-east storm is expected’, and so on (Husserl [1893–1921] 2005: 53). While the
PHENOMENOLOGY AND SEMIOTICS 113

signal is indifferent to the various ways these significations can be combined, Husserl
insists that these sentences are different as regards their significations: i.e. the object
is intended through differently composed meaning intentions; or to put it in cognitive
linguistic terms: such expressions articulate different construals of the state of affairs.
Let’s take a closer look at the differences between signals and signs in order, once again,
to develop an understanding of the functioning of linguistic signs that, despite other kinds
of essential differences, is close to the one developed within cognitive linguistics. Signals
are univocal, clear and distinct. However, if you consider a sentence like ‘The lion is over
there’, its meaning is by no means unambiguously anchored in the sign, but rather10 in the
addresser’s meaning intentions, it is relative to what he intends to express when he uses
the sign in such and such situation. So, trivially, the above sentence could mean that the
lion cage in the zoo is over there, that the toy lion is under the sofa over there, that the
portrait of the lion is over there, that the person dubbed Lion in my family is sitting over
there, and so on and so forth.
Another difference, which is already implied in what Husserl said about the
indication (the signalling) of a state of affairs and its linguistic expression, is that
signals do not have any or imply any viewpoint (i.e. they don’t express any construals).
When you become conscious about a signal, the latter leads you into consciousness
about the object it points at. So, you hear SOS, and you then, subsequently, pass to
the consciousness of an object, namely a ship in distress. There is nothing between the
sign and the object it univocally makes you conscious about. This is not the case for
linguistic signs which express the way in which an objective state of affairs has been
intended and therefore bring the referent object to a certain manner of categorial
manifestation, what Husserl calls the categorial ‘Erscheinungsweise’ of the referent
scene (Husserl [1893–1921] 2005: 125). Simple examples are sentences such as: (1)
A is parallel to B; (2) B is parallel to A; (3) A and B are parallel. The object referred
to by the sentences is of course the same, but the way it is categorially shaped, in
terms of perspective and fore- and backgrounding, is not. Husserl puts it like
this: Linguistic signs contrary to signals are ‘articulated expressions of a cognizing
consciousness [Erkentnissbewusstsein]’, i.e. they express the way in which such a
cognizing consciousness has comprehended the state of affairs it is or was intending.
In this sense, and even though Husserl’s technical language is far from psychological,
his conception of what linguistic signs express, or what is expressed through language,
seems to be a forerunner of what has been marshalled in Cognitive Linguistics in
the past forty years: language expresses construals of referent scenes and through
the expression of such construals the speaker intends to elicit representations in the
addressee that match such construals, that is to say, as we saw it above: speaker wants
to elicit meaning-fulfilling simulations.

Linguistic signs do not designate objects, they construct meaning formations

Here’s then a final difference between signals and linguistic signs, but also a difference
with respect to Husserl’s definition of sign-processing in the first LI (Thomas Byrne [2018]
has insisted on this new definition of the sign). In the case of the signal, we saw that first
there is a phenomenal consciousness of the signal as, say, an acoustic phenomenon Z,
and then we have a passage to another type of consciousness, namely of the acoustic
phenomenon as a sign, and finally we have a consciousness of the object signified by
Z: GZ. Similarly for linguistic signs in the Logical Investigations, you would first have a
114 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

phenomenal consciousness of, say, certain marks on paper, and then you would have a
passage into another kind of consciousness when these marks are apprehended as signs
signifying something, and finally a consciousness of what is meant by the signs. Now, in the
manuscripts from 1913 to 1914, Husserl proposes a strong amendment of this structure
in that for him there is no such passage from a phenomenal consciousness to another type
of consciousness (of the signified object). Marks taken as linguistic signs are intimately
united with what they name and mean; in this respect, Husserl claims, they behave like
pictures ([1893–1921] 2005: 126). In a picture you can, indeed, analytically distinguish
between the depicting moments and the depicted moments, but only analytically: there
isn’t any first intake of the picture as an aggregate of physical marks on a canvas, and then,
subsequently, another kind of intake, i.e. another kind of consciousness, namely of the
depicted objects. Both these sets of moments coincide or, as Husserl has it, cover (decken
sich, Deckung) each other or overlap. The same is then supposed to be the case for the
representing moments of the sign and its represented moments. There is not first one type
of apprehension of the former, which then leads to another type of apprehension, namely
of the latter.
How is this coextension or coverage between representing and represented moments
in the sign to be understood? The point is the following: once linguistic signs are taken
as signs, that is to say, as expressing a meaning intention and a given way of attending
to a state of affairs, we are immediately in a realm of significations in which an object is
being constituted: the object as it is meant; that is, an object that is not identical with the
object referred to, but that object such as it is intended. In this specific sense, linguistic
signs do not ‘designate’ (bezeichnen] anything; they do not have any direct referential
(hinweisende) function proper (Husserl [1893–1921] 2005: 128), as signals have, because
they don’t mean the object as such, but the way in which it comes to appearance for a
cognizing consciousness, as we saw before. The distinction between meaning and object
of reference of course still obtains, but what the signs do is not simply to point to some
state of affairs, but to shape it, gestalt it, as it were, to a meant meaning; and what
the receiver does is similarly to continuously process the signs’ manner of articulating a
meaning formation, relative to speaker’s intention, and relative to the situation or objects
referred to, and crucially this manner of processing does not require a specific kind of
consciousness – one uniquely sign-oriented – which then leads us into an entirely different
kind of consciousness, one which is object-oriented. We acknowledge the meant object as
it is being articulately expressed by the signs.11
It is worthwhile remarking that essential elements of the analogy with the picture
and the corresponding picture consciousness still obtain here. As already remarked, in
general, there is no such thing as a specific consciousness of physical marks on a surface
that constitutes the gateway into another, different consciousness of the objects depicted
in the picture. Now, just as was the case for the linguistic, this does not imply that the
relation to a reference object (whether fictional or not) has been cut off. The fact is simply
that I don’t directly experience the very object represented in the picture, or the picture is
not a gateway to that object: I see that it is Napoleon and the Alps, but what I experience
is him triumphantly crossing the Alps such as Jacques-Louis David meant him to do it.12
It is in this sense that, in Husserl’s later developments of his theory, signs can be said
to signify rather than designate (bezeichnen), and also in this sense that sign consciousness
and meaning consciousness constitute from the outset a unitary semiotic consciousness
in which sign processing is coextensive with the grasping of an articulated, meant state
of affairs.
PHENOMENOLOGY AND SEMIOTICS 115

Having reached the, at least provisory, endpoint of this presentation of Husserl’s

theory of signs, let me sum up some of its main points, along with the specific pragmatic
and cognitive semiotics extensions I have proposed of it.

­CONCLUSION: THE COMPONENTS OF A

PHENOMENOLOGICAL SEMIOTICS
In the guise of a conclusion, let me further discuss the four main tenets of what Holenstein
(1992) already called Husserl’s phenomenological semiotics:
4. The meaning-giving acts
5. The intimating and demanding sign function
6. The meaning-fulfilling acts
7. Sign use as meaning construction.

Meaning-giving acts
The integration of meaning-conferring cognitive acts into the very definition of the
sign is a core element of Husserl’s semiotics and theory of meaning. It paves the way
for a dynamic understanding of human sign use in that meaning is not seen as simply
carried by or attached to a sign vehicle; rather signs are defined as expressing intentional
relation to a given object (broadly taken), and the meaning they express is conferred to
them by virtue of this intentional relation to the object. In this respect, Husserl is clearly
a – relatively ignored – forerunner for one of the fundamental theses in modern Cognitive
Linguistics: namely that language does not simply signify objects as such, but expresses
construals of objects, specifically structured experiences of objects. Construals of referent
scenes or states of affairs come with a perspective, which is a speaker’s way of attending
to the given object; and ‘perspective’ here simply means the way in which one intends the
relation between the elements in the referent scene (writ large) to be; or how speaker has
distributed his attention over the referent scene, foregrounding this, backgrounding that.
Now, perspective is not necessarily limited to experiential or intuitive cases, but could be
claimed to play a shaping role also in formal uses of expressions, e.g. in the articulation
of ideal objects so that the difference between [2+2=4] and [√16=4] pertains to the
manner in which these objects have been categorially intended, that is, it pertains to their
categorial manner of manifestation. It is also worthwhile noting that the essential role
assigned by Husserl to the meaning-giving acts also makes him capable of accommodating
a key property of language: namely that language by essence underdetermines its referent,
but still is an efficient tool for communication in that signs in use are sets of prompts
for the addressee to represent the meant situation and thus the meaning-giving acts that
articulate speaker’s intentional relation to that situation.

The intimating function; the demand and the Should

We have seen that another essential property of communicative signs is their intimating
function – the fact that for something to be a sign it must announce an intention to
express something about something to someone. Husserl’s own extension of this function
in his 1913–14 manuscripts stipulates that the sign does not simply announce a meaning
intention, but doing so also comes with the demand that the addressee attend to it, that
116 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

he should know this or pay attention to it. In that respect the addresser’s demand is to
be understood as a claim of relevance, which again is correlated with the addressee’s
expectation of relevance. Here again, we have a determination of a logical property of
signs, the intimating, demanding function and the ensuing claim of relevance, which
forces the semiotician to address the nature of signs outside signs themselves as it were:
namely, relative to the intersubjective dynamics in which they are used.
As explained above, Husserl may nevertheless operate with a too restricted scope of
demand, or a too restricted semiosis, which is then in need of further extension. The
point is here that signs used in communication do not simply come with the demand that
the addressee should listen to them because the information they convey is relevant; they
also, and crucially, come with the demand that the addressee act adequately on them.
This implies that in the absence of such adequate action, the sign signifies or expresses in
the void, a bit like a road signal does when it is not in use. This may seem straightforward
for cases like ‘Fire!’ or storm signals and imperatives, but what does ‘act upon’ mean
for everyday communicative exchanges, where physical action is not required? In such
cases Husserl’s minimal definition of the ‘should’ may suffice: it may be enough to take
into account what is being communicated; yet, ‘to take into account’ probably implies a
reaction of some sort, which again is to be considered the final interpretant of the sign
as it is used by the addresser. If someone tells me ‘It’s raining outside’, and I answer
‘Hmm’ without lifting my eyes from the crossword section of my paper, the other person
may consider his attempt to communicate as failed, and me as a rude person. Similarly,
if I am told that Trump won Florida, but lost Pennsylvania, the finality of that piece of
communication is not simply that information, but rather to elicit some sort of emotional
response. In short, signs used in communication call for (re)action; they are by essence
perlocutionary, as Austin (1962) had it.

The meaning-fulfilling acts

Expressions in their communicative use are prompts for the addressee to simulate the
referent scene as intended by the addresser. So, the sign-processing acts that expressions
elicit are simulations of the meaning-intending acts and their relation to the object.
Such simulations are in some situations necessarily quasi-perceptual, but come of course
in degrees of abstraction according to the nature of the meaning-giving acts. In short,
‘the mouse approaches the fence’ necessarily triggers of a quasi-perceptual simulation
(which need not be detailed), whereas ‘Freedom of speech is one of the pillars of a free
democratic society’, or ‘√16=4’ does not require such acts. It is, however, still much
debated whether even very abstract propositions or concepts (such as ‘equity’ or ‘justice’
or ‘moral’) require recruitment of intuitive contents, i.e. quasi-experiential situations.
It may very well be difficult, perhaps impossible, to explain the concept of ‘justice’
purely in abstracto, i.e. without recurring to situations likely to illustrate (aspects of)
the concept (e.g. ‘See, this could be called a just action’, or ‘This is something unjust’).
In that sense understanding a concept amounts to simulate situations where the concept
has been put into play.
A last word on the cognitive-pragmatic extension of Husserl’s notion of meaning-
giving and meaning-fulfilling acts. Cognitive Linguistics provides abundant well-founded
illustrations of what it means for an act to bestow meaning on signs in accordance with
the way an object has been intended, and of the design features of language that specify
this meaning. Similarly, the cognitive psychology of language-processing has elaborated
PHENOMENOLOGY AND SEMIOTICS 117

concise theories about and given enlightening examples and evidence of the nature of
meaning-fulfilling acts as quasi-perceptual simulations: i.e. simulations of the referent
scene such as the addresser has intended it. The ideal meaning of signs – their finality – in
a pragmatic, communicative situation of use is to elicit simulations that match sufficiently
well their constitutive meaning intentions. It should be noticed that the fact specific
meaning-fulfilling acts can be mistaken implies that the meaning bestowed upon the sign
cannot be reduced to the way in which the meaning-fulfilling acts realize the relation of
the sign to the object. It is still the case, however, that the finality of the sign in use is to
have these meanings cover each other adequately.13

Sign use as construction of meaning formations, and sign processing as the

apprehension of meaning formations
The revisions from 1913 to 1914 unfold a seemingly radical claim to the effect that
word signs do not designate (bezeichnen) an object, they do not simply refer to it (but
are of course related to it). They are rather said to unfold, by virtue of their internal
articulation, a certain predicative apprehension of a state of affairs; they are, as we
saw, the articulated expressions of a cognizing consciousness (‘Erkenntissbewusstsein’).
This means that instead of referring directly to such and such objects or situations, they
construct a meaning formation which is the object such as it has been acknowledged,
apprehended or experienced. Correlatively, sign-processing is not a two-step endeavour
where signs qua signs with given contents are first computed, after which their relation to
an object is represented; sign-processing consists in apperceiving, as it were, the meaning
formation as it is being shaped, the relation to the object as it is being articulated within a
dynamic, intersubjective communicative system. Here’s how Husserl puts it:

[The communicative function] is activated by the fact that the addresser and the
addressee, the sign giver and the sign receiver constitute themselves in a intersubjective
consciousness, and by the fact that they face each other in mutually understanding acts
of consciousness, where one transmits the sign with its signification, and where the
other understands it exactly with this signification.
(Husserl [1893–1921] 2005: 54)

The reason why we can talk about a phenomenological semiotics in this context is
not simply because its author is a phenomenologist called Husserl, but because it is a
theory of sign which from the outset defines the communicative sign system as something
essentially different from a self-contained language system. Instead, a phenomenological
semiotics founds the communicative sign system on the interaction between experiencing,
perceiving and acting intentional subjects and their objects intended and meant in such and
such a way, as well as on the intersubjective interaction of communicating consciousnesses
in given situations, relative to given objects. These elements – cognitive acts, intentional
relations to construed objects, signs and interactions between intersubjectively related
and situated consciousnesses – constitute the semiotic ecology of communicative sign use.
The semiotic legacy of Husserl’s phenomenology in general, his theory of signs
in particular, as well as of the work of disciples such as Roman Ingarden (1973) and
Merleau-Ponty (1945, 1964) extend, of course, far beyond the aspects of his theory
introduced and discussed here. Husserl has for example devoted a considerable amount
of work on and proposed acute descriptions of the nature of images or pictorial signs,
118 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

both plain and aesthetic, as well as of the type of consciousness they elicit – a work,
which, unfortunately, does not seem to have had the same kind of (indirect) impact on
the semiotic community as his analyses of the linguistic sign and the language system:
a remarkable exception is again Stjernfelt (2007). Similarly, and as already suggested,
Husserl’s decisive attempt to constitute the essence of the sign as something irreducible
to an encapsulated semiological formal structure, that is, by integrating interacting
communicative minds and the situations of sign use, may be considered an anticipation
and a still not sufficiently well-explored foundation for recent and not so recent branches
of cognitive science and semiotic and linguistic theory, stressing the enactive and the
embodied nature of cognition and semiosis (Varela et al. 1991; Lakoff and Johnson 1999;
Pelkey 2017, 2018, Mittelberg 2018), its social, interactive nature (Tylén et al. 2013).
This history of influence, as it were, is still by and large to be written (see Gallagher
1997, though). It sets off early with the texts I have briefly examined here, and it goes on
with the abovementioned analyses of the image signs (present also in the Ideen, Husserl
1952), the anti-representationalism developed in The Idea of Phenomenology (Husserl
1950b), the thorough examinations of the relations between experience, judgement and
language in Experience and Judgement (Husserl [1939] 1999) as well as in the meditations
about the constitutive function of intersubjectivity as exposed in the dense Cartesian
Meditations (Husserl 1950a). The manner in which these hypotheses, claims and insights
have sedimentated in contemporary theories of human meaning-making constitutes a rich
and promising horizon for future research.

NOTES
1 Logical Investigations will henceforth be abbreviated ‘LI’ and referred to as ‘Husserl (1900–1)’.
References are to the English translation.
2 Hjelmslev does, however, mention Husserl elsewhere (1928: 40), in rather negative terms,
when commenting on ‘the philosopher Husserl’s strange hypothesis’ (‘la thèse étrange du
philosophe Husserl’).
3 Two caveats: (1) they have roots directly back to Husserl, but have been recruited via Maurice
Merleau-Ponty; (2) their being generated from Husserlian phenomenology does of course
not mean that they are structurally identical; Husserl would most likely have considered
Cognitive Linguistics much too psychologistic.
4 There is abundant literature within Cognitive Linguistics on the relation between language,
semantic structure and ‘construals’; see, e.g. Langacker (1987/91; 2015) and Talmy (2000).
5 In the following I shall interchangeably be using ‘meaning-conferring’, ‘meaning-giving’ and
‘sense-giving’ acts for those cognitive acts where meaning is assigned to a sign in accordance
with a meaning intention.
6 The fullness does not need to be perceptual; I can, for example, in a conversation with my
brother evoke common memories of past experiences we had; in that case, the meaning
intention would also be full.
7 In the concluding section of this chapter, I shall again address the issue about expressions
which hardly can be said to elicit quasi-experiential simulations, e.g. abstract expressions or
propositions.
8 These observations are mine. Husserl doesn’t consider the existence of a graduated link
between the signifying indication and its signified.
9 Cf. Greimas (1982), Greimas and Courtés (1986): art. ‘Énonciation’.
PHENOMENOLOGY AND SEMIOTICS 119

10 Here is not the place to answer the question whether linguistic signs or concepts have core
meanings and then different sorts of extensions from that core, or whether they do not have
any such core meanings, but are vague predicates that are specified in the situation of use.
Casasanto and Lupyan (2015) have made a strong case for the latter option; I myself have
marshalled cognate ideas in Bundgaard (2019). Be it as it may, the non-univocity of linguistic
signs is hardly disputed.
11 Husserl’s argument is here of a Chinese Room sort (cf. Searle 1980). There is not first
some syntactic computation in one compartment which is then handed over to another
compartment in which the semantic outcome of the first computation is extracted. The
articulation is semantic from the outset, namely of the significant moments of the object as
they are meant and expressed linguistically.
12 Again: the analogy with picture consciousness is Husserl’s own, so it does not at all follow
that images are like linguistic signs; it’s the unitary structure of linguistic sign-processing that
is like the unitary structure of picture consciousness. I have dealt with Husserl’s theory of
depiction in Bundgaard (2022); see also Stjernfelt 2007: chapter 14).
13 For a non-psychological, realist phenomenological interpretation of meaning-fulfilling acts
as quasi-perceptual acts, cf. Roman Ingarden (1973). Here, he qualifies readers’ ‘filling-out’
of the famous ‘spots of indeterminacy’ in literary artworks, but also in texts in general, as ‘a
function analogous to perception’. See Bundgaard (2013) for a presentation of Ingarden’s
theory.

REFERENCES
Austin, J. L. (1962), How to Do Things with Words, Oxford: Oxford University Press.
Barsalou, L. (1966), ‘Perceptual Symbol Systems’, Behavioral and Brain Sciences, 22: 577–660.
Brandt, P. Aa. (1992), La charpente modale du sens, Amsterdam: John Benjamins.
Bundgaard, P. F. (2003), ‘The Ideal Scaffolding of Language: Husserl’s Fourth Logical
Investigation in the Light of Cognitive Linguistics’, Phenomenology and the Cognitive
Sciences, 3 (1): 49–80.
Bundgaard, P. F. (2010), ‘Husserl and Language’, in D. Schmicking and S. Gallagher (eds),
Handbook of Phenomenology and Cognitive Science, 368–99, Dordrecht: Springer.
Bundgaard, P. F. (2013), ‘Roman Ingarden’s Theory of Reader Experience: A Critical
Assessment’, Semiotica, 194: 171–88.
­Bundgaard, P. F. (2019), ‘The Structure of Our Concepts: A Critical Assessment of Conceptual
Metaphor Theory as a Theory of Concepts’, Cognitive Semiotics, 12 (1): 1–11.
Bundgaard, P. F. (2022), ‘Aesthetic Perception as Vision for Appearance – On Husserl’s Theory
of Depiction’, in F. Vassiliou (ed.), Phenomenology, Art, and the Aesthetic Mind. The
New Yearbook for Phenomenology and Phenomenological Philosophy, vol. XIX, London:
Routledge.
Byrne, T. (2018), ‘The Evolution of Husserl’s Semiotics: The Logical Investigations and Its
Revisions (1901–1914)’, Bulletin d’analyse phénoménologique, XIV (5): 1–23.
Casasanto, D. and G. Lupyan (2015), ‘All Concepts Are Ad Hoc Concepts’, in E. Margolis and
S. Laurence (eds), The Conceptual Mind: New Directions in the Study of Concepts, 543–66,
Cambridge: MIT Press.
Gallagher, S. (1997), ‘Mutual Enlightenment: Recent Phenomenology in Cognitive Science’,
Journal of Consciousness Studies, 4 (3): 195–214.
Greimas, A. J. (1966), Sémantique structurale, Paris: PUF.
120 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Greimas, A. J. (1970), Du sens, Paris: Seuil.

Greimas, A. J. (1982), Du sens, II, Paris: Seuil.
Greimas, A. J. and J. Courtés (1979), Semiotique, Paris: Hachette.
Greimas, A. J. and J. Courtés (1986), ‘Énonciation’, in J. Courtés and A. J. Greimas (eds),
Sémiotique: Dictionnaire raisonné de la théorie du langage, vol. 2, 75–77, Paris: Hachette.
Hjelmslev, L. (1928), Principes de grammaire générale, Copenhagen: Det Kgl. Danske
Videnskabernes Selskab, Historisk-Filologiske Meddelelser, vol. XVI (1).
Holenstein, E. (1976), Roman Jakobson’s Approach to Language – Phenomenological
Structuralism, Bloomington and London: Indiana University Press.
Holenstein, E. (1992), ‘Phenomenological Structuralism and Cognitive Semiotics’, Scripta
Semiotica, 1 (interview by Roberto Benatti), 133–50.
Husserl, E. ([1900–1] 1970), Logical Investigations, trans. J. N. Findlay, London: Routledge.
Husserl, E. ([1890] 1970), Philosophie der Arithmetik: Logische und psychologische
Untersuchungen, ed. L. Eley, Husserliana XII, Den Haag: Nijhoff.
Husserl, E. ([1893–94, 1921] 2005), ‘Logische Untersuchungen Ergänzungsband. Zweiter Teil.
Texte für die Neufassung der VI. Untersuchung. Zur Phänomenologie des Ausdrucks und
der Erkenntnis’, in U. Melle (ed.), Husserliana, XX–2, Dordrecht: Springer.
Husserl, E. (1950a), ‘Cartesianische Mediationen und Pariser Vorträge’, in S. Strasser (ed.),
Husserliana I, Haag: Nijhoff. English translation: Cartesian Meditations, trans. D. Cairns,
The Hague: Nijhoff. 1960.
Husserl, E. ([1907] 1950b), ‘Die Idee der Phänomenologie’, in W. Biemel (ed.), Husserliana II,
Haag: Nijhoff. English translation: The Idea of Phenomenology, trans. L. Hardy, Dordrecht:
Kluwer Academic, 1999.
Husserl, E. ([1913] 1952), ‘Ideen zu einer reinen Phänomenologie und phänomenologischen
Philosophie II: Phänomenologische Untersuchungen zur Konstitution’, in W. Biemel
(ed.), Husserliana IV, The Hague: Nijhoff. English translation: Ideas Pertaining to a Pure
Phenomenology and to a Phenomenological Philosophy. First Book: General Introduction to
a Pure Phenomenology, trans. F. Kersten, Dordrecht: Kluwer Academic, 1982.
Husserl, E. ([1898–1925] 1980), ‘Phantasie, Bildbewusstsein, Erinnerung. Zur Phänomenologie
der anschaulichen Vergegenwärtigungen 1898–1925’, in E. Marbach (ed.), Husserliana,
XXIII, The Hague: Nijhoff. English translation: Phantasy, Image Consciousness, and
Memory (1898–1925), trans. J. B. Brough, Dordrecht: Springer, 2005.
­Husserl, E. ([1939] 1999), Erfahrung und Urteil, Hamburg: Meiner. English translation:
Experience and Judgment, trans. S. Churchill, Evanston: Northwestern University Press,
1973.
Ingarden, R. (1973), The Literary Work of Art, Evanston: Northwestern University Press.
Jakobson, R. (1990), On Language, Cambridge: Harvard University Press.
Lakoff, G. and M. Johnson (1980), Metaphors We Live by, Chicago: Chicago University Press.
Lakoff, G. and M. Johnson (1999), Philosophy in the Flesh, New York: Basic Books.
Langacker, R. (1987–91), Foundations of Cognitive Grammar, vols 1–2, Stanford: Stanford
University Press.
Langacker, R. (1991), Concept, Image, and Symbol. The Cognitive Basis of Grammar, Berlin:
Mouton de Gruyter.
Langacker, R. W. (2015), ‘Construal’, in E. Dąbrowska and D. Divjak (eds), Handbook of
Cognitive Linguistics (Cognitive Linguistics 39), 120–142, Berlin: Walter de Gruyter
Merleau-Ponty, M. (1945), Phénomenologie de la perception, Paris: Gallimard. English
translation: Phenomenology of Percepion, trans. C. Smith, London: Routledge and
Kegan Paul, 1962.
PHENOMENOLOGY AND SEMIOTICS 121

Merleau-Ponty, M. (1960), Signes, Paris: Gallimard. English translation: Signs, trans.

R. McCleary, Evanston: Northwestern University Press, 1964.
Mittelberg, I. (2018), ‘Gestures as Image Schemas and Force Gestalts: A Dynamic Systems
Approach Augmented with Motion-capture Data Analyses’, Cognitive Semiotics, 11 (1):
1–21.
Pelkey, J. (2017), The Semiotics of X. Chiasmus, Cognition and Extreme Body Memory,
London: Bloomsbury Academic.
Pelkey, J. (2018), ‘Upright Posture and the Meaning of Meronymy: A Synthesis of Metaphoric
and Analytic Accounts’, Cognitive Semiotics, 11 (1): 1–22.
Petitot, J. (1982), Pour un schématisme de la structure, Doctoral Thesis, Paris: École des Hautes
Études en Sciences Sociales.
Petitot, J. (1992), Physique du Sens, Paris: éditions du CNRS.
Searle, J. (1980), ‘Minds, Brains and Programs’, Behavioral and Brain Sciences, 3: 417–57.
Sonesson, G. (2015), ‘The Mirror in-between Picture and Mind. A Phenonomenologically
Inspired Approach to Cognitive Semiotics’, Chinese Semiotics, 11 (2): 159–80.
Stjernfelt, F. (2007), Diagrammatology: An Investigation on the Borderlines of Phenomenology,
Ontology, and Semiotics, Dordrecht: Springer.
Talmy, L. (2000), Toward a Cognitive Semantics, Cambridge: MIT Press.
Tylén, K., R. Fusaroli, P. F. Bundgaard and S. Østergaard (2013), ‘Making Sense Together:
A Dynamical Account of Linguistic Meaning-Making’, Semiotica, 194: 39–62.
Varela, F., E. Thompson and E. Rosch (1991), The Embodied Mind, Cambridge: MIT Press.
Zlatev, J. (2018), ‘The Semiotic Hierarchy and Phenomenology’, Cognitive Semiotics, 11 (1):
1–18.
Zwaan, R. (2004), ‘The Immersed Experiencer – Toward an Embodied Theory of Language
Comprehension’, The Psychology of Learning and Motivation, 44: 35–62.
Zwaan, R. (2014), ‘Embodiment and Language Comprehension: Reframing the Discussion’,
Trends in Cognitive Sciences, 18 (5): 229–34.
Zwaan, R. (2016), ‘Situation Models, Mental Simulations, and Abstract Concepts in Discourse
Comprehension’, Psychonomic Bulletin and Review, 23: 1028–34.
122
 ­C HAPTER SIX

Hermeneutics and
Semiotics
RONALD C. ARNETT AND SUSAN MANCINO

Hermeneutics and semiotics share an intellectual objective: to discern meaning and to

provide conversational dwellings for and to another sign that can direct interpretation
differently than expected. The connecting link between semiotics and hermeneutics is
twofold: attentiveness to what is before us and interest in the ‘not yet’ envisioned. Both
fields of study move from what is to the possible. Text and sign welcome respectful
learners who seek insight about current conditions that make possible a future that one
cannot predict but can potentially surmise. Martin Buber queried: What happens when life
seems to turn in a problematic direction? And you ask: How did I get here? He stated that
whenever one ignores existential signs, one invites unforeseen challenges. Buber wrote,
‘Each of us is encased in an armour which we soon, out of familiarity, no longer notice.
There are only moments which penetrate it and stir the soul to sensibility’ (Buber [1947]
2002: 12–13). To fail to understand the text, to disregard the signs, invites a practical and
existential ignorance that makes one vulnerable to a world that goes unacknowledged.
Without alertness to text and sign, one falls prey to taken-for-granted assumptions that
ignore both hermeneutics and semiotics, attempting to live in unreflective reaction alone
(Schutz 1967: 74). Attentiveness to text and sign, hermeneutics and semiotics, invites
dialogic participation in and with existence. Interpreters are not in control of the signs
they encounter but must be responsive to them.
Responsiveness engages semiotics and hermeneutics in performative action; this
chapter explicates that interaction in the following three sections. The first, ‘Textual
Traces: Historical Examinations’, provides an overview of hermeneutics followed by an
examination of semiotics. These brief historical accounts yield a dialogic perspective that
intersects with the mutual insight in each area of study. The second section, ‘Dialogic
Voices: Gadamer and Eco’, centres on our reading of hermeneutics and semiotics, stressing
Hans-Georg Gadamer’s Truth and Method and Umberto Eco’s A Theory of Semiotics
as representative texts that demonstrate the interconnections between each interpretive
tradition. The third section, ‘Dialogue and Semiosis in Action: Numero Zero’, explores
Eco’s final novel, Numero Zero, as an interpretive artefact elucidating the interplay of
hermeneutics and semiotics in action, emphasizing the interconnecting themes of dialogue
and semiosis. This chapter frames hermeneutics and semiotics as essential co-players in
the pursuit of dialogic knowledge and understanding.
This chapter provides a descriptive history of hermeneutics and semiotics with a
stress on two related performative metaphors: dialogue and semiosis. Hermeneutics
124 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

and semiotics explicate signification of human meaning. Hermeneutics depends upon

interpretive difference and uniqueness of questions that open a text within dialogic
understanding. Semiotics reveals an existential fact: one sign yields interpretive insight
for and about another. Multiplicity of possibilities, interpretations and discernment
originate with signs, questions, and texts. For example, Eco’s first novel and international
bestseller, The Name of the Rose,1 ‘offers a dialogue of many texts’ that moves the
reader to hypertextual connections embedded within history and ‘textual traces’ (Arnett
2017: 145–6). These textual traces announce dialogic semiosis in the interplay between
hermeneutics and semiotics.

TEXTUAL TRACES: HISTORICAL EXAMINATIONS

This section outlines descriptive histories of hermeneutics and semiotics as distinct
areas of inquiry. For both interpretive traditions, key texts recount the emergence and
evolution of each field of study. An emphasis on dialogue and semiosis informs our brief
historical examinations, beginning with hermeneutics and then moving to semiotics, as
they manifest reciprocal insights.

Hermeneutics
Three texts guide a short history on hermeneutics: Richard Palmer’s Hermeneutics:
Interpretation Theory in Schleiermacher, Dilthey, Heidegger, and Gadamer; Jean Grondin’s
Sources of Hermeneutics; and Maurizio Ferraris’s History of Hermeneutics. Each text
surveys the history of hermeneutics distinctively, offering a context for understanding
Gadamer’s philosophical hermeneutics.
Palmer’s2 work surveys the history of hermeneutics and provides an overview of
defining themes and theses. He delivers a thoughtful examination of the evolution of
interpretive inquiry from its ancient roots to the mid-twentieth century. Through
Friedrich Schleiermacher (1786–1834), Wilhelm Dilthey (1833–1911), Martin Heidegger
(1889–1976) and Gadamer, hermeneutics moves from precision of answer and reified
signs to a horizon of multiple possibilities. Palmer expands biblical interpretation to an
understanding of human experience. His final chapter lists thirty theses on interpretation
grouped into five central themes.
1. Palmer conceptualizes the hermeneutical experience as situated historically,
linguistically and dialectically, with the whole offering insight into the part and
the part into the larger understanding of the text (1969: 242). The hermeneutical
experience is ontological as it discloses being in the interplay of text and interpreter.
The hermeneutical experience is a ‘language event’ that both illuminates and
conceals (Palmer 1969: 243). Furthermore, it is objective; Palmer underscores that
one cannot impose meaning upon a text. One must disclose the ground from which
one stands, looks, observes and interprets (Palmer 1969: 243). The hermeneutical
experience is text driven (Palmer 1969: 244). It illuminates meaning within the
present and is a temporal ‘disclosure of truth’, attending to that which is before us
(Palmer 1969: 244–5). Finally, the hermeneutical experience recognizes aesthetics
as dependent upon the encounter of interpreter and text (Palmer 1969: 245).
2. Palmer conceptualizes transcendence as the hermeneutical experience inclusive of
interpreter and text, forgoing a subject-object framework (1969: 246).
HERMENEUTICS AND SEMIOTICS 125

3. Palmer conceptualizes autonomy and objectivity as an interpretive fact: the meaning

of a text goes beyond interpreter sensibilities (1969: 246–7).
4. Palmer conceptualizes methods as inclusive of the situatedness of the interpreter,
contrary to any effort to master a text – attentive to implications and not just form
(1969: 247–8). Palmer understands the text as speaking, requiring an interpretive
engagement that does not seek to dominate or smother (1969: 248). The interpreter
is a participant within the experience of the parole of the text (Palmer 1969: 248–9).
Hermeneutics is an encounter as one brings attentiveness beyond innocence to the
experience (Palmer 1969: 249). Hermeneutical experience seeks to understand the
interaction of interpreter and text (Palmer 1969: 249). A hermeneutical method
discloses temporal, not final, answers (Palmer 1969: 250). Finally, questions that
open the horizon of the text emerge within the experience of a given interpreter
(Palmer 1969: 250).
5. Palmer conceptualizes interpretive historical consciousness as one’s own experience
interrogating the historical consciousness of a given text (1969: 250–1). All literature
is historical and shaped within experience; such is the reason that hermeneutics
requires historical distance, respect for the text and recognition that understanding
is dependent upon the standpoint of a given interpreter (Palmer 1969: 251–2).
Understanding a text is never abstract or non-temporal; one must attend to the
dwelling of the work (Palmer 1969: 252). History and experience yield interpretive
insight. The task of hermeneutics is not scientific objectivity but historical encounter
between a given text and the particular lived experience of a given interpreter.
Palmer’s introductory text provides the coordinates from which we understand
the contributions of Grondin and Ferraris as they announce the evolving influences of
metaphysics, phenomenology and authorial intent.
Grondin’s3 (1995) Sources of Hermeneutics examines metaphysics and hermeneutics,
with an explication of phenomenology’s influence. From the work of Heidegger forward,
metaphysics increasingly takes on questionable status. Grondin contrarily relies upon the
insights of Gadamer to renew and broaden hermeneutic insights exemplified with the
importance of metaphysics. Metaphysics carries with it a given method of thought that
uplifts meeting and learning from recognition of bias in interpretive engagement with
a text from a particular standpoint. Grondin attends to phenomenology’s contribution
to hermeneutics, as seen in his chapter emphasizing ‘Husserl’s Silent Contribution to
Hermeneutics’. Husserl’s notion of intentionality explicates an existential fact: ‘the thing
itself’ changes in its engagement with a given event, object or person. The lifeworld
calls forth particularity of the things themselves that morph and shift with responsive
perspective to a changing historical moment.
Grondin examines the early work of Husserl’s student Heidegger and his young
Hegelian insights, which emphasized the notion of the ethical (1995: 47). Emmanuel
Levinas, however, provided a public counter to Heidegger’s position, with a focus on ethics
as first philosophy. Grondin contends that there was a serious and significant implication
in Heidegger’s stress on the awaking of an ethical consciousness. This orientation reflects
a Hegelian perspective in which an original ethical consciousness can seduce one into
the search for the correct moral conscience. Grondin continues this caution with Jürgen
Habermas’s argument that Heidegger’s philosophical turn lacked argumentative rigour
and actually justified Nazi involvement. Heidegger radicalizes a path towards thinking,
free from metaphysics and attentive to the immediacy of thinking itself. Grondin points
126 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

to a rigidity that rests within a critical disposition towards metaphysics. The existential
reality of particularity and ‘the things themselves’ provides, for Grondin, a call for
multiplicity that undergirds Gadamer’s project.
Grondin announces phenomenological influences on hermeneutics with particular
insight into how they contributed to Gadamer’s philosophical hermeneutics. Grondin
indicates that Gadamer’s most well-known tome, Truth and Method, originally published
in 1960, introduces philosophical hermeneutics as the primary focus of Gadamer’s corpus
with an emphasis on a triad of philosophy, art and history. Grondin adds a perspective
that language is the channel, the driver of ontological content and a guide for philosophy,
art and history. Such insight philosophically does not come from a method or a final
story. Understanding emerges from an interpreter tied to a given event or object situated
within a particular historical moment. Philosophical hermeneutics is always reflexive,
working within limits that are organically before us and require acknowledgement.
Grondin stresses that both Gadamer and Augustine point to hermeneutics as universal.
Hermeneutics, for Augustine, houses the revelatory, and Gadamer stresses hermeneutics
as the home of the unexpected and ‘not yet’.
Gadamer presupposes humanism in hermeneutics, but not an anthropocentric
humanism. The human is not at the centre, but is the responsive recipient within
experience and the historical moment. The philosophical humanism of Gadamer offers
an embedded and situated sense of interpretation, forever responsive to limits. Heidegger,
of course, abandons all humanism and develops an anti-humanism. Gadamer, however,
emphasizes a humanism deeper than the communicative agent alone, recognizing that
we live within a moment historically defined. The interpretive task is one of dialogic
responsiveness, not imposition of one’s own demands and expectations of a given text.
Gadamer’s hermeneutics ultimately situates the human being within a dialogic moment,
one that is ever responsive to the past, the present and the ‘not yet’. Understanding is
an engagement of difference responsive to a bias that is not psychological in nature but
historically given and dialogically understood.
In History of Hermeneutics, Ferraris4 ([1988] 1996) acknowledges the shift from
authorial intent into the realm of an interpretive exchange between and among texts,
readers, authors and the historical moment, with each temporally situated. He begins in
the classical period with Hermes, the messenger god, who witnesses to the practical and
communicative nature of interpretation. Hermeneutics, in a classical sense, announces
expression; early work on hermeneutics privileges authorial intent. This authorial
orientation continues through the Enlightenment with increasing emphasis on the
communicative agent. This line of inquiry undergirds Friedrich Nietzsche’s (1844–1900)
and Sigmund Freud’s (1856–1939) discussions of a deconstructive hermeneutic of
suspicion. The ontological project of Heidegger brings forth an existential reality: the
human’s thrownness into existence.
Heidegger undercuts authorial intent and stresses existential givenness. He shifts
hermeneutics into ontological existence, the field of interpretive understanding and
play, which opens the text for the reader. Gadamer, a student and research assistant for
Heidegger, adds bias to the notion of existential thrownness. Gadamer’s philosophical
hermeneutics situates bias as the central disruption of the assumption of authorial intent.
Philosophical hermeneutics presupposes that meaning resists ownership by authorial
mandate. This position finds its greatest emphasis in Jacques Derrida’s (1930–2004)
deconstruction; Gadamer acknowledges multiple biases that open texts differently and,
at times, contentiously.
HERMENEUTICS AND SEMIOTICS 127

Ferraris concludes with Gadamer and Ricoeur, perhaps the most definitive scholars on
philosophical hermeneutics. Gadamer provides a dialogic entrance into the humanities
and the human sciences, recognizing the importance of a community of scholars in the
discernment of temporal validity of a given interpretation. Similarly, Ricoeur immersed
hermeneutics in multiple conversations that seek to emancipate interpretation from the
text alone. Ricoeur, however, distinctively embraced the narrative nature of interpretation,
as well as the existential reality of a hermeneutics of suspicion played out thoroughly by
Freud, Nietzsche and postmodern discourse that called a hermeneutics of the universal
into question.
Ferraris describes the historical shifts in the hermeneutic project from a classical
era of expression, to one of authorial intent, and finally to the onset of philosophical
hermeneutics and its interpretive engagement between and among author, text, history
and interpreter. Hermeneutics assumes the interpreter dwells within existence, bringing
particularity of perspective and experience in the meeting of the lifeworld. Bias is
a pragmatic and existential reality that opens up and limits interpretive insights. This
perspective guides the insights of semioticians such as Susan Petrilli, Augusto Ponzio and
John Deely.

Semiotics
We turn to two texts on the history of semiotics: Petrilli and Ponzio’s Semiotics
Unbounded: Interpretive Routes through the Open Network of Signs and John Deely’s
Semiotic Animal: A Postmodern Definition of ‘Human Being’ Transcending Patriarchy
and Feminism. Each text situates semiotics historically. We begin with Petrilli and
Ponzio’s text, which provides the historical scope of semiotics and semioethics, and
then transition to Deely’s work, which situates semiotics in a postmodern recognition of
multiple interpretive possibilities co-present in the signification of signs. Petrilli, Ponzio
and Deely collectively assert that, in an era of difference and hypertextual meaning, the
interpretation of signs carries semioethical implications. Signs and interpretation are
value laden, not neutral or objective.
Petrilli and Ponzio5 (2005), in Semiotics Unbounded, provide a thorough and accessible
history of semiotics. Their approach is attentive to dialogue, otherness, communication,
ethics and responsibility; their project is a value-laden interpretive undertaking. We know
of no other scholars who provide such a consistent and extensive commitment to the
intersections between and among communication, semiotics and ethical implications.
Semiotics Unbounded covers more than 600 pages. It begins with an introduction to
their orientation towards semiotics followed by an overview of semioticians central to
their work: Charles Sanders Peirce (1839–1914), Lady Victoria Welby6 (1837–1912),
Mikhail Bakhtin (1895–1975), Charles Morris (1901–79), Thomas Sebeok (1920–2001),
Ferruccio Rossi-Landi (1921–85) and, ultimately, Eco. Petrilli and Ponzio include forms
of application commencing with a section on ‘Modelling, Writing, and Otherness’ (2005:
341) that stresses dialogue. The next section, ‘Predicative Judgment, Argumentation, and
Communication’ (Petrilli and Ponzio 2005: 429), examines (mis)understanding, closed
and open communities, global communication, biosemiotics and semioethics. Petrilli and
Ponzio engage semioethics in a number of projects. They explicate semiotics and semiosis
within an interpretive context of responsibility, attending to issues of dispossession,
extralocalization (Petrilli and Ponzio 2005: 558) and our biosemiotic condition. They
engage in critical, philosophical, theoretical and interpretive judgement that moves
128 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

otherwise than conventional efforts of dominated technical control. Their intellectual

contribution situates an interpretive position within semiotic responsibility ever attentive
to semiosis.
Petrilli and Ponzio explicate the semiotic significance of otherness and ethics in semiosis
as they review Peirce’s emphasis on Firstness, Secondness and Thirdness, which they
connect, respectively, ‘to abduction, deduction, and induction in logic;7 to icon, index,
and symbol in semiotics;8 and to agapism, anancism, and tychism in ontology9’ (2005:
28). Petrilli and Ponzio emphasize Firstness as ‘originality’, in phenomenological terms,
a ground, undifferentiated material that requires eventual differentiation. Secondness is
an ‘over-againstness’ that offers a first step of differentiation in relation to an object
(Petrilli and Ponzio 2005: 28). Thirdness is that which emerges in between signs in triadic
relations, enhancing differentiation dialogically through mediation (Petrilli and Ponzio
2005: 28–31). With each move among Firstness, Secondness and Thirdness, greater
clarity of signification emerges.
Petrilli and Ponzio provide an examination of Eco’s reliance on Peirce and the
triadic relationship of sign, object and interpretant. This triadic notion forms semiosis,
which makes otherness consistently elusive. Petrilli and Ponzio emphasize Eco’s ‘limits
of interpretation’ as a reminder of the otherness of signs (2005: 298);10 signs do not
adhere to our demands but yield meaning in their otherness. Understanding otherness as
difference makes interpretive insight of semiosis possible. Understanding interpretation
as potentially solidified and unduly clear moves contrary to Eco’s understanding of
semiotics in everyday engagement. His contribution influenced Italy and made a global
contribution to semiotics.
Petrilli and Ponzio credit Eco with the introduction of ‘interpretation semiotics’ as
a passage beyond decodification semiotics (2005: 299). Decodification presupposes an
ability to stand above the human context and the historical moment; interpretation
relies upon such existential realities. For Eco, however, codes provide interpretive
paths connecting source and destination without guaranteed agreement of signification.
Semiotics, particularly as understood through Peirce’s ontology, walks between the
objective and the subjective, and is often described as a form of ‘objective idealism’ (Petrilli
and Ponzio 2005: 306). Eco understood the importance of unlimited semiosis without
falling prey to deconstructionism. Following the work of Peirce, Eco unites interpretation
with unlimited semiosis and an emphasis on openness (Petrilli and Ponzio 2005: 303).
Eco distinguishes between open texts and closed texts as a means for understanding the
interpretive engagement among authors, texts and readers. While a closed-text implements
constraints to guide readers in the interpretation of signs, the open text contains an excess
of information that requires readers to venture deep into the path of unlimited semiosis,
navigating multiple possibilities for meaning and uncovering interpretive plurality.
Eco’s project moves contra to the extremes of ideology as witnessed by his notion
of communication as a theory of sign production. Petrilli and Ponzio refer to Eco’s
description of ideology as ‘a false consciousness’, placing it within sign production
inattentive to the existential normality of semiosis (2005: 321–2).11 Interpretation and
semiotics, for Eco, are kindred activities announcing ongoing dialogic relationships
between and among multiple texts and signs. With attentiveness to Eco and Bakhtin,
Petrilli and Ponzio presuppose that dialogue constitutes how human beings make sense
of the world. Interpreting meaning within the dialogue between and among signs engages
abduction as a response to unlimited semiosis. Abduction presupposes the necessity of
background in order to discern foreground issues in a hypertextual historical moment.
HERMENEUTICS AND SEMIOTICS 129

Eco defined hypertextuality as the co-presence of multiple texts, codes and signs in a
given moment.
Deely12 (2010) situates semiotics in a hypertextual and postmodern historical moment
with his book Semiotic Animal. Deely and Eco deal with postmodernity as an interpretive
revolution, reclaiming attentiveness to a hypertextual recognition that does not fall prey
to modernity’s linear perspective and hypermodernity’s emphasis on the individual.
Postmodernity is often confused with hypermodernity. Postmodernity, however, resists
the paradigmatic progression of modernity, which blinds one to the responsibility of
being ever-vigilant semiotic animals.
Deely contends that human interpreters act as semiotic animals in an age thematized by
an existential fact: we disagree on the value implications and signification of signs, making
the notion of semioethics vital. Semioethics attends to how signs matter within a human
community and how their meaning shifts, ever attentive to semiosis. Deely reminds us
that semiotics does not live within an ideological system in a dwelling of semiotic animals
that attends to semiosis in individual and collective lives (2010: 32–3). Historical epochs
shape signification of signs and codes. Postmodernity presupposes hypertextual meaning,
suggesting multiple interpretive possibilities are co-present in a given sign.
Awareness of semiosis is the semiotic heart of a postmodern/hypertextual age (Bain
2006). As a single sign morphs in semiosic engagement, it carries forth hypertextual
meanings. Hypertextuality suggests temporal clarity without universal final answers by
presupposing meaning situated within signs, relations and intersubjectivity. Deely attends
to an intersubjective world that lends itself to Peirce’s triadic relations of representamen
(form of/potential for meaning), object (sign vehicle represented by sign) and interpretant
(action of sign).13 From this pragmatist perspective, the structure of a sign is a triadic
relationship, which forms a trinity that makes semiosis and the advancement of insight
organically inevitable.
Deely discusses the semiotic animal as beyond mere separation, ever responsive to
intersubjective understanding. The semiotic animal dwells distinctly situated in place,
space and nature. He emphasizes that semioethics does not view the human as merely
rational, but called forth as a semiotic animal that is responsible, upright and willing to
attend to existential dilemmas. The semiotic animal responds to a life of responsibility
inclusive of the biosemiotic sphere. Semioethics functions as an interpretive code for
semiotic maturity defined by responsibility for others.
Semiotics, like hermeneutics, opens possibilities for interpretive insight and recognizes
the historical, cultural and temporal situatedness of signs, texts and interpreters. The history
of semiotics witnesses movement from universal reification and certitude to an increasing
awareness of uncertainty and ongoing semiosis. The history of hermeneutics follows a
similar course, moving from certitude and psychological imposition to the admission
of bias and prejudice as fundamental to the act of interpretive understanding. These
historical examinations illuminate two key thinkers within hermeneutics and semiotics:
Gadamer and Eco, respectively. Gadamer and Eco offer dialogic voices that build upon
historical attentiveness. From a continental understanding of dialogue, this interpretive
engagement commences long before one meets another, a text, or an event (Buber [1947]
2002). Dialogue lives within history, meeting the present with responsiveness to the ‘not
yet’. Dialogue functions as a communicative phenomenon between and among historical
moments, with semiosis engaging a similar creative interplay of meaning. Gadamer and Eco
recognize dialogue and semiosis as guiding metaphors that elucidate the interconnections
between hermeneutics and semiotics.
130 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­DIALOGIC VOICES: GADAMER AND ECO

The dialogic voices of Gadamer and Eco represent philosophical hermeneutics and
interpretation semiotics, respectively. The following discussion situates each thinker
within his historical moment before outlining a work reflective of his thinking – Truth
and Method for Gadamer and A Theory of Semiotics for Eco. These works characterize
their authors’ expansive contributions to hermeneutics and semiotics. Their insights
illuminate dialogue within philosophical hermeneutics and semiosis within the pragmatic
perspective of interpretation semiotics. The temporal pursuit of truth and understanding
coordinates the contributions of hermeneutics and semiotics.

Gadamer’s Truth and Method

Gadamer (1900–2002) studied at the University of Marburg and the University of
Freiburg, where he was a student of and research assistant for Heidegger. He began his
teaching career at Marburg in 1929 before teaching at Leipzig University from 1938
to 1948 and then at Heidelberg University, where he succeeded Karl Jaspers as chair
and where he would later retire as professor emeritus in 1968 (Mueller-Vollmer [1985]
2006: 256). At the age of sixty, Gadamer penned Truth and Method, what Georgia
Warnke (1987) named his magnum opus, what Palmer referred to as a ‘masterwork’
(1987: 135) and what Klaus Dockhorn described as ‘a liberating breakthrough to new
horizons’ (1980: 160). According to Palmer, Truth and Method introduced philosophical
hermeneutics and revolutionized existing perspectives on aesthetics, history and language
with practical insight and a counter to modern thought (1987: 135).
Truth and Method has three parts: the first is on truth and art, the second on truth and
the human sciences, and the third on the ontological shift to language in hermeneutics.
Ultimately, Gadamer illustrates philosophical hermeneutics with truth as a temporally
situated principal objective and method as a desire for application. Gadamer’s scholarship
rests within a humanities-based commitment to temporal understanding, in which one
attempts to discern appropriate questions and responds to the conditions of a given
historical moment. He frames understanding as temporal, finite and dialogically responsive.
The aesthetic dimension is important to Gadamer as he emphasizes a humanistic tradition
of the human sciences.
Gadamer begins with an emphasis on four concepts central to the aesthetic dimension:
Bildung, sensus communis, judgement and taste. Gadamer invokes the notion of Bildung
as more than culture alone; he emphasizes its ancient mystical tradition, in which the
image of God, carried in one’s soul, requires the cultivation of self ([1975] 1993: 11).
This cultivation is a practical spirit of working with historical consciousness in order to
attend to the questions of a given moment and to render proper temporal responses.
Likewise, Gadamer contends that sensus communis, Vico’s notion of common sense, is
attentive to community. Sensus communis initiated Vico’s use of a humanistic rhetoric
against a modern understanding of science and makes the sympathy that the Earl of
Shaftesbury discussed in the early Scottish Enlightenment possible (Gadamer [1975]
1993: 24). Sensus communis manifests the practical knowledge of phronesis (Gadamer
[1975] 1993: 21).
Gadamer links sensus communis to judgement where one learns that common sense
emerges in judgements about ‘right and wrong, proper and improper’ ([1975] 1993:
32). The break from sensus communis to the notion of judgement comes with Immanuel
HERMENEUTICS AND SEMIOTICS 131

Kant (1724–1804). His doctrine of schematism understands the human being as engaged
in self-legislation. However, Kant places judgement of taste within communities, the
social life of persons. Judgements attending to sensus communis require taste educated
within, or at least about, a given community. Kant actually equates taste with what
could be termed a ‘true sense of community’, in which he brings the transcendental to
his understanding of sensus communis (Gadamer [1975] 1993: 34). Kant enters into
pure judgement, which endangers observations of and about a culture or phenomenon
contrary to one’s own, freeing the notion of taste from sensual and rational prejudices.
This transcendental engagement brings Kant to the connection between taste and genius,
with genius beginning to remove the notion of taste from consideration. As taste loses its
power, it becomes something more akin to testimony (Gadamer [1975] 1993: 56).
Gadamer enters into a critique of pure rationality with the concept of Erlebnis, or
experience, which he connects to historical consciousness via tradition. Consciousness, for
Gadamer, is not a given event, but a time in which there is a form of consciousness where
one experiences a sense of knowing in the lifeworld. His understanding of experience and
a historically developed sense of consciousness contrasts with the transcendental position.
Gadamer critiques the abstraction of the aesthetic. Gadamer argues that speculative
idealism challenged the aesthetic subjectivism and agnosticism of Kant with the adoption
of a standpoint of ‘infinite knowledge’ ([1975] 1993: 99). Contrarily, Gadamer emphasizes
finiteness; he underscores the rightness of Heidegger’s modern subjectivism that opens
understanding temporally without reliance upon the transcendent and the infinite. Being
dwells, for Heidegger, within experience and existential life, a point that Gadamer carries
forth in his emphasis on experience and tradition. Interestingly, phenomenology returns
to the aesthetic experience tied to the transcendental, with existential phenomenology
attentive to experience and traditions. However, when Gadamer refers to tradition, he
emphasizes the plural: traditions. He jettisons a modern universal for acknowledgement
of multiplicity of traditions.
The notion of play is Gadamer’s signature display of the union of temporal experience
and embodiment. One cannot transcend play; one is within it – a characteristic of
experience is one’s transformation into and within the structure of play. It is not a second
version, copy, imitation or representation, but rather engagement in the activity itself.
Play requires structure and engaging the confines of coordinates. The art of play attending
to historical consciousness and particularity results in an interpretive self-forgetfulness
within the engagement of play. There are limits on one’s ability to play; undue reflection
interrupts play as an embodied doing (Arnett 2018). The embodiment of experience
permits a dramatic piece or musical work to announce itself differently, no matter how
many performances. Play has a communion component of dialogic engagement.
Gadamer understood the ‘subjectivist attitude’ of play in modern aesthetics as essential
to ‘the event of art proper’ ([1975] 1993: 145). Play underscores his discussion of pictures,
decorative arts and architecture in relation to signs and symbols. Gadamer situates play in
relation to a picture, which rests somewhere halfway between a sign (what he understands
as ‘pure substitution’) and a representation (what he refers to as ‘pure indication’),
offering an experiential space that renders a temporal glimpse of play in action ([1975]
1993: 152). Gadamer identifies an ontological difference between picture and sign: ‘The
picture does not disappear in pointing to something else but, in its own being, shares in
what it represents’ ([1975] 1993: 153). Likewise, the picture dwells halfway between a
sign (‘pure taking-the-place-of-something’) and a symbol (‘pure pointing-to-something’)
(Gadamer [1975] 1993: 154). The task of play is not one of restoration, but temporal
132 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

understanding driven by knowing imagination. Hermeneutics is akin to play in that

experience, the historical moment and the activity invite dialogic meeting.
Gadamer’s dialogic engagement with philosophical hermeneutics relies upon
Schleiermacher’s effort to make texts understandable. Schleiermacher employs a
psychological move of one seeking to place oneself in the space of a writer or reader
(Gadamer [1975] 1993: 187). Schleiermacher, however, points to a ‘paradoxical formula’:
the original writer is generally not the best interpreter of a given text (Gadamer [1975]
1993: 193). Schleiermacher understood the expressive nature of interpretation. Gadamer
contends that there is a beginning shift from individual expression in interpretation to
historical thinking, which offers a space for interpretive insight. Productivity, meaning
and understanding dwell within history. Gadamer cites Leopold von Ranke (1795–1886),
tying hermeneutics to historical thinking and the power of necessity (Gadamer [1975]
1993: 204–12). This understanding of history embraces interpretive engagement within
tradition. There is a movement from Schleiermacher’s romantic theory of understanding,
where interpretation belongs to a single interpreter, to attentiveness to shifts in the
historical moment. For Gadamer, dialogue between and among historical moments
situates interpretive understanding.
As Gadamer shifts to Dilthey’s influence, he writes, ‘Dilthey’s hermeneutics of life
fundamentally seeks to retain the historical worldview’ ([1975] 1993: 226). History
moves interpretation from a psychological framework into the temptation of a historical
consciousness that flirts with objectivity, universality and scientific certainty. Dilthey
sought a human science that challenged the dogmatic, while attempting to forge an
insight closer to objectivity than raw subjectivity. Dilthey understood the dangers of
objectivism and the reification of insight; he sought a horizon of interpretation without
a ‘rigid boundary’ (Gadamer [1975] 1993: 245). Dilthey began with experience; yet,
Heidegger attacked his desire for a transcendental reduction for an emphasis on the
existential thrownness of human life. For Heidegger, history does not lend itself to a
transcendental lens. There is no such thing as ‘a particular ideal of existence’ (Gadamer
[1975] 1993: 263). Thrownness indicates not a transcendental gaze, but a situated and
embodied effort to understand temporally. All insight, whether historical or situated,
begins with an existential position resulting from thrownness.
Gadamer outlines the Enlightenment prejudice against bias itself ([1975] 1993: 271). It
is impossible to understand a historical moment or tradition from a transcendental posture.
This perspective evades a recognition that the interpreter begins with assumptions, not
transcendental clarity. These suppositions are sometimes known, assumed and unearthed
by others. Understanding is finite; one cannot gain an absolute, perfect or pure historical
knowledge of existence. Western history and the Enlightenment share a love for progress
that permits us to distance ourselves from the preconditions of tradition, history and
environment. This interpretive stance is existentially wrongheaded and opposed to a
hermeneutical attitude respectful of the text and the limits of one’s own gaze. Gadamer
emphasizes historicity for the text, the interpreter and the finiteness of possibilities for
interpretation. Historicity offers a unity of contraries: finiteness linked directly with
multiplicity of possibilities. Likewise, interpretive horizons embody both multiplicity and
limitations (Gadamer [1975] 1993: 302).
Historical knowing makes practical knowledge possible and absolute knowledge
unattainable. Gadamer also stresses the relevance of Aristotle’s emphasis on action and
its implications for interpretation engaged in practical knowledge, contrary to technical
language ([1975] 1993: 315). Practical knowledge represents a deep interpretive
HERMENEUTICS AND SEMIOTICS 133

humanities engagement, with application void of absolute certainty associated with

technical language. Interestingly, Gadamer cites the Pietist theologian tradition as a basis
for the issue of application ([1975] 1993: 307). Gadamer’s humanities-centric form
of hermeneutics emphasizes reading as a form of application that dwells in tradition
responsive to change and not infinitude ([1975] 1993: 340).
Gadamer’s emphasis on experience tied to finitude invites dialogic engagement
with the ‘thou’ of historical consciousness and assumes interpretive bias, perpetually
announcing limits. Gadamer places interpretation within the engagement of questions
and answers; he writes, ‘Thus interpretation always involves a relation to the question
that is asked of the interpreter. To understand a text means to understand this question’
([1975] 1993: 370). Gadamer understands opinion as an effort to answer a question
within a historical horizon. He writes, ‘To think historically always involves mediating
between those ideas and one’s own thinking. To try to escape from one’s own concepts
in interpretation is not only impossible but manifestly absurd’ (Gadamer [1975] 1993:
397). Gadamer articulates the fundamental alignment of dialogue, interpretation and
understanding.
As dialogical creatures, humans understand within limited historical perspectives,
permitting questioning and interpreting within particular experience. Interpretation
requires greater knowledge and reflection beyond the immediacy of contexts between
persons whose practices unite them. The further one is from a given text historically, the
greater the need for reflection upon the biases of text and interpreters. One of Gadamer’s
most fundamental contributions is moving hermeneutics from the psychological and the
individual to the historical, emphasizing temporal limits, dialogue, multiple horizons of
interpretive possibilities and the reality of incorrect interpretations.
For Gadamer, limits surround texts, warning against the extremes of subjective
and relativistic interpretation without assuming an objective, universal hermeneutic.
Gadamer’s philosophical hermeneutics privileges dialogue in the emergence of meaning,
as interpreters and texts meet in the discernment of multiple horizons of insight.
Multiplicity of possibilities unites Gadamer’s philosophical hermeneutics and Eco’s
interpretation semiotics. For Eco ([1990] 1994), a commitment to multiple paths of
meaning, particularly through semiosis, limits the possibilities of overinterpretation,
unresponsive to the constraints of a text. Eco’s semiotic engagement, like Gadamer’s
hermeneutics, positions interpretation as a meeting of texts between the extremes of
relativism and objectivity, which is central to Eco’s highlighting of semiosis.

Umberto Eco’s A Theory of Semiotics

Eco (1932–2016) studied medieval aesthetics under Luigi Pareyson at the University of
Turin,14 where he penned what would become his first book, The Aesthetics of Thomas
Aquinas. His early work focused on cultural aesthetics, which framed his hermeneutic
entrance into semiotics. Eco approached semiotics as ‘a logic of culture’, which he describes
in the first chapter of A Theory of Semiotics, the inaugural text in Indiana University
Press’s Advances in Semiotics series edited by Sebeok. Deely portrays the work as ‘one
small step for philosophy, one giant leap for the doctrine of signs’ (1997: 82). Reviews
from Richard L. Lanigan (1978), Frank E. X. Dance (1977) and Thomas W. Benson
(1977) announce its creative relevance for communication studies. Lanigan emphasizes
the work’s distinctive and novel contribution in its theoretical union of communication
theory and rhetorical theory (1978: 345). Although Eco’s semiotic project continued to
134 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

evolve throughout his corpus, A Theory of Semiotics outlines the heart of his interpretation
semiotics (Deely 1997: 82; Petrilli 1997).
In A Theory of Semiotics, published in 1976, Eco translates his existing research on
semiotics into English and introduces interpretation semiotics (1976: 314). The work
contains five parts:
1. A framing of semiotics as a logic of culture
2. An examination of the issues of signification and communication
3. An explanation of signification as a theory of codes
4. An exploration of communication as a theory of sign production
5. An acknowledgement of the human interpreter as ‘the subject of semiotics’
In the foreword, Eco outlines four major objectives as he advances his semiotic project
(1976: viii):
1. Seeking a semiotic theory of referents
2. Bringing semantics and pragmatics together
3. Questioning the common classical topologies that underscore and define the notion
of signs
4. Advancing his critique of iconism beyond the excesses of trivialization and undue
clarity Additionally, Eco’s theory of sign production moves beyond a typology of
signs to a typology of sign production
Eco begins by framing semiotics as ‘a logic of culture’ (1976: 3). As one sign yields
insight into another, it is simultaneously a trace of yet another sign – each contextually and
experientially understood within culture(s). Much like Gadamer’s use of tradition, Eco’s
discussion of culture lies within a multiplicity of cultures. To apprehend a sign, one must
understand the cultures situating it. Sign functions lend insight into a given time, place and
mode of interaction. Eco’s logic of culture differentiates information, signification and
communication. Information measures probability void of cultural meaning. Signification
considers the codes, patterns and possibilities of meaning within cultural exchange
between people. Communication requires embodied labour to produce and interpret
meaning within the experience of cultural signs. Eco understands communication and
signification as deeply intertwined in the recognition that the meaning of signs emerges
in the meeting between and among persons and codes. With reliance upon Peirce, Eco
attends to the cultural import of semiosis between and among persons. As one navigates a
culture composed of sign functions, one participates in signification and communication
emergent in temporal groupings. Eco refers to signification as a theory of codes and
communication as a theory of sign production; codes offer semiotic spaces for the labour
of sign production. Eco’s theory of sign production revolves around the notion of labour
beginning with the mere ‘act of uttering’ (1976: 151). In the labour of sign production,
semiotics and interpretation of signs guide the interplay of experience and culture.
Eco overviews multiple types of labour, including abduction, overcoding, undercoding,
rhetorical code switching and code making (1976: 153–5). Semiosis itself is an act of
labour as one sign lends itself to producing another (Eco 1976: 71). Labour is essential
to Eco’s understanding of semiotic literacy. Semiotics as a logic of culture augments
judgement that recognizes semiosis in action. Semiosis navigates a multiplicity of codes and
engages texts according to the perceptual possibilities of interpretation (Eco 1976: 159).
HERMENEUTICS AND SEMIOTICS 135

For Eco, ‘perceptual signification’ is one of ‘semiotics’ limits’ (1976: 167). He traces
conceptualizations of perceptual meaning as it applies to semiotics through Locke,
Berkeley, Peirce and Husserl. From this position, Eco emphasizes how perceptions form
ideas; their meaning evolves in the infinite regression of unlimited semiosis, necessitating
interpretation of signs.
Eco cites Peirce’s understanding of ideas as signs that emerge from experience and
cultural engagement (1976: 165). Eco describes this perspective as Peirce’s ‘philosophico-
semiotical position’ (1976: 166); from this insight, Eco situates linguistic and perceptual
meaning in the philosophical tradition of phenomenology of perception as being
equivalent to the semiotic cultural unit. Eco references Husserl, writing, ‘semiotic
meaning is simply the socialized codification of a perceptual experience in which the
phenomenological epoché should restore to us in its original form’ (1976: 167). Signs,
for Eco, have phenomenological points of origin. No matter the source, experience and
cultural framings within a social world influence the interpretation of signs.
Eco then shifts to a critique of Peirce’s trichotomy of symbols, icons and indices, which
he describes as ‘untenable’, particularly when placed within the context of signification
(1976: 178). This chapter primarily focuses on Eco’s critique of iconism. Eco jettisons
the notion that iconism relies on similarity. He provides a phenomenological reading
that attends to uniqueness. For Eco, the key is discerning and engaging difference, not
embracing a modern understanding of commonality. Within semiotic study, ‘isomorphism’
depicts a modern cul-de-sac. Isomorphism is simply unresponsive to difference. In
phenomenological terms, Eco is receptive to Husserl’s later work The Crisis of the
European Sciences and Transcendental Phenomenology, in which a given event, or in this
case sign production, is like an onion yielding multiple layers and possibilities (Husserl
[1954] 1970: 11–12). Gadamer, of course, would refer to this reality as a horizon in which
there are multiple points of connection, with none providing isomorphic agreement.
To resolve what Eco finds as unsatisfactory and naive in the trichotomy, he replaces
Peirce’s typology of signs with a classification of modes of sign production (1976: 217).
His discussion of sign production emphasizes two essential coordinates: grammatical
units in ‘isolated sign functions’ and global ‘textual units which assume the role of large-
scale (undercoded) sign functions’ (Eco 1976: 217). These macro-units are impossible
to separate completely from grammatical units in the communicative labour of sign
production. Eco’s modes of sign production emphasize labour in four distinct and related
dimensions:
1. Labour physically engaged prior to discerning a given sign
2. Labour that determines how closely signs adhere to established codes
3. Labour that moves beyond assumptions of similarity to attend to signs materially
and/or arbitrarily representing referents
4. Labour that reflects cultural knowledge of pre-existing codes or invents new
possibilities for interpretive meaning. Modes of sign production are a priori,
including the complexity of that which one discerns and the necessity of context.
Eco offers four forms of physical labour that produce signs: recognition, ostension,
replica and invention. Recognition produces signs via empirical traces motivated by
experience and pre-existing codes. Ostention accounts for representations that aim to
produce meaning through signs created via perceived similarities between expression
and content (Eco 1976: 227). Replica creatively employs arbitrarily assigned signs that
136 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

move between and among pre-established codes and proposed texts. Invention produces
signs while simultaneously constructing codes. Invention permits the possibility of
code making based on the recognition, ostension and replica of signs situated in codes;
invention produces what is ‘not yet’ culturally known, grounded in what is (Eco 1976:
249–50). Eco relies upon Vico to explain that ‘[n]o new culture can ever come into
being except against the background of an old one’ (1976: 256). Moving from existing
signs, invention produces new sign functions in uncoded contexts based upon interpretive
inferences of general rules and specific cases (Eco 1976: 131–2). Invention, ‘the not yet’,
requires attentiveness to what is before us and what has preceded us – the invention of
new signs emerges only from semiosis of existing signs.
For Eco, invention embodies the aesthetic experience and, specifically, the aesthetic
text as an act of productive labour. Eco suggests that philosophical aesthetics and the
layperson embrace invention. What differentiates the two are rhetorical devices that
invite ‘imaginative interplay of metaphors’ (Eco 1976: 262). The aesthetic is a poetic
function with possibilities for both ambiguity and directive focus. Eco suggests that an
aesthetic text has ambiguity in its expression-plane (transmission of signs and denotative
markers) and content-plane (channel of sign transmission with connotative markers),
which interrupt conventional pairings of sign functions and require reflection of new
possible meaning (1976: 264). A self-focusing aesthetic experience is a moment in which
the ambiguity ratio is minimal in relation to temporal clarity. When the aesthetic text is
overcoded, in either expression or content, its organization increases and becomes more
ambiguous with increasing information.
The ambiguity of aesthetic texts leads Eco to ‘an idiolect’ or ‘new coding possibility
[. . .] spoken by only one speaker and understood by a very restricted audience’ (1976:
272). For Eco, the idiolect is a space of intuition in which expression and content
have not yet fully correlated in a sign function. In the invention of aesthetic texts, one
seeks the expression and content of rhetoric, elocution and overcoding. Particularity of
cultural knowledge grants overcoding clarity of expression and content, which permits
the rhetorical labour of code switching (Eco 1976: 276). When the capacity of code
changing is absent, one falls within the semantic category of ideology, refusing to attend
to semiosis and transformation. Social and cultural transformation occurs through the
rhetorical labour of inventing aesthetic texts that uphold or divert from existing cultural
codes. All transformations rely upon existing cultural sign systems generated through the
labour of sign production that frames semiotics as a path for both ‘social criticism’ and
‘social practice’ (Eco 1976: 298). In Eco’s concluding chapter, he turns to the semiotic
subject engaged in possibilities for semiotic criticism and practices (1976: 314). Eco
articulates the embodied and performative nature of semiosis in his ongoing emphasis on
interpretation semiotics.
The ‘paramount’ term for Eco is semiosis; labour of sign production generates new
insights of signification (1976: 316). He does not rely upon a methodological subject
but rather a subject embodied in the labour of producing signs within codes that lend
themselves to overcoding, undercoding and code switching. This semiotic experience
brings forth new insights. Eco understands labour tied to sign production through social
practice, social criticism and responsiveness to traces. He addresses the sender-addressee
relationship in the semiosic act. The speaking subject is a situated ‘concrete historical,
biological, psychic subject’ (Eco 1976: 314). For Eco, the subject engaged in the act
of utterance becomes a sign that carries forth his or her situatedness and brings forth
‘conveyed content’ (1976: 315). Eco works carefully to respect the cultural boundaries
HERMENEUTICS AND SEMIOTICS 137

of semiotics. Eco focuses semiotics on signification and the labour of sign production,
bypassing psychological ‘“deep” individual origins’ (1976: 315). Eco stresses codes and
sign functions tied to situated understanding, a particular ‘way of looking at the world’,
that brings new or different cultural insights made available through semiosis (1976: 315).
His understanding of semiotics veers from idealism into the practical, the pragmatic, the
social and the cultural.
Eco shares with Gadamer the practical, pragmatic, social and cultural importance of
interpretation. Their work unites hermeneutics and semiotics with differing descriptions of
how one idea, one event, one activity shifts and morphs into another. Their work reminds
us that interpretation is a revelatory activity, not an action of transcendental objectification.
Gadamer’s hermeneutics and Eco’s semiotics articulate the ‘how’ and the ‘why’ of the ‘not
yet’. The central metaphors for this ongoing communicative transformation are dialogue
and semiosis. The following section engages Gadamer’s dialogue and Eco’s semiosis in
relationship to Eco’s final novel Numero Zero, which embodies the possibilities of multiple
of interpretive directions, each with a bias and a shifting sense of signification.

DIALOGUE AND SEMIOSIS IN ACTION: NUMERO ZERO

This section turns to Eco’s seventh and final novel, Numero Zero, as a literary case study
demonstrating the intersections between hermeneutics and semiotics. Eco compared
Numero Zero, published in 2015, to his second novel, Foucault’s Pendulum, published in
1988. In an interview with National Public Radio’s (NPR’s) Scott Simon, Eco described
both novels as stories about ‘losers’, because they are ‘more interesting than the winners’
and are more likely to resonate with readers since losers outnumber winners (Simon and
Eco 2016: 1:30–1:50). Eco’s interest in losers announces the reality of interruption as
the heart of dialogue. Differing perspectives interrupt and educate us routinely. Losers
unfortunately ignore semiosis and fail to learn from interruptions. Eco’s work dwells
within dialogue attentive to interruption and semiosis that transforms one’s perception
of existential reality. Rocco Capozzi’s (2015) review of Numero Zero identifies Eco’s
semiotic stress on sign, perception, the power of media and interpretation. Capozzi
captures Eco’s work within the realm of questioning and uncertainty, the home of the
revelatory power of dialogue and semiosis.
Numero Zero provides a satirical examination of a tabloid press set in Italy in 1992 and
is told from the perspective of Dottor Colonna, a middle-aged failed journalist described
as ‘a compulsive loser’ (Eco 2015: 4). Colonna finds himself recruited to work for a
newspaper called Domani (which translates to ‘tomorrow’ in Italian); it is an outlet for
blackmail, conspiracy theories and speculation meant to threaten the realms of institutional
power, such as the government, the church, the business sector and the military. Eco
describes these dwellings as the ‘inner sanctum of finance and politics’ capable of disrupting
human lives (2015: 17). Commendator Vimercate, owner of a television station, several
magazines and a hotel chain, finances Domani. Vimercate functions as a parvenu;15 he
wants acceptance and access to the ruling power and institutions. Continually, he finds
this social status denied. The parvenu lives in a space void of dialogue and semiosis that
has the power of labour to transform an individual life and a sense of place. The parvenu
rigidly meets monologic engagement with those in power, denying the possibility of
transformation. Expectedly, Vimercate’s audience for Domani is the very group he seeks
to join without success – the powerful and socially elite. He approves stories capable of
drawing their eye and appreciation.
138 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Vimercate secretly informs the paper’s editor, Simei, about the ruse. Vimercate and
Simei understand that the paper will go undistributed, used merely as a tool for power
and influence. As Simei leads an editorial team of ‘losers’ commissioned to complete
Domani’s issue number zero, he reminds his team that the paper’s audience must
‘draw the conclusions’ about the questions raised in its articles (Eco 2015: 19). This
enthymematic characteristic is consistent with Eco’s theory of interpretation semiotics,
requiring the communicative labour of sign production. Readers encounter texts without
certainty of there being a single correct interpretation. In accordance with the theme of
Numero Zero, Eco acknowledged in the NPR interview his hope that readers ‘become
more suspicious and attentive’ of news media (Simon and Eco 2016: 3.06). He laments
public indifference to tragic news stories. This indifference is the ‘real tragedy’ and the
‘final moral’ of Numero Zero (Simon and Eco 2016: 4:00–4:15). Both dialogue and
semiosis work within a realm that necessitates active participation and awareness of the
fallacy of absolute truth. Eco calls for reflective and thoughtful interpretive engagement
as a primary tool, which protects against ideological false consciousness, the anathema of
dialogue and semiosis.
Both dialogue and semiosis illuminate revelatory insight, contrary to a subjective
monologic imposition upon existence. One of Eco’s characters, Romano Braggadocio,
substitutes dialogue and semiosis for imposition of conspiracies. Braggadocio is an
anxious and paranoid man, who questions historic events from the Gulf War and the
moon landing to Mussolini’s death. Braggadocio suspects that Mussolini escaped to
Argentina at the end of the Second World War, perhaps with the protection of the
papacy. As he joins one conspiracy with another, Braggadocio contrives stories about
bombings in Italy in the 1970s and 1980s, the kidnapping and murder of Aldo Moro,
and the death of Pope John Paul I. He suspects that Gladio, the Italian branch of
NATO’s stay-behind operations after the Second World War, attempted to destabilize
the political environment for a potential resurgence of Mussolini’s power. Through
Braggadocio, Eco emphasizes the role of questioning run amok, arrogantly unresponsive
to the world around us.
As Braggadocio finalizes the series of conspiracies for publication in Domani, Colonna
learns that ‘they’ve killed Braggadocio’ (Eco 2015: 169). Although Eco never identifies
the ‘they’, Colonna and Simei are confident that Braggadocio was too close to the truth
in at least one of his own conspiracies (2015: 171). As they navigate official investigations
surrounding Braggadocio’s death, Colonna and Simei work hard to conceal the theories
Braggadocio shared with them. Ultimately, Vimercate views Domani as ‘too dangerous’
and shuts the paper down (Eco 2015: 171). For Eco, the possibility to lie characterizes
semiotics (1976: 7). Semiosis, like dialogue, does not guarantee truth, just different
insights. Both semiosis and dialogue take us to the ‘not yet’, with neither offering assurance
about a pure truth. Interpretation is the house of semiotic and dialogic insight. For both
Eco and Gadamer, the home of truth dwells forever within interpretive questioning.
Eco and Gadamer are attentive to the revelatory: semiosis and dialogue, respectively.
Concrete meetings and understanding lead to interpretive insight and richness. Eco and
Gadamer do not suggest a universal truth. Their work pursues understanding within
temporality. Semiosis and dialogue are essential continuations of the story of the human
condition; they illuminate the ‘not yet’ and the revelatory. Eco and Gadamer are respectful
of the past, the shifting present and the anticipatory next, yielding a pragmatic display of
the significance of semiosis and dialogue in action.
HERMENEUTICS AND SEMIOTICS 139

­NOTES
1 The novel tells the story of William of Baskerville, a detective monk who investigates a series
of deaths in a medieval abbey. As he enters the infinite regression of semiosis, one sign leads
towards signification of another in the dialogic engagement between and among texts.
2 Palmer (1933–2015) was a renowned American scholar in hermeneutics who dedicated his
life to translating and interpreting Gadamer’s corpus. He received two Fulbright Fellowships
to study with Gadamer. The first of his six books was Hermeneutics: Interpretation Theory in
Schleiermacher, Dilthey, Heidegger, and Gadamer, published in 1969.
3 Grondin (1955–) was Gadamer’s pupil, specializing in the thought of Kant, Gadamer and
Heidegger. Translations of his work on hermeneutics, phenomenology and metaphysics
appear in twelve languages. His well-known books include Introduction to Philosophical
Hermeneutics (1991), Sources of Hermeneutics (1995) and Hans-Georg Gadamer: A
Biography (2003).
4 Ferraris (1956–) was a student at the University of Turin, taught in the tradition of Luigi
Pareyson and Gianni Vattimo (under whom Ferraris studied directly). Eco also attended the
University of Turin, completing his graduate work under the direction of Pareyson, who had
significant influence on Eco’s cultural aesthetics as a precursor to his semiotic project. From
this tradition, Ferraris examines hermeneutics and is particularly attentive to the interplay
of Nietzsche and Heidegger, as well as to the later work of Derrida, Foucault and Deleuze.
Additionally, Ferraris studied under Gadamer at Heidelberg University during the 1980s.
Ferraris also contributed to developing ‘New Realism’ as an orientation shared by continental
and analytic philosophers alike.
5 Susan Petrilli (1954–) is the seventh Thomas A. Sebeok Fellow of the Semiotic Society of
America. Augusto Ponzio (1942–) has been instrumental in introducing the works of Bakhtin,
Levinas, Marx, Rossi-Landi and Sebeok to an Italian audience. Their work includes over
three dozen articles and four books exploring the intersection of ethics and semiotics. They
rely upon dialogic scholars including Bahktin and Levinas as they frame semioethics as a
pervasive and unceasing call of responsibility to and for the other, grounded in the sign.
6 Petrilli’s well-known contributions on Lady Victoria Welby include Signifying and
Understanding: Reading the Works of Victoria Welby and the Signific Movement and book
Victoria Welby and the Science of Sciences: Significs, Semiotics, Philosophy of Language.
­7 Deduction relies upon generalizable principles and theories to understand specific cases,
whereas induction moves from observation of special cases to draw generalizable principles.
Peirce introduced abduction as a form of logic that relies upon deductive and inductive
reasoning to creatively respond to and anticipate insights.
8 The icon, index and symbol are three types of signs introduced by Peirce. The icon emphasizes
resemblance and likeness to the object. The index demonstrates a relation, a trace of the
object. The symbol is a conventional agreement of meaning oftentimes based upon culture.
9 Peirce identifies agapism, anancism and tychism as three modes of evolution in the cosmos.
In ‘Evolutionary Love’, he describes agapism as ‘evolution by creative love’, anancism as
‘evolution by mechanical necessity’ and tychism as ‘evolution by fortuitous variation’ (Peirce
1893: 188).
10 Eco’s notion of the limits of interpretation is central to his semiotic perspective and is the title
of a collection of essays published in 1990.
11 Petrilli and Ponzio are critical of the ‘weakness’ of Eco’s placement of ideology within his
theory of sign production and emphasize his failure to define ideology (2005: 321–2).
140 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

12 John Deely (1942–2017) was an American philosopher and semiotician. He was a research
fellow under Mortimer J. Adler at the Institute for Philosophical Research during the 1960s.
While studying the work of Jacques Maritain and John Poinsot, he met Sebeok. Deely helped
Sebeok found the Semiotic Society of America in 1975 and 1976. Important works of Deely’s
include Introducing Semiotic: Its History and Doctrine (1982), Basics of Semiotics (1990),
Four Ages of Understanding (2001) and Semiotic Animal (2010).
13 For Peirce, the representamen is the form of the sign and its potential to carry meaning as it
stands for something else. The object is the socially constructed and agreed-upon entity of
what a sign represents. Peirce’s notion of the interpretant is the action and sense produced by
the sign.
14 Luigi Pareyson (1918–91) completed his graduate work at the University of Turin in 1939,
where he later taught Eco and Vattimo. Pareyson’s work aims to understand interpretation
of aesthetic texts.
15 A parvenu participates in a wicked social game that includes (1) an invitation, (2) a rejection,
(3) another invitation, (4) another rejection and (5) a continuation of this pattern. The
parvenu witnesses impossibility in attempts to change class status (Arendt [1957] 2000).

­REFERENCES
Arendt, H. ([1957] 2000), Rahel Varnhagen: The Life of a Jewess, Baltimore: Johns Hopkins
University Press.
Arnett, R. C. (2017), Levinas’s Rhetorical Demand: The Unending Obligation of
Communication Ethics, Carbondale: Southern Illinois University Press.
Arnett, R. C. (2018), ‘Gadamer: Ethics and the Dialogic Character of Play’, Language and
Semiotic Studies, 4 (2): 19–35.
Bain, P. (2006), The Primacy of Semiosis: An Ontology of Relations, Toronto: University of
Toronto Press.
Benson, T. W. (1977), ‘A Theory of Semiotics’, Philosophy & Rhetoric, 10 (3): 214–16.
Buber, M. ([1947] 2002), Between Man and Man, trans. R. Gregor-Smith, New York:
Routledge.
Capozzi, R. (2015), ‘Umberto Eco’s Numero Zero: A J’Accuse of Forgeries, Lies, Conspiracies
and Bitter Truths’, Italica, 92 (1): 222–40.
Dance, F. E. X. (1977), ‘A Theory of Semiotics’, Quarterly Journal of Speech, 63 (3): 333–5.
Deely, J. (1982), Introducing Semiotic: Its History and Doctrine, 1st edn, Bloomington: Indiana
University Press.
Deely, J. (1990), Basics of Semiotics, 1st edn, Bloomington: Indiana University Press.
Deely, J. (1997), ‘Looking Back on A Theory of Semiotics: One Small Step for Philosophy,
One Giant Leap for the Doctrine of Signs’, in R. Capozzi (ed.), Reading Eco: An Anthology,
82–110, Bloomington: Indiana University Press.
Deely, J. (2001), Four Ages of Understanding: The First Postmodern Survey of Philosophy from
Ancient Times to the Turn of the Twenty-first Century, Toronto: University of Toronto
Press.
Deely, J. (2010), Semiotic Animal: A Postmodern Definition of ‘Human Being’ Transcending
Patriarchy and Feminism, South Bend: St Augustine’s Press.
Dockhorn, K. (1980), ‘Hans-Georg Gadamer’s “Truth and Method”’, Philosophy & Rhetoric,
13 (3): 160–80.
Eco, U. (1976), A Theory of Semiotics, Bloomington: Indiana University Press.
HERMENEUTICS AND SEMIOTICS 141

Eco, U. (1979), The Role of the Reader: Explorations in the Semiotics of Texts, Bloomington:
Indiana University Press.
Eco, U. (1984), Semiotics and the Philosophy of Language, Bloomington: Indiana University
Press.
Eco, U. ([1990] 1994), The Limits of Interpretation, Bloomington: Indiana University Press.
Eco, U. (2015), Numero Zero: A Novel, trans. R. Dixon, New York: Houghton Mifflin
Harcourt.
Ferraris, M. ([1988] 1996), History of Hermeneutics, trans. L. Somigli, Atlantic Highlands:
Humanities Press International.
Gadamer, H. ([1975] 2013), Truth and Method, trans. J. Weinsheimer and D. G. Marshall,
New York: Bloomsbury.
Gadamer, H. (1976), Philosophical Hermeneutics, trans. and ed. D. E. Lange, Berkeley:
University of California Press.
Grondin, J. ([1991] 1994), Introduction to Philosophical Hermeneutics, trans. J. Weinsheimer,
New Haven: Yale University Press.
Grondin, J. (1995), Sources of Hermeneutics, Albany: SUNY Press.
Grondin, J. (2003), Hans-Georg Gadamer: A Biography, Princeton: Yale University Press.
­Husserl, E. ([1954] 1970), The Crisis of the European Sciences and Transcendental
Phenomenology: An Introduction to Phenomenological Philosophy, trans. D. Carr, Evanston:
Northwestern University Press.
Lanigan, R. (1978), ‘Contemporary Philosophy of Communication’, The Quarterly Journal of
Speech, 64: 335–60.
Mueller-Vollmer, K. ([1985] 2006), ‘The Historicity of Understanding: Hans-Georg Gadamer’,
in K. Mueller-Vollmer (ed.), The Hermeneutics Reader: Texts of the German Tradition from
the Enlightenment to the Present, 256–92, New York: Continuum.
Palmer, R. E. (1969), Hermeneutics: Interpretation Theory in Schleiermacher, Dilthey,
Heidegger, and Gadamer, Evanston: Northwestern University Press.
Palmer, R. E. (1987), ‘Gadamer’s Hermeneutics: A Reading of “Truth and Method” by Joel
Weinsheimer’, Philosophy & Rhetoric, 20 (2): 135–8.
Peirce, C. S. (1893), ‘Evolutionary Love’, in P. Carus (ed.), The Monist: A Quarterly Collection
of Essays, 3, 176–200, Chicago: Open Court Publishing Co.
Petrilli, S. (1997), ‘Toward Interpretation Semiotics’, in R. Capozzi (ed.), Reading Eco: An
Anthology, 111–36, Bloomington: Indiana University Press.
Petrilli, S. (2009), Signifying and Understanding: Reading the Works of Victoria Welby and the
Signific Movement, Berlin: De Gruyter Mouton.
Petrilli, S. (2015), Victoria Welby and the Science of Sciences: Significs, Semiotics, Philosophy of
Language, New York: Transaction Publishers.
Petrilli, S. and A. Ponzio (2005), Semiotics Unbounded: Interpretive Routes through the Open
Network of Signs, Toronto: University of Toronto Press.
Schutz, A. (1967), The Phenomenology of the Social World, Evanston: Northwestern University
Press.
Simon, S. and U. Eco (2016), ‘“Numero Zero” Reprises Umberto Eco’s Fascination with
“Losers”’, NPR, 20 February, https://www.npr.org/2016/02/20/467468433/numero-zero-
reprises-umberto-ecos-fascination-with-losers
Warnke, G. (1987), Gadamer: Hermeneutics, Tradition and Reason, Stanford: Stanford
University Press.
142
 ­C HAPTER SEVEN

Translation Studies
and Semiotics
EVANGELOS KOURDIS AND RITVA HARTAMA-HEINONEN

INTRODUCTION
Despite the fact that Roman Jakobson linked translation practice to semiotics as early as
in 1959, and Gideon Toury introduced a novel approach to translation – the semiotics
of translation – in 1980, and regardless of the interdisciplinarity that characterizes these
research fields interested in cultural communication, the breadth and the depth of the
semiotic nature of translating have not thus far been extensively studied. Furthermore,
although distinguished semioticians like Charles Sanders Peirce ([1866–1913] 1931–58),
Victoria Lady Welby (1903), Juri Lotman (1990, 2005, 2009), Umberto Eco (1976, 2001,
2004) and Paolo Fabbri (2008) have carried out research on the concepts of translation
and interpretation, thereby emphasizing the translational and interpretative nature of
the semiosic processes involved in all understanding, their contributions have not yet
significantly influenced the field of translation studies.
Semiotics became misinterpreted and was pushed aside, seen as yet another complex
philosophical theory – which semiotics might be for many, though it definitely reaches
far beyond. Translation studies has recently adopted a more positive attitude towards
semiotics, especially when it was widely realized that multimodal cultural communication
with its ever-expanding digital spaces creates new data on the translatability of texts, which
are not necessarily verbal, but polysemiotic, multimodal or intermodal as well. During
this process, it has become more and more apparent that there is neither one monolithic
semiotics nor one conception of translation; instead, there are many complementary
approaches and schools that can be gathered under the respective conceptual umbrellas of
both semiotics and translation. This makes any effort to combine translation practice, sign
theory and translation theory an intriguing vantage point and a task worth attempting to
complete, and yet, finally, a complicated pursuit.
In this chapter, we will present mainly theoretical approaches to the research area and
field of knowledge called translation semiotics, and examine two key perspectives: first,
translation scholars’ positions on how semiotics can contribute to translation studies and,
second, semioticians’ positions on translation and translatability, which were, and are still
in part, influenced by three dominant semiotic schools:
1. Translation from the perspective of Peircean semiotics, which is the approach that,
with its fundamental concepts such as ‘semiosis’ and ‘interpretant’, created the basis
for a general semiotic theory of translation and interpretation. This theory has
144 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

gradually been constructed further, by testing and proving the applicability of the
cornerstones of Peirce’s semeiotic.
2. Translation from the perspective of the Paris School of Semiotics which provided
semiotic theory with important concepts, such as intertextuality (the original idea
though from Mikhail Bakhtin) and transposition (a designation also employed
in Jakobson 1959), which were passed on to translation, mainly through literary
theory and literary translation. Although this school has not focused on translation,
it endorsed the concept of the translatability of all semiotic systems.
3. Translation from the perspective of the Moscow-Tartu School of Semiotics
(nowadays also known as the Tartu School of Semiotics) which connected translation
to culture via the conception of semiotization, and adopted the stance that culture
is an inherent mechanism of translation, and vice versa.
Finally, we will examine some subfields where a semiotic theory can make a significant
contribution. These include phenomena, such as intericonicity, adaptation and transduction,
that emerge within the discipline of translation studies, and are considered to have a
semiotic core.
The purpose of this chapter is to show that these two interdisciplinary areas, translation
studies and semiotics, share to some extent a common ground, the application of which,
together with the recognition that there may also be shared interests, can primarily deepen
our idea of translating, and highlight the multi-dimensionality of translational practices in
cultural communication and beyond – in all communication.

SEMIOTICS FROM THE TRANSLATION

SCHOLARS’ PERSPECTIVE
We begin our overview of how translation researchers have approached semiotic vantage
points by placing the above-mentioned proposal by Jakobson at the centre of attention.
His proposal is not at the centre within either semiotics or translation studies today,
but it will function here as a link between the predecessors and the interpreters with
their echoing or non-echoing stances. In this hub-like function, Jakobson’s proposal may
explain the complex yet evolving relation between those who approach translations and
translating from a semiotic angle and those whose approach is translation-theoretical
(see, e.g., Hartama-Heinonen 2015, 2008: 29–80). The following may also explain the
growing awareness of a shared ground and shared interests, or interdisciplinary unification
that Jakobson appears to have pushed forward within translation studies and semiotics.
Jakobson, the linguist and semiotician, introduced in 1959 his pioneering division
of translation into the intralingual, interlingual and intersemiotic types, describing how
verbal signs can be interpreted within verbal or non-verbal sign systems: within the
same verbal sign system; between two different verbal sign systems; between a verbal
and a non-verbal one. The three-part typology is language-centred (with the choice
of the verbal as the source system), yet sign-theoretical (signs, sign systems), featuring
simultaneously a linguistic and a semiotic line of approach and respecting both of them.
This proposal constitutes a landmark the importance of which has been evaluated and
re-evaluated more prominently since the 1980s. When not directly ignored, his proposal
has been continuously referred to and revisited: sometimes accepted as such, sometimes
criticized and rejected, and sometimes reread, updated and modified to suit the changing
TRANSLATION STUDIES AND SEMIOTICS 145

spatiotemporal constraints, contexts and purposes. This applies even today, with some
exceptions: see, for example, Sütiste (2008) on Jakobson in linguistic, semiotic and
translation-theoretical reference works; note also Toury’s (1986) rethink of the typology,
resulting in the dichotomy of intrasemiotic and intersemiotic types, where the intrasemiotic
forms consist of intrasystemic (such as translating within a language) or intersystemic
(between two languages) translation; or Gottlieb who advocates a polysemiotic approach,
attesting that Jakobson’s contribution can, today, be passed without comment (2017, cf.
also [2000] 2004).
Jakobson’s article is both innovative and seminal, and heralds a semiotic turn in
translation research, a circumstance that has not been fully comprehended despite the
decades of reassessment. The reason is obvious: interlingual translation, Jakobson’s
translation proper, has been and is still for many translation scholars the prototype of
translation and the only reasonable focus of interest; their translation-theoretical studies
continue to strengthen the interlingual translation type. To suggest a semiotic turn might
be ambitious, the acceptance being far from self-evident. In The Turns of Translation
Studies: New Paradigms or Shifting Viewpoints? Mary Snell-Hornby mentions Jakobson
for introducing the Peirce-inspired typology, and she also observes the novelty and
applicability of Jakobson’s proposal when studying modern translation genres such as
intralingual subtitling or intersemiotic multimodal translation (2006: 21). For Snell-
Hornby, this is apparently all that can be said about Jakobsonian or, for that matter, other
semiotic influences; they have hardly launched a semiotic turn or paradigm.
Translation studies appears to be somewhat selective in its approach to Jakobson.
While interlingual translation between two natural languages is acknowledged, and
intersemiotic translation from verbal signs to non-verbal ones (and vice versa), such as
when a novel is filmed, has found more adherence, intralingual translation has not until
recently received due attention or use as a term. For Jakobson, the intralingual type
comprises choosing synonyms and paraphrases within the same language (1959: 233).
But in a modernized version, intralingual translation can manifest itself together with the
intermodal, such as in speech-to-text interpreting or subtitling for the deaf and hard of
hearing, when the signs in one mode are rendered with signs in another; spoken and aural
with written and visual. In this intralingual-intermodal-intersemiosic type, the transfer
mechanism inherent in translation is harnessed to serve accessibility and barrier-free
communication – observe also adaptations and translations into easy language or texts in
such varieties as inclusion-enhancing clear administrative language.
According to Karen Zethsen (2009), we ought to study more extensively the
similarities and differences between interlingual and intralingual translation to refine our
idea of translation, even towards a more open-ended definition. There are also critical
voices. Brian Mossop (2016) maintains that despite the shared features, variance-oriented
intralingual activities – rewording such as plain writing, paraphrasing or editing – should
not be included in the scope of translation studies: they are so different from interlingual
translating conceived as equivalencing; invariance is a criterion that neither intersemiotic
translation might meet (Mossop 2019). Jakobson’s translation typology, however,
requires us to broaden our notion of translation beyond the threshold of invariance, and
do it in accordance with Peirce’s fundamental idea which originally inspired Jakobson:
that signs become signs through translating (1903: CP 5.594).
We may gain some insight into the direct and indirect role semiotics has recently played
when we examine what semiotic(s) appears to be for translation scholars, or what sort of
context may prompt them to apply semiotics, whatever it may mean in individual studies.
146 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

A search with the entry semiotic in the Translation Studies Bibliography (Gambier and
van Doorslaer 2004–) results in 100 abstracts from 2010 to 2019. We base the following
outline on some fifty explicit hits in these abstracts, but our account is definitely not
exhaustive, since the bibliography material presents no balanced cross-section. Furthermore,
contributions included in the bibliography, qualifying as translation studies, can represent
other disciplines. In any case, this account presents some signposts to areas in which semiotic
ideas gradually establish foothold.
First, semiotic(s) most often occurs in references to semiotic resources, that is, together
with mention of Halliday, multimodality, sociosemiotics, meaning-making, and semiotic
modes and codes. Second, there are some genres of specialized translation that apparently
have a close connection to what can be called the semiotic nature of the phenomenon in
question. These include audiovisual translation (subtitling, surtitling, audiodescription or
adaptation). Translation of comics, picture books and advertisements also belongs to this
group as well as song translation and theatre translation, genres of intersemiotic nature.
Third, some abstracts explicitly refer to a certain semiotic branch, a particular semiotic
school or certain semioticians. Fourth, we encounter some mentions of linguistic as opposed
to semiotic (culture, transfer) – so language is not always considered as a semiotic system.
In an increasingly cross-semiotic world, intersemiotic translation has gained
wide currency. For many, this type constitutes the noteworthy novelty in Jakobson’s
contribution and is actually its eye-opening core finding: translation occurs between
verbal systems, but also between verbal and non-verbal sign systems and, in addition, can
proceed from a non-verbal to a non-verbal one as well. The step from the dominantly
interlingual to the intersemiotic and multimodal facet in translational communication
means a definite step towards semiotic thinking.
The world of professional translating – language industry, language service providers
and freelance or in-house translators – is facing new challenges, and so is Translation
Studies. We see changes in its research objects: products, processes, actors and contexts
involved. These include a gradual shift from studying translation products towards
studying translation processes reaching from the cognitive aspects of human translation
to the development and the growing use of translation technologies. The changes
also concern new emphases and focuses (Translator Studies instead of Translation
Studies; user-centredness) and non-profit endeavours (non-professional translation
and interpreting; crowdsourcing). Despite the developments in translation practice,
semiotics will assumedly preserve its place in the research contexts of today, and even
Jakobson’s typology will exist as a basic yet implicit framework for what translation
scholars focus on.

THE TRANSLATION SEMIOTICS RESEARCH FIELD

To explore the division of labour between fields and disciplines is not easy. This applies to
Translation Studies in relation to Semiotics (of Translation) as well. Semiotics, the general
sign theory, also characterized as the reuniting ‘master science’ (Ketner 1998: 306) and the
language connecting scholars across disciplines constitutes the foundation and umbrella
for all disciplines (in practice often integrated into disciplines). Translation Studies in turn
positions itself as an interdiscipline, explicitly since 1994 (see Snell-Hornby [1988] 1995:
133, 2006: 70–2). Its current level of interdisciplinarity is reflected in its methodology
which as expected is not its own, but borrowed from nearby disciplines, such as semiotics,
modified and refined, thus both adopted and adapted.
TRANSLATION STUDIES AND SEMIOTICS 147

To illustrate this point, in the article ‘Semiotic Modalities in Translation Causality’

(2002), Andrew Chesterman, one of the leading Translation Studies scholars, draws on
A. J. Greimas’s idea of modalities and analyses the work of a translator as reflections
of these modalities, relating them to causality and translation effects. This Greimas-
inspired approach to translatorial activities contributes, no doubt, to Translation Studies,
but does it simultaneously, de facto, contribute to Semiotics? Or should it be classified
more cautiously among semiotic translation theory, semiotic translation studies, semiotic
approaches to translation (studies), semiotic translation research, semiotics of translation,
semiotics and translation, semiotranslation, or translation(al) semiotics – designations
denoting research that combines semiotic ideas and translation? Instead of discussing
issues like what can count as semiotics of translation (Semiotics of Translation), or
whether this discussion is relevant at all, we will concentrate on how translational matters
are approached from two disciplinary angles and emphases.
In the 1980s–90s, when Translation Studies took its first steps towards a success, there
were, according to Snell-Hornby, several factors that increased its viability: enthusiastic
scholars with merging interests, disciples (scholars-to-come), conferences, the European
translation scholars’ society, new journals, expanding translator education, a series of
‘turns’ and research innovations that later on proved to be landmarks, such as skopos
theory ([1988] 1995: 133–5, 2006: 69–70).
Translation semiotics cannot present similar achievements, if this branch is supposed
to be to some extent comparable to an already-established discipline. Ritva Hartama-
Heinonen (2008: 70) describes the status of this subfield as vague, when compared to such
acknowledged semiotic branches as semiotics of culture, musical semiotics or semiotics of
theatre. There are scattered scholars and translation-semiotic agendas and pursuits that
apparently do not aim at unified views; only one international conference series dedicated
expressly to semiotics and translation, SemTra, with thirteen conferences thus far (Imatra,
Finland, and Kaunas, Lithuania, 2000–17); no own society nor a journal, instead special
issues of journals (e.g. Athanor, Parallèles, Punctum, Semiotica, Sign Systems Studies, Signata,
Versus), and what is particularly noteworthy, visibility-promoting state-of-the-art overviews,
as our chapter, in semiotic and translation-theoretical encyclopaedias, reference works
and handbooks (from Toury 1986 to Kourdis 2015 and Gottlieb 2017); mostly separate
university-level courses; and finally, one particular innovation, semiotranslation. To sum
up, the infrastructure of this semiotic branch is despite the efforts still under construction.
Eero Tarasti (2000: vii, 3–4) characterizes the development of general semiotics with
three generations: the first-generation semioticians represent classical semiotics from Peirce,
Ferdinand de Saussure, Claude Lévi-Strauss and A. J. Greimas to Thomas A. Sebeok; the
second generation consists of Michel Foucault, Roland Barthes, Julia Kristeva, Eco and Jacques
Derrida, among others; the third is neo-semiotics, actually Tarasti’s existential semiotics which
aims at renewing semiotics to serve our millennium and its new challenges. When outlining
the following short description of the advent and evolution of translation semiotics, we have
combined Tarasti’s proposal of generations with the ideas and designations employed for
instance, by Snell-Hornby (2006: 5, 20, 30, 47; based on Radnitzky [1968] 1970).
The first generation, Peirce and Welby – translation semioticians avant la lettre – are
the precursors who in their writings (classics) laid the foundations for the generations
to come. The second generation are the pioneers: Jakobson, Lotman, Anton Popovič1
and Toury, semioticians who reflected the relation between translation and semiotics,
and prepared the way for emerging translation-semiotic thinking. The third generation
consists of Eco, Dinda L. Gorlée, Susan Petrilli and Peeter Torop, the acknowledged
148 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

first masters who have during years and decades devotedly constructed their semiotic
approaches to translation and become forerunners in and through their programmatic,
leading-edge research. We might also categorize Pirjo Kukkonen and Douglas Robinson
among the third-generation scholars, as widely published and consequent in their
individual contributions and applications of semiotic insights. The fourth generation are
disciples and potential masters-to-be: Hartama-Heinonen, Evangelos Kourdis, Bruno
Osimo, Ubaldo Stecconi and Elin Sütiste might qualify as candidates. Their duty is to put
the masters’ work to the test, and challenge the tradition.
In this partly translation-studies-inspired reading of translation semiotics, we have
observed slow semiotic progress and movement. This picture must be completed and
anchored further. Translation semiotics emerges from the mainstream schools (Prague,
Paris, Moscow-Tartu) as well as from the theoretical and empirical (applied) methods and
approaches of its mother discipline. Translation semioticians have drawn on philosophical
(Peirce, Morris, Bakhtin), linguistic (Hjelmslev, Jakobson), cultural (Lotman), structuralist
(Saussure, Greimas) and poststructuralist (Barthes, Kristeva) insights. Impacts from
several main branches and attitudes are also manifest: semiotics of culture, of language,
of literature and of theatre, legal, musical and visual semiotics, bio- and zoosemiotics, and
semioethics, to mention a few. Synergy can be created when studying translation from
several semiotic perspectives (for instance, from a biosemiotic angle and complement this
approach with a Peircean one, as in Marais and Kull 2016).
To understand the mentioned still vague situation and the scholars’ preferences,
some observations, definitely not generalizations, are appropriate. Very few translation
scholars appear to have a permanent interest in exploring translation with the explicit
help of semiotics proper. In addition, very few semioticians have a permanent interest
in translations and translating, while keen translation semioticians are not necessarily
interested in what Translation Studies has achieved, nor always committed to a certain
semiotics but may act in their pursuit of knowledge eclectically, also using mixed
methods. Furthermore, even though English is an academic lingua franca, you ought
to know major languages like Chinese, or minor languages like Finnish and Swedish
(to read, e.g., Kukkonen 2014), if you aim at getting an up-to-date, overall view of the
translation-semiotic field of study.
To succeed in its mission, translation semiotics has to be competitive and bring added
value by addressing questions that no other field has on its agenda, and to justify its
further existence, create new theoretical and empirical knowledge and, thereby, enhance
our understanding of translational phenomena. The strength of semiotics does not lie
so much in the versatility of its objects of study than in its flexibility: like a chameleon,
semiotics adjusts itself to its surroundings and, while recontextualizing itself, does not
lose its sign-theoretical characteristics. What this means in a translation-semiotic context
and inquiry (and probably for semiotics applied) becomes evident below through the
presented achievements.

­TRANSLATION FROM THE PERSPECTIVE

OF PEIRCEAN SEMIOTICS
Semeiotic, Peirce’s uncompleted semiotic project, has been characterized as interpretive/
interpretative, processual or pragmatic/pragmatistic semiotics. These are designations that
are employed to highlight possible approaches to and facets of Peirce’s non-structuralist
TRANSLATION STUDIES AND SEMIOTICS 149

semiotic thought, according to which the meaning and the nature of a sign are revealed
through successive interpretative (translative) efforts, in a practically endless sign process
called semiosis, resulting in the growth of knowledge through pragmatic consequences
and effects, interpretants.
Peirce’s general theory of signs is not so much about signs (icons, indices and symbols)
or sign systems as it is about the ubiquitous two-sided semiosis that embraces both
meaningful dynamic sign action and sign interpretation, that is, a sign functioning as
a sign and determining an interpretant of itself, or more precisely, creating a triadic
interrelation between the sign, its object (which empowers the sign) and interpretant.
This interpretant, a translation of its preceding sign, is a sign as well, and can, through
its own action, determine a new interpretant, or even a chain of interpretants. This is
how the meaning of the sign (its object) is gradually unfolded in a semiotic and semiosic
action: the sign is translated into, renders itself as, another sign (1903: CP 5.594), an
interpretant which is an equivalent or perhaps a more enhanced version (c.1897: CP
2.228), and a future-oriented interpretation, translation, inference, effect. This is why
any semiosis is inherently translational semiosis, with interpretants being translations of
antecedent signs. Since translating goes from sign to sign and occurs between signs, the
process is intrinsically intersemiotic.
From the viewpoint of semiosis, the world manifests itself as a universe of signs and
sign translation. When we add to this the notion that the involved signs can be verbal or
non-verbal (verbal language being only one possible sign system), we have produced a
simplified Peircean account of translation(al signification and communication). Peircean
semiotics can namely be approached as an unwritten general theory of translation as
sign translation (including sign interpretation and sign production); often cryptic and
fragmentary yet constituting the foundation and framework for Peircean translation
semioticians to build on, theorize about, apply, modernize and elaborate.
Translation appears to amount to semiosis and semiosis to translation (Gorlée 1994:
226–7), with translation being realized in and through the universal mechanism of
semiosis and vice versa. Furthermore, translation emerges wherever we encounter signs
and semiosic processes, that is, in the semiobiosphere (Ponzio 1999–2000: 5–6). For
many, these starting-points are too vague, obscure and open-ended. But it is also evident
why Peirce’s semiotic thought has also inspired scholars to explore translation and
interpreting semiotically.
Dinda L. Gorlée, Douglas Robinson and Janice Deledalle-Rhodes were among the
first in the 1980s who saw how Peirce’s sign theory can enhance translation research.
Even though Gorlée, the semiotician, translation scholar and Peirce scholar, has since
1986 published exceptionally widely on translation semiotics, her Semiotics and the
Problem of Translation (1994) is even today the most influential single introduction to
Peircean translation studies, not least because she was the first to apply Peirce’s thought
to translation extensively. Gorlée’s proposal for a Peirce-based approach to translation
as practice and theory is realized in her interpretation of translational semiosis, semio(-)
translation. Gorlée has gradually integrated translation-related interpretations of Peirce’s
central concepts, ideas and often triadic distinctions into semiotranslation theory. These
include semiosis, types of signs and interpretants, the role of the object, signhood, the
ontological categories, the modes of being, truth in semiosis, abduction, induction,
deduction in translatorial reasoning, etc. Semiotranslational research is highly theoretical,
but Gorlée has since the 2000s also developed applied semiotranslation, such as Peirce-
based analyses of vocal translation and intersemioses and partly Peirce-inspired readings
150 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

of Wittgenstein (see Gorlée 2012 and the references therein). Semiotranslation is no

longer exclusively Peirce-centred, but draws on other semioticians as well.
Semiotranslation is Gorlée’s coinage, but the term has been used without commitment
to Gorléean semiotranslation, for instance, by Goethals et al. (1999–2000) when they
demonstrated the practical applicability of Peircean semiotics to multimedia, biblical texts
and comics, the black box, and reasoning. Of these researchers, Robinson has published
widely on translation and semiotics, also on Peirce, but selectively. His Becoming a Translator
([1997] 2003) is a book with applications of Peirce’s three types of reasoning. Robinson
has also elaborated other Peircean concepts, and re-interpreted semiotranslational views.
Stecconi (e.g. 2004, 2007 and the references therein) advocates semiotics and Peirce’s
sign theory, since the latter in its generality reassesses many modern ideas of translation.
Stecconi’s (2004, 2007) contribution to Peircean translation semiotics is what he calls
the Foundation of translation, characterizing the conditions to translation semiosis with
the triplet of similarity (First), difference (Second) and mediation (Third) (note also the
triplet of conditions, events, norms/habits). For him, translation represents a specific type
of semiosis. Neither does Eco accept the above proposal that interpretation is translation
(and semiosis), since they are not coextensive (2001: 67–71; cf. also Vassallo 2015):
Jakobson drew on Peirce when associating interpretation with translation, but did
not mean it literally, only that it is worth approaching interpretation and meaning as
translations. As we can see, Peirce can be read in several ways, and his followers do not
necessarily agree on the interpretations.
Hartama-Heinonen (e.g. 2008, 2015) has explored the status of translation semiotics
within both semiotics and Translation Studies, and particularly Peircean semiotranslation.
Her main focus is though on the art of inferring from signs in translation and inquiry, or
Peirce-based abductive translation (studies), probing such claims as ‘reasoning is (sign)
interpretation is translation is semiosis’ and ‘translation is translation and beyond’ (2008:
61, 179, 299). We will also mention the Peircean contributions of Susan Petrilli and
Cécile Cosculluela, and add that there are only few, if any, ‘full-time’ Peircean translation
semioticians, and many of those who are active may draw on Peirce eclectically and
sporadically.
To provide a wider picture of Peircean translation semiotics, we end this section
with three recent studies, and ask, perhaps provocatively, whether they point to future
tendencies.
João Queiroz and Daniella Aquiar (2015) discuss semiosis and iconic processes, when
developing their model of intersemiotic translation, which they illustrate with an example
of a transcreation from text to dance. Would it be time to give up the ambitious and
extensive theoretical pursuits and begin to scrutinize more restricted topics empirically?
Adopting Gorlée’s term semiotranslation and also applying Peirce, Kobus Marais and
Kalevi Kull (2016) call for an interdisciplinary dialogue between biosemiotics and
translation studies, since that can lead to a conceptual renewal and promote mutual
understanding of what constitutes translation. Could biosemioticians succeed in what
Peircean translation semioticians have not, in convincing that translating definitely
reaches beyond language, to the whole semiobiosphere? Sergio Torres-Martínez (e.g.
2018, 2020) has developed his idea of semiosic translation, a multi-disciplinary approach
that fuses translation, Peircean semiotics, abduction, sign and signification, and certain
Wittgensteinian ideas. He appears to resume topics already studied, but could it be so
that the future of the field emerges from the abductive power and synergy of similar
symbioses, parallels and combinations?
TRANSLATION STUDIES AND SEMIOTICS 151

TRANSLATION FROM THE PERSPECTIVE OF

THE PARIS SCHOOL OF SEMIOTICS
This section presents the positions of semioticians of the Paris School of Semiotics (also
known as Semio-linguistic Research Group) concerning semiotics and translation in an
effort to reveal their approach on translation in a period when translation was not an
autonomous field of research and it was studied under a linguistic prism. This broadening
was done thanks to the elaboration of concepts like signification, meaning, intertextuality,
transposition, synergy, fusion, syncretism, redundancy, anchoring function, significant
concepts provided by scholars of the Paris School of Semiotics. Even though this semiotic
school had no direct links to translation studies, we believe that it provided concepts
which were adopted by translation studies scholars and contributed to the autonomy of
translation studies, thus escaping the predominance of the semiotic system of language,
although it continues to be considered by semioticians as the primary semiotic system for
the study of culture.
From the 1980s, some semioticians began to study translation more systematically
through semiotic theories. This current, known today as Semiotics of translation,
contributed, in our opinion, significantly to the revival of translation studies, and in this
effort the contribution of the French school is evident. As Joseph Courtés mentions,2
what interests the Paris School of Semiotics is ‘rather knowing what is happening “under
the signs” or “between the signs”, what which is the basis of their mutual relationships’
(2001: 123). The translatability of signs also belongs to this context. The subject’s
activities and attitudes towards the object, as well as towards its modalities (see Tarasti
2017: 44), gave to the notion of translatability of semiotic systems a central position in
the discourse of the Paris school. It should be noted that, according to the Paris school of
Semiotics, ‘translatability stands as one of the fundamental properties of semiotic systems’
(Greimas and Courtés 1982: 351), and, they continue, ‘translation, comes in between
the existential judgment “there is meaning” and the possibility to say something about it.
“To speak of meaning” is both to translate and to produce signification’ (1982: 351). We
adopt this position considering that the first attempts at a certain widening of the notion
of translation were made by semioticians who are classified within the Paris School of
Semiotics or are considered close to it.
Jakobson (1959), a leading figure of the Paris School, proposed a typology of translation
and with the introduction of the concept of intersemiotic translation or transmutation,
he clearly linked, for the first time, translation to semiotics. The concept of intersemiosis
as the act of intersemiotic translation was further promoted by Julia Kristeva (1969),
who analysed not only the notion of intertextuality, an important concept in the study of
intersemiosis, but also the notion of transposition (Kristeva 1974), a term also proposed
by Jakobson (1959: 238). In our view, Jakobson is ‘scientifically cautious’. He takes a
careful step but avoids insisting on his analysis on intersemiotic translation. His scholarly
background seems to confirm this observation. Jakobson is no longer concerned with
the phenomenon of intersemiosis although he has included polysemiotic/syncretic
systems, as well as non-linguistic systems, in the translation process. This appreciation
of non-linguistic signs is in keeping with the need to include them in the study of daily
communication.
For Greimas, the founder of the Paris School of Semiotics, a translation that is not
properly linguistic is also possible. As Greimas remarks, ‘any signifying ensemble of
a different nature from that of the natural language can be translated, with more or
152 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

less exactitude, into any natural language: for instance, painting and its translation by
pictorial criticism’ (1983 [1966]: 11). However, the fact that for Greimas language always
remains the primary semiotic system the participation of which in the translation process
is considered to be primary prevents him from deepening his above-mentioned position.
It should be noted that all these proposals come from linguists who very early noticed
the importance of non-verbal communication, at a time when mass communication was
not as widespread as today. It is also interesting that the legitimacy of the exclusive use of
linguistics for the study of translation is refuted by another Paris school, that of the Paris
School of Translation. As Marianne Lederer mentions ‘whatever the merits of linguistics
[. . .] it cannot claim to explain the complexity of translation’ (1994: 92). We will see
later that this complexity included a series of semiotic facts which had already influenced
linguistic and literary studies before studying them for translation.
The concept of intertextuality has been known in European thought since the 1920s
thanks to the translinguistic analysis of Mikhail Bakhtin and it was disseminated in France
thanks to the studies of Julia Kristeva (1969). As Greimas and Courtés note, ‘covering and
expanding, without contradicting it, the concept of intertextuality, that of intersemiosis
is imposed, in semiotic theory, in the name of respect for immanence. Its construction
is made possible by the distinction between the two types of macrosemiotics which are
semiotics of the natural world on the one hand, and natural languages on the other’
(1986: 119). Just like Jakobson (1959), Greimas and Courtés (1986) could not imagine a
phenomenon of intersemiosis which excluded the language system.
But how did this transition from one key semiotic concept to another come about
so early if one takes into account that even today intersemiotic translation is a concept
carefully studied and not so widely used in semiotic and translational studies? In our view,
it was Roland Barthes’s essay ‘Rhétorique de l’image’ (1964), based on the notion of
synergy of semiotic systems, which laid the foundations for the study of intersemiosis. He
showed that it is a selective and intentional coexistence where different signs can coexist
and say the same thing, even at the connotative level. But how is this concept defined?
Parret defines synergy as ‘[. . .] the coordinated action of several systems [. . .] from which
results the accomplishment of a function, the execution of a movement’ (2006: 121).
Parret adds that ‘synergy [. . .] as a harmony of trends oriented by the same purpose, must
be thought of in the world of fusion’ (2006: 130).
The notion of fusion is not a new term, since there is also the related notion of
syncretism. Jakobson argues that ‘the study of communication must distinguish between
homogeneous messages which use a single semiotic system and syncretic messages based
on a combination or merger of different sign patterns’ (1971: 705). It is this synergy/
fusion in visual communication that Barthes (1964: 43) became aware of when he carried
out the first semiotic study of advertising, and he wondered if the image duplicated certain
information from the text, through a phenomenon of redundancy, or the text adds new
information to the image. It does not escape our attention that, when Barthes (1964)
deals with the anchoring function in the case where a ‘legend’ accompanies an iconic
message, the linguistic message (the legend) can share the same load of information with
certain iconic elements. Thus, two different signs can say the same thing which classifies
this semiotic fact in the field of translation. In other words, the fact of intersemiosis is
produced in order to combat the ambiguity of the image.
In our view, the notions of synergy, fusion, syncretism, redundancy have paved the
way for the concept of intersemiosis. The definition of intersemiosis by François Rastier
and Carine Duteil-Mougel, ‘intersemiosis defines an interaction between system of signs’,
TRANSLATION STUDIES AND SEMIOTICS 153

relates to the contribution of the Paris School (2009: 215). As early as 1973, Jakobson
emphasized that ‘there is a variety of semiotic systems that include various substitutes for
spoken language’ (1973: 28). Jakobson clearly links polysemiosis to translation, citing
that all forms of communication ‘are accompanied by certain verbal utterances or other
semiotic manifestations or both [. . .] they are not verbalized, they are all verbalizable, that
is to say into verbal messages, or, at least, lower’ (1973: 34). In our view, this position,
which harmonizes with Barthes’s positions, has advanced visual semiotics which, in turn,
has developed and disseminated the concept of intersemiosis.
The Paris School of Semiotics greatly influenced the Italian School of Semiotics. Eco
(2001), an iconic figure in Italian semiotics, wrote an important book on this field. For
Eco (2001, 2004), translation is based on negotiation, a process through which one
renounces something to gain something else, and in the end, those involved should end
up with a sense of reasonable and mutual satisfaction. Eco also made two further points
that are of particular interest. First, Eco suggests a new definition for intersemiotic
translation, where intersemiotic translation is to be understood as an adaptation (2001:
118, 125). Second, Eco argues that interpretation is not always synonymous with
translation, since the universe of interpretations is much larger than that of translation
(2001: 73).
Paolo Fabbri, an equally important figure in Italian semiotics, diverges from Eco in
both the above points. Fabbri posits that every semiotic system can be translated into
another semiotic system (2008: 161). Thus, novelistic writing can be translated into film
for television or cinema, and when instances of untranslatability occur, it is a question of
changing strategy in order to allow for every fundamental element of the source text to
come through. Fabbri seems to accept the view that interpretation is a form of translation
(2008: 112).
The positions and research proposals mentioned above only point to some of the
topics of the Semiotics of translation, an innovative approach to translation, different
from the classic approach that accompanied translation in the past. In our view, to the
Semiotic School in Paris we owe the widening of the notion of translation. From this
perspective, the influence of the Paris Semiotic School is important. Even if a large number
of translators limit the contribution of semiotics to the presence of the iconic messages
which accompany the written text, the semiotic notions presented in this study can help
in the cultural understanding of the function of translation, a phenomenon which seems
to be primarily verbal, but not always.

TRANSLATION FROM THE PERSPECTIVE OF THE

MOSCOW-TARTU SCHOOL OF SEMIOTICS
The Moscow-Tartu School established a theoretical framework for the semiotics of
culture under the influence of the Russian semioticians, and in particular Mikhail
Bakhtin. Eminent scholars belonging to the Moscow-Tartu School of Semiotics are Juri
Lotman, Vjacheslav Ivanov, Boris Uspenskij, Alexander Pjatigorskij, Vladimir Toporov,
Peeter Torop, Kalevi Kull, to mention but a few. Juri Lotman is considered the most
eminent scholar of the Moscow-Tartu School. His work on translation is far less known
than the rest of his work, despite his belief that ‘the instrument of semiotic research is
translation’ (1990: 271). The School broadened the notion of translation based on a series
of notions such as cultural text, semiosphere and cultural boundary, cultural equivalence,
154 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

(hetero)communication and autocommunication, and introduced the notion of cultural

translation in semiotic studies.
As concerns the concept of cultural text, for the Moscow-Tartu School, language is not
used in the sense of a natural language but in the ‘specifically semiotic sense’ described
by Uspenskij et al. ([1973] 2003: 297) which is applied ‘also to any carrier of integral
(“textual”) meaning – to a ceremony, a work of the fine arts, or a piece of music’.
According to Lotman (1990: 125), cultural texts ‘relate to each other along the spectrum
which runs from complete translatability to just a complete mutual untranslatability’.
The disengagement of the concept of the text from the linguistic text had already been
proposed in the field of semiotics by Roland Barthes in the 1960s, when Barthes (1964)
spoke not only of linguistic message in advertising, but also of uncoded iconic (image) and
coded iconic or symbolic message.
A new concept also introduced into semiotics by Lotman is the concept of semiosphere,
which is used quite often in semiotic studies for translation. Lotman defines the
semiosphere as ‘that same semiotic space, outside of which semiosis itself cannot exist’
(2005: 208–9). In other words, the semiosphere includes all these signs that belong to
a cultural system and which seem to be not only a first, but also a second set of signs,
that is, they are a record of cultural information. For Lotman, translating information
across these semiospheric boundaries creates meaning, creating new information (2005:
215). This new information could be culturally equivalent to an information of another
cultural system. The notion of equivalence was also proposed by Jakobson (1959). For
Lotman, ‘at the basis of every act of exchange lies the contradictory formula, “equivalent
but different”: the first part of the formula makes an exchange technically possible and
the second part makes it meaningful in content’ (1977: 96). The translated semiosphere
has to appear more or less intact in its translated representation and this semblance of
formal equivalence, i.e. equivalence of signifiers of the two semiotic systems, is the job
of this intermediate semiosphere, the sphere where informational loss is, as it were,
manufactured. This notion of an intermediate sphere is useful, even though it creates an
ad absurdum infinite creation of semiospheres for it accounts for the fact that semiosis
occurs within systems of meanings with different, competing, cross-destructive or cross-
complementary (even self-destructive) semantic values.
Scholars of the semiotics of culture have often underlined the contribution of the
notion of equivalence in cultural translatability, as a function of controlling informational
loss. According to Uspenskij et al., ‘translation from one system of text to another always
includes a certain element of untranslatability’ ([1973] 2003: 311).3 Untranslatability
is detected in the function of equivalence from one semiotic system to another.
Translation will draw its informational material from those elements that the translated
and the translatable semiospheres have in common and to this task the contribution of
intersemiotic translation seems very useful.
For Lotman translation is connected to human thought. Lotman (1990: 43) claims that
‘an elementary act of thinking is translation’, explaining that ‘the elementary mechanism
of translating is dialogue’. Lotman (2009: 6) also adds that ‘[. . .] even the nature of
the intellectual act could be described in terms of being a translation, a definition of
meaning as a translation from one language to another, whereas extra-lingual reality may
be regarded as yet another type of language’. If extra-lingual reality is another type of
language, then it follows that intersemiosis is a wider phenomenon, as well as cultural
translation in the cultural space, as already Eco (1976) aptly has claimed.
TRANSLATION STUDIES AND SEMIOTICS 155

This dialogue could be realized in the frame of heterocommunication (also known as

communication) or autocommunication (Lotman 1990). In Lotman’s autocommunication
the subject is transmitting a message to him- or herself. It is worth mentioning that
translation as communication and autocommunication is also approached by Sütiste
and Torop who mention that ‘as the concept of translation broadens, it approaches the
concept of understanding – understanding through translation and understanding the
translation itself. To understand different kinds of translation means to understand both
communication and autocommunication processes’ (2007: 202).
The most important contribution of the Moscow-Tartu School is the correlation of
the concept of culture with the concept of translation. This school not only linked culture
with translation, since ‘[culture] is not only the place where meanings are born, but the
space in which they are being exchanged, “transmitted” and seek to be translated from
one cultural language into another’ (Toporov 1992: 30) but went one step further than
the French school by linking the concept of text/semiotic system with translation. They
argued that a message can be considered a text when it is open to later translations and
interpretations (Lotman and Pjatigorskij 1969).
An important contemporary figure of this school is Peeter Torop (1998, 2002, 2003,
2007, 2008, 2012) who sees the concepts of translation and culture as overlapping. As
Torop (2002: 603) states, ‘in the discipline of the semiotics of culture it comes naturally to
say that culture is translation, and also that translation is culture’, in the sense that meaning
is always something to be transferred from one locus to another, and this metaphoric act
is at once an act of translation and an act of culture. Torop’s (2003) introduction of the
concept of total translation is of particular importance. As Torop (2003: 271–2) explains,
total translation describes different types of textual communication in culture, not only
interlingual translation, but metatextual, intratextual, intertextual and extratextual
translation as well. The change of the text’s ontology in contemporary culture due to the
possible simultaneous existence of various forms of the same text in different media and
discourses allows us to regard culture as a process of intersemiotic translation.
An interesting approach and expansion of the Lotmanian theory constitutes the
openness to translation by the Lund School of Semiotics and mainly by Göran Sonesson.
Sonesson (2014a) approaches translation as a double act of communication and proposes
the replacement of the notion of ‘translation’ with the notion of ‘interference’ in the three
kinds of translation offered by Jakobson in 1959. Moreover, Sonesson (2014a, 2014b)
argues that translation is a special case of transposition and that Jakobson’s intersemiotic
translation, in contrast to intralingual translation, maintains the characteristics of
translation proper, which is (usually) a double act of communication.

­SIMILAR CONCEPTS FOR INTERSEMIOSIS IN

TRANSLATION STUDIES
We argue that there are subfields where a semiotic theory can make a significant
contribution, and that is, phenomena, such as adaptation, interpictoriality, intericonicity
and transduction. It is worth mentioning that the terms intersemiotic translation and
transmutation were proposed by Jakobson (1959) to describe the interpretation of verbal
signs by non-verbal, but many semioticians accept that intersemiotic translation may occur
among non-verbal sign systems as Lawendowski (1978), Klinkenberg (1996), Sonesson
(1996), Torop (2000), Petrilli (2003), Fabbri (2008) argue, to mention but a few.
156 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

What is widely known in Semiotics and Translation Studies as intersemiotic translation

is widely known in film and theatre studies as adaptation. According to Bastin, ‘adaptation
may be understood as a set of translative interventions which result in a text that is
not generally accepted as a translation but is nevertheless recognized as representing a
source text’ (2008: 5). Eco argues that ‘many adaptations are translations in the sense
that they isolate one of the levels of the source text, and it is this level that the adapter
wishes to render in another continuum’ (2001: 125). It is worth noting that this kind of
intersemiotic translation concerns Jakobson’s definition of translation of verbal signs to
non-verbal signs systems which is the transfer of a literary work for the theatre, cinema
or television. But what is happening when the verbal system is not involved? In this case,
other terms are used instead, such as interpictoriality, intericonicity or transduction.
According to Sonesson, ‘we will have to take into account the possibility of intrapictorial
translation (e.g., exchanging one drawing for another) and interpictorial translation
(e.g., substituting a photograph for a drawing)’ (1996: 10). Sonesson (1996) classifies as
translation practices the case of the exchange and substitution of an iconic message by
another. He goes on to describe how the production of a cultural text could be based on
oppositions in absentia.
The term of interpictorial translation might not be widely diffused in semiotic
studies, but in Interart and Translation Studies the term intericonicity is frequently used.
According to Gamer,

[T]he concept of ‘intericonicity’ itself – other terms or descriptions for the phenomenon
are ‘interpictoriality’, ‘intervisuality’ or simply ‘intertextuality of images’ – is located at
the point of intersection of art history and literary criticism and is widely thought of as
analogous to the concept of literary intertextuality.
(2013: 115)

Gamer defines intericonicity as the ‘relationships between images as well as the modi
of their transformation from one into another’ (2013: 116). Gamer also highlights that
‘on closer inspection, it does not seem necessary to limit the usage of this term to art/art
relationships’ (2013: 215). In this perspective, Kourdis and Yoka (2014) tied to relate
this notion to translation studies based on Lotman’s and Fabbri’s theory. For Kourdis
and Yoka, words and images are inseparable and all pictures are not just similar images
presupposing one another but actual semiotic transpositions (2014: 174–5). Like with
every intericonic incident, intersemiosis between pictures (visual signs) and images
(on different media) points to the constructedness of the pictorial and the linguistic
signification underlying and cross-supporting it.
A special type of intersemiotic translation is transduction. Even though the term
transduction derives from biology, it is used in both semiotics and translation studies, two
disciplines that essentially study cultural communication. Kull mentions:

[T]he process of translation is central to the genetic code, which is rather a complex
of processes that manages the correspondence between nucleotide triplets and amino
acids. While the concepts of ‘code’ and ‘translation’ were initially used metaphorically,
one of the more important conclusions of biosemiotics is that these are in fact semiotic
processes, that is, actual aspects of semiosis.
(2015: 521)
TRANSLATION STUDIES AND SEMIOTICS 157

Influenced by Thomas Sebeok, Petrilli mentions that ‘transduction consists of a series of

transformations or translations in the source and in the destination, operated on the basis
of the interpretation of a probable homology between meaning and an externalized serial
string (e.g., speaking, writing or gesturing)’ (2014: 256). Petrilli is not the only scholar
who puts side by side the notions of transformation and transduction. Bezemer and
Kress, in their framework, use translation as the general semiotic term, transformation
for changes within a mode and transduction as the more specific term for the move of
semiotic material from one mode to another (2008: 169, 175).
The notion of transduction seems to be used in translation semiotics in a metaphorical
sense. Thus, for Fabbri, transduction is ‘translation between different semiotic systems’,
not excluding the language system (2008: 161). Thus, Fabbri supported the idea of a
meta-painting, i.e. of a painting that can not only formulate through language general
theoretical hypotheses, but which can also function through clearly iconic signifieds,
which cannot be expressed in words (2008: 161). Though Fabbri relies more on a
semiotic leap of faith to hold on to the idea of a non-linguistic sign that can be translated
into another non-verbal sign, he is certain that in the case of iconic signs we can actually
talk of transduction. Gorlée also highlights that transduction moves beyond language
and ‘is the new term that relates to the parallel of “translation” not applied to language
but within intermediality of speeches of non-linguistic texture between the doctrinal, the
educational, and the emotional aspects of different arts’ (2015: 9).
All these terms are based on the notion of similarity and of the transfer of the
information load in cultural communications. In our view, they refer to intersemiotic
phenomena and could be studied through the lens of translation semiotics.

CONCLUDING REMARKS
As scholars, we choose our theoretical approach and, consequently, methods with
respect to our preconceived idea of the research object. If this object is translation
and we consider it to exemplify linguistic activities, we probably search for a suitable
framework in linguistics. If we conceive translation rather as a phenomenon of cultural,
cognitive, sociological or technological nature, we acquaint ourselves with research into
such disciplines as cultural studies, cognitive science, sociology or digital humanities
respectively. If translation, in our view, primarily represents the realm of signs, we
presumably look for potential sign-theoretical insights. In this way, our approach finds
its justification in providing an angle and answers, interpretants of the sign ‘translation’,
which together can challenge, complement and widen our state-of-the-art notion of
translations and translating.
The present chapter has concentrated on discussing how translation semiotics can
construct and enhance our big-picture-perspective on translation. We have presented
how translation studies can benefit from semiotic views and how translation has recently
been studied within semiotics. Focusing on semiotics and its mainstream schools and
branches, we have endeavoured to emphasize that translation semiotics has early on,
since Jakobson, questioned many of the received or prevailing views, putting forth and
recommending an (often theoretical) alternative which originates from the persistent
revisiting and rethinking activities of translation semioticians.
If we translate our observations into semeiotic, the growth of translation semiotics
attests that what emerged as a mere possibility has, step by step, developed into the
158 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

actuality of today, reaching slowly towards the tendenciality of tomorrow. Despite this
encouraging development, the short history of the branch cannot thus far be described
as a success story. As members of the research community, translation semioticians are
the agents who with their diverse motifs, goals and emphases, and in their Greimasian
actantial roles as subjects and objects, senders and receivers, helpers and opponents, are
themselves the key to the future of their field.

NOTES
1 Popovič (1975: 16) is probably the first to refer to the semiotics of translation, denoting the
semiotic nature and aspect that characterizes the translation process due to spatiotemporal
differences.
2 All translations from French are by the authors.
3 For Lotman’s influence on translation studies see also Gorlée (1994: 19–21).

REFERENCES
Barthes, R. (1964), ‘Rhetorique de l’image’, Communications, 4: 40–51.
Bastin, G. (2008), ‘Adaptation’, in M. Baker and G. Saldanha (eds), Routledge Encyclopedia of
Translation Studies, 3–6, London: Routledge.
Bezemer, J. and G. Kress (2008), ‘Writing in Multimodal Texts: A Social Semiotic. Account of
Designs for Learning’, Written Communication, 25 (2): 166–95.
Chesterman, A. (2002), ‘Semiotic Modalities in Translation Causality’, Across Languages and
Cultures: A Multidisciplinary Journal for Translation and Interpreting Studies, 3 (2): 145–58.
Courtés, J. (2001). ‘La sémiotique, comprendre l’univers des signes’, in J.-F. Dortier (ed.), Le
langage. Nature, histoire et usage, 121–126, Auxerre : Editions Sciences Humaines.
Eco, U. (1976), A Theory of Semiotics, Bloomington: Indiana University Press.
Eco, U. (2001), Experiences in Translation, trans. A. McEwen, Toronto Italian Studies,
Toronto: University of Toronto Press.
Eco, U. (2004), Mouse or Rat? Translation as Negotiation, London: Phoenix.
Fabbri, P. (2008), Le tournant sémiotique, Paris: Lavoisier.
­Gambier, Y. and van Doorslaer, L. eds (2004–), Translation Studies Bibliography, Amsterdam:
John Benjamins.
Gamer, E.-C. (2013), ‘Configurations of Emptiness: Intericonic Blanks in Louise Lawler’s A Movie
without the Picture and Hiroshi Sugimoto’s Theatres’, in C. Taban (ed.), Meta- and Inter-
Images in Contemporary Visual Art and Culture, 115–31, Leuven: Leuven University Press.
Goethals, G., R. Hodgson, G. Proni, D. Robinson and U. Stecconi (1999–2000),
‘Semiotranslation: Peircean Approaches to Translation’, Athanor: Semiotica, Filosofia, Arte,
Letteratura, anno X, nuova serie (2): 139–50.
Gorlée, D. L. (1994), Semiotics and the Problem of Translation: With Special Reference to the
Semiotics of Charles S. Peirce, Amsterdam: Rodopi.
Gorlée, D. L. (2012), Wittgenstein in Translation: Exploring Semiotic Signatures, Berlin: de
Gruyter Mouton.
Gorlée, D. L. (2015), From Translation to Transduction: The Glassy Essence of Intersemiosis,
Tartu: Tartu University Press.
Gottlieb, H. ([2000] 2004), Screen Translation: Seven Studies in Subtitling, Dubbing, and
Voice-over, 4th edn, Center for Translation Studies, Department of English, University of
Copenhagen.
TRANSLATION STUDIES AND SEMIOTICS 159

Gottlieb, H. (2017), ‘Semiotics and Translation’, in K. Malmkjær (ed.), The Routledge

Handbook of Translation Studies and Linguistics, 45–63, London: Taylor & Francis.
Greimas, A. J. (1983 [1966]), Sémantique structurale, Paris: Presses universitaires de France.
Greimas, A. J. and J. Courtés (1986), Sémiotique. Dictionnaire raisonné de la théorie du
langage, vol. 2, Paris: Hachette.
Hartama-Heinonen, R. (2008), Abductive Translation Studies: The Art of Marshalling Signs,
Imatra: International Semiotics Institute at Imatra & Helsinki: Semiotic Society of
Finland.
Hartama-Heinonen, R. (2015), ‘Herding Together: On Semiotic-translational Branches, Fields
and Disciplines’, Punctum: International Journal of Semiotics, 1 (2): 39–52.
Jakobson, R. (1959), ‘On Linguistic Aspects of Translation’, in R. A. Brower (ed.), On
Translation, 232–39, Cambridge, MA: Harvard University Press.
Jakobson, R. (1971), Selected Writings, vol. 2, The Hague: Mouton Press.
Jakobson, R. (1973), Questions de poétique, Paris: Editions Du Seuil.
Ketner, K. L. (1998), His Glassy Essence: An Autobiography of Charles Sanders Peirce,
Nashville: Vanderbilt University Press.
Klinkenberg, J.-M. (1996), Précis de sémiotique générale, Bruxelles: De Boeck Université.
Kourdis, E. (2015), ‘Semiotics of Translation: An Interdisciplinary Approach to Translation’, in
P. P. Trifonas (ed.), International Handbook of Semiotics, 303–20, Dordrecht: Springer.
­Kourdis, E. and C. Yoka (2014), ‘Intericonicity as Intersemiotic Translation in a Globalized
Culture’, in W. Yongxiang and J. Haihong (eds), Our World: A Kaleidoscopic Semiotic
Network. Proceedings of the 11th World Congress of the IASS/AIS, 5–9 October 2012,
Nanjing Normal University, volume 3, 162–76, Nanjing, Hohai University Press.
Kristeva, J. (1969), Σημειωτική. Recherches pour une sémanalyse, Paris: Seuil.
Kristeva, J. (1974), La révolution du langage poétique. L’avant-garde à la fin du XIXème siècle:
Lautréamont et Mallarmé, Paris: Seuil.
Kukkonen, P. (2014), I språkets vida rum: Salens språk – språkets sal: Volter Kilpis
modernistiska prosaepos Alastalon salissa i Thomas Warburtons svenska översättning I salen
på Alastalo, Helsinki: The Semiotic Society of Finland.
Kull, K. (2015), ‘Introduction to Biosemiotics’, in P. Trifonas (ed.), International Handbook of
Semiotics, 521–34, Berlin: Springer.
Lawendowski, B. (1978), ‘On Semiotic Aspects of Translation’, in T. Sebeok (ed.), Sight, Sound
and Sense, 264–82, Bloomington: Indiana University Press.
Lederer, M. (1994), La traduction aujourd'hui, Paris: Hachette.
Lotman, J. (1977), ‘Primary and Secondary Communication-Modelling Systems’, in D. Lucid
(ed.), Soviet Semiotics: An Anthology, 95–8, Baltimore and London: The Johns Hopkins
University Press.
Lotman, J. (1990), Universe of the Mind: A Semiotic Theory of Culture, Bloomington and
Indianapolis: Indiana University Press.
Lotman, J. (2005), ‘On Semiosphere’, Sign Systems Studies, 33 (1): 205–29.
Lotman, J. (2009), Culture and Explosion, Berlin: Walter de Gruyter.
Lotman, J. and A. Pjatigorskij (1969), ‘Le texte et la fonction’, Semiotica, 1: 205–17.
Marais, K. and K. Kull (2016), ‘Biosemiotics and Translation Studies: Challenging
“Translation”’, in L. van Doorslaer and Y. Gambier (eds), Border Crossings: Translation
Studies and Other Disciplines, 169–88, Amsterdam: John Benjamins.
Mossop, B. (2016), ‘“Intralingual Translation”: A Desirable Concept?’ Across Languages and
Cultures, 17 (1): 1–24.
160 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Mossop, B. (2019), “‘Intersemiotic Translating”: Time for a Rethink?’ Translation and

Interpreting Studies, 14 (1): 75–94.
Parret, H. (2006), Sutures sémiotiques, Limoges: Lambert-Lucas.
Peirce, C. S. ([1866–1913] 1931–58), The Collected Papers of Charles Sanders Peirce, vols. 1–6,
C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University
Press. Cited as CP.
Petrilli, S. (2003), ‘Translation and Semiosis. Introduction’, in S. Petrilli (ed.), Translation
Translation, 17–37, Amsterdam: Rodopi.
Petrilli, S. (2014), Sign Studies and Semioethics: Communication, Translation and Values,
Berlin: de Gruyter Mouton.
Ponzio, Augusto (1999–2000), ‘Presentazione/Presentation’, Athanor: Semiotica, Filosofia, Arte,
Letteratura, anno X, nuova serie (2): 5–7.
Popovič, A. (1975), Dictionary for the Analysis of Literary Translation, Edmonton: Department
of Comparative Literature, University of Alberta.
Queiroz, J. and D. Aquiar (2015), ‘C. S. Peirce and Intersemiotic Translation’, in P. P. Trifonas
(ed.), International Handbook of Semiotics, 201–15, Dordrecht: Springer.
Radnitzky, G. ([1968] 1970), Contemporary Schools of Metascience, 2nd edn, Göteborg:
Göteborg University Press.
­Rastier, F. and C. Duteil-Mougel (2009), ‘Intersémioticité’, in D. Ablali and D. Ducard (eds),
Vocabulaire des études sémiotiques et sémiologiques, 215–16, Paris: Honoré Champion &
Besançon, Presses Universitaires de France-Compté.
Robinson, D. ([1997] 2003), Becoming a Translator: An Introduction to the Theory and Practice
of Translation, 2nd edn, London and New York: Routledge.
Snell-Hornby, M. ([1988] 1995), Translation Studies: An Integrated Approach, revised edn,
Amsterdam: John Benjamins.
Snell-Hornby, M. (2006), The Turns of Translation Studies: New Paradigms or Shifting
Viewpoints?, Amsterdam: John Benjamins.
Sonesson, G. (1996), ‘The Quadrature of the Hermeneutic Circle: The Picture as “Text”’, in
D. Wilske (preface), LSP and Theory of Translation. Acts of the XVI Vakki symposion, Text
and Image, Vöyri, February 10–12,1996, 9–33, Vaasa: University of Vaasa.
Sonesson, G. (2014a), ‘Translation as a Double Act of Communication: A Perspective from the
Semiotics of Culture’, in W. Yongxiang and J. Haihong (eds), Our World: A Kaleidoscopic
Semiotic Network. Proceedings of the 11th World Congress of the IASS/AIS, 5–9 October
2012, Nanjing Normal University, vol. 3, 83–101, Nanjing: Hohai University Press.
Sonesson, G. (2014b), ‘Translation and Other Acts of Meaning: In between Cognitive Semiotics
and Semiotics of Culture’, Cognitive Semiotics, 7 (2): 249–80.
Stecconi, U. (2004), ‘Interpretive Semiotics and Translation Theory: The Semiotic Conditions
to Translation’, Semiotica, 150 (1–4): 471–89.
Stecconi, U. (2007), ‘Five Reasons Why Semiotics Is Good for Translation Studies’, in
Y. Gambier, M. Shlesinger and R. Stolze (eds), Doubts and Directions in Translation Studies:
Selected Contributions from the EST Congress, Lisbon 2004, 15–26, BTL 72, Amsterdam:
John Benjamins.
Sütiste, E. (2008), ‘Roman Jakobson and the Topic of Translation: Reception in Academic
Reference Works’, Sign Systems Studies, 36 (2): 271–314.
Sütiste, E. and P. Torop (2007), ‘Processual Boundaries of Translation: Semiotics and
Translation Studies’, Semiotica, 163 (1): 187–207.
Tarasti, E. (2000), Existential Semiotics, Indianapolis: Indiana University Press.
TRANSLATION STUDIES AND SEMIOTICS 161

Tarasti, E. (2017), ‘The Semiotics of A.J. Greimas: A European Intellectual Heritage Seen from
the Inside and the Outside’, Sign Systems Studies, 45 (1): 33–53.
Toporov, V. (1992), ‘Translation: Sub Specie of Culture’, Meta, 37 (1): 29–49.
Torop, P. (1998), ‘The Limits of Translation: The Socio-Semiotic Aspect of Translation
Semiotics’, Sign Systems Studies, 26: 136–50.
Torop, P. (2000), ‘Intersemiosis and Iintersemiotic Translation’, European Journal for Semiotic
Studies, 12 (1): 71–100.
Torop, P. (2002), ‘Translation as Translating as Culture’, Sign Systems Studies, 30 (2): 593–605.
Torop, P. (2003), ‘Intersemiosis and Intersemiotic Translation’, in S. Petrilli (ed.), Translation
Translation, 271–82, Amsterdam: Rodopi.
Torop, P. (2007), ‘Methodological Remarks on the Study of Translation and Translating’,
Semiotica, 163: 347–64.
Torop, P. (2008), ‘Translation and Semiotics’, Sign Systems Studies, 36 (2): 253–57.
Torop, P. (2012), ‘Semiotics of Mediation’, Sign Systems Studies, 40 (3): 547–57.
Torres-Martínez, Sergio (2018), ‘Semiosic Translation’, Semiotica, 225: 353–82.
­Torres-Martínez, Sergio (2020), ‘Translating Wittgenstein: A Semiotic Translation of the
Tractatus’, Semiotica, 233: 91–123.
Toury, G. (1980), In Search of a Theory of Translation, Tel Aviv: The Porter Institute for
Poetics and Semiotics, Tel Aviv University.
Toury, G. (1986), ‘Translation: A Cultural-semiotic Perspective’, in T. A. Sebeok (gen. ed.),
Encyclopedic Dictionary of Semiotics Tome 2, 1111–24, Berlin: Mouton de Gruyter.
Uspenskij, B.-A., V.-V. Ivanov, V.-N. Toporov, A.-M. Pjatigorskij and J.-M. Lotman ([1973]
2003), ‘Theses on the Semiotic Study of Cultures (as Applied to Slavic Texts)’, in M.
Gottdiener, K. Boklund-Lagopoulou and A.-Ph. Lagopoulos (eds), Semiotics, vol. 1,
293–316, London: Sage.
Vassallo, C. (2015), ‘What’s So “Proper” about Translation? Or Interlingual Translation and
Interpretative Semiotics’, Semiotica, 206: 161–79.
Welby, V. Lady ([1903] 1983), What Is Meaning? Studies in the Development of Significance,
Amsterdam: John Benjamins.
Zethsen, K. K. (2009), ‘Intralingual Translation: An Attempt at Description’, Meta, 54 (4):
795–812.
162
 ­C HAPTER EIGHT

Pragmatics and Semiotics

PER AAGE BRANDT

Pragmatics, as well as semiotics, may be described as a discipline studying meaning and

language. A possible way of distinguishing these disciplines would be to let pragmatics
be the study of expressed meaning in a context of pragma, while semiotics takes care of
the internal syntax and semantics of expressed meaning, or in other words, to understand
semiotics as the study of the immanent aspects of meaning, and pragmatics as the study
of its transcendent aspects.1 If meaning only had transcendent aspects, in this sense,
pragmatics would be semiotics, and inversely. This view, however, presupposes, as the
French-Lithuanian semiotician A. J. Greimas says,2 a general theory of language, thus of
the general architecture of existing aspects of expressed meaning.3 In this respect, our
discussion necessarily involves linguistics. What would an immanent-transcendent model
of the componential structure of language look like? To answer this question would be
to describe the relations between semiotics and pragmatics, and it would yield a view of
a more comprehensive discipline that so far has no name. A semio-pragmatics? We will
venture to draft a version of such an answer by presenting a general model of language in
its structure and in its use.

MODELLING THE RELATION BETWEEN

PRAGMATICS AND SEMIOTICS
The first such model is based on the notion of semiosis. A semiosis is a ‘little machine’,
some say a ‘coded’ function, that makes an expression refer to something, its ‘content’.4
As Louis Hjelmslev ([1943] 1993) and, after him, Roland Barthes (1985) suggested,
expressions and contents can also each be entire semioses; so, one semiosis can be
the expression of, that is, connote, another semiosis.5 And it can be the content of
another semiosis as a text in a painting.6 Language can be described as an inter-semiosic
agglomeration of these little semiotic machines, an inter-semiosic architecture. This is
what the model shows (Figure 8.1, below).7 Let us take a look at the composition of this
inter-semiosic model of language.
The most conspicuous of the involved semioses is the one that makes phonetic strings, as
expressions, refer to grammatical meanings, as their contents; these strings can be morphemes,
words, phrases, clauses, sentences, entire texts, insofar as they have phonetic properties.
The lexical, grammatical and discursive meanings are the meanings thus conveyed.
Furthermore, phonetics is itself a semiosis, since the involved bodily, substantial
expressions (writing, gesture, phonation) mean the formal phonemes and syllables
that are the linguistic expressions. And the syntactic meaning of the resulting linguistic
164 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

expressions, that is, of the words, phrases, clauses, sentences and trans-phrastic units,
which all have conceptual meaning to contribute, is a linguistic semantics. Therefore,
we have, at the core of linguistic immanence, a general grammatical semiosis of two
semioses, the phonetic semiosis and the semantic semiosis. The bodily ‘input’ (phonetics)
is grammatically connected to a mental ‘output’ (semantics) of this process, namely what
we call conceptual meaning. Conceptual meaning exists in different formats, of course:
from single concepts to entire stories or doctrines.
However, this triple semiosis – phonetics-through-grammar-to-semantics – producing
conceptual meaning has to be inserted in an equally complex transcendent superstructure
in order to give rise to what we are especially interested in here, namely referential
meaning, i.e. meaning as we experience and understand it in context. Conceptual meaning
has to be converted to referential meaning, as semantics feeds into pragmatics.
In order to understand this wider framework, we first need to introduce the semiotic
component of enunciation (from Benveniste 1966). Enunciation is a semiosis, whose ‘little
machine’ lets certain expressions indicate states and attitudes of speaker’s and hearer’s
subjectivity, social identity and condition. The simplest manifestations of enunciation in
language are the systems of personal pronouns and the deictic forms (demonstratives,
shifter adverbs, etc.) that languages use. The dimension we call prosody superimposes
enunciational meanings onto the immanent linguistic semiosis. These meanings include
speaking from a higher or lower status than the hearer’s; speaker and hearer knowing
or not knowing each other; gender and gender difference; generational difference; and
(personal or social) emotional tonality. The enunciational semiosis overdetermines all
utterances uttered, and the result is then transposed into situational meaning through
a specific pragmatic component that takes the personalized utterances as appropriate
signifiers and the specific situation as signified. This pragmatic component is a semiosis
in the sense that it makes the personalized utterances mean their situational context,
contribute actively and intentionally to making the context specific and in this respect,
meaningful. Contexts become meaningless if we do not ‘intend’ them by letting our
language signify them. Ceremonial rituals become invalid if the formulaic language they
involve is not displayed appropriately; conversations are not taking place if our utterances
do not ‘converse’ in the right way (lecturing, for instance, is not conversing properly).
Even quarrelling does not happen if our language does not know how to ‘quarrel’. The
pragmatic ‘little machine’ will therefore have to know the culturally defined situational
categories that language is supposed to signify; otherwise we would speak out of order,
use wrong turn-taking, etc. Pragmatics in this sense is really a particular semiosis, and an
important one, since it is when language is used in culturally defined situations that its
conceptual meaning is converted into referential meaning.
Finally, the personalization that enunciation provides, and the ‘situationalization’ that
pragmatics provides, integrate in discourse, that is, in a discursive, rhetorical semiosis, in
which an enunciative personalization expresses pragmatic situationalization: the roles
we play as speakers and hearers theatrically determine and mean the situations we are
in; so celebrative rhetoric sounds unlike political rhetoric or judiciary rhetoric, and the
rhetorical ‘sounds’ will again mean these different situations, where language therefore
will carry distinct referential meanings. Enunciation, pragmatics and discursive, theatrical
rhetoric constitute the overarching, transcendent semiosic complex by which language is
‘carried’ into the world of its ‘use’.
We might want to follow Ferdinand de Saussure (1915) or reinterpret him and call
the lower triad of semioses, in French, la langue, or the language system, as opposed
PRAGMATICS AND SEMIOTICS 165

FIGURE 8.1 The complex functions of language.

to the upper triad of semioses, la parole, or language use. The hinge between the lower
(immanent) and the upper (transcendent) part of the complex is the core utterance
semiosis we also called grammar. A theory only accepting to study the lower semiosic
triad would just be a grammar theory, rather than a comprehensive, linguistic, general
science of language embracing both the immanent and the transcendent parts of language
as a constellation of semiosic ‘little machines’.

PRAGMATICS AS SEMIOSIS AND DIEGESIS

What is a situational context of an utterance? Eric Landowski8 suggests that such a
‘semiotics of situations’ would coincide with a complete narrative grammar, including an
account of inter-actantial relations of ‘manipulation’, aspectualizations, modalizations,
etc. A main point of Greimasian semio-narratology is in fact to theorize the narrated
content, the story, apart from narration itself, thus as a series of situations where things
simply happen to people, and the narrator has not yet arrived, so to speak (the narrator
supposedly arrives through subsequent ‘discursivization’). The semantic level of the
narrated is separated from the semantic level of the narrating, in the same way as the
signified is seen as distinct from the signifier. A situation is thus simply an episode in
the signified diegesis of a (possible) story. The next question will then be what the little
pragmatic machine, or the semiosis of situations, recognizes as diegetically relevant
episodes, which may thus contextualize language use. Is there really a canonical set of
situational parts in a canonical sequence forming the ‘whole’ of a story? The claim to have
found it would be strong, and no doubt far too strong, unless we believed that Propp and
Greimas really solved the universal mystery of defining narrative structure.9 Nevertheless,
the problem of classifying situations and thereby, if possible, opening a path towards a
structural pragmatics is relevant; it has to be addressed, since as Landowski intimates in his
1983 lecture; otherwise pragmatics would let psychology and sociology define not only
166 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

contexts but pragmatics altogether. Such a relinquishment would in fact be unfortunate,

because the sociological categories would be too distant from the situational format, and
the psychological categories would be too proximal, too intimate, to match the format.
Let us then assume that language is used in situations, and that situations are diegetic.
Speaker and hearer are ‘actants’ and inscribed in a presupposed narrative and ‘actantial’
semantics.
Now let us consider an example. The expression to make sure, Cambridge Dictionary:
‘to act so that, or check that, something is certain or sure’. Whether by acting or by
checking to produce a state of belief. The conceptual meaning of the expression is open,
so the task of the pragmatic setting of its use is to deliver its referential meaning in context.
Here is a context10:
1. Two Hunters are out in the woods when one of them collapses. He doesn’t seem to
be breathing and his eyes are glazed over.
2. The other guy whips out his phone and calls the emergency services. He gasps, ‘My
friend is dead! What can I do?’
3. The operator says, ‘Calm down. I can help. First, let’s make sure he’s dead.’ There
is a silence, then a shot is heard. Back on the phone, the guy says, ‘Ok, now what?’
(Italics mine).
The diegetic unfolding inscribes the expression /make sure/ in a dialogue, again
inserted in a story. We may see this narrative context as consisting of four phases: (1) the
collapse; (2) the call; (3) the advice, containing the performative utterance, let’s make
sure; (4) the killing.
The ‘actants’ are staged as hunters, therefore armed: (1) presents a crisis; (2) makes it
a catastrophe: the speaking hunter, in panic, makes a mistake by mentioning his partner
as deceased; (3) displays the consequence: the operator’s epistemic advice reuses the fatal
expression; (4) thereby triggering the conclusion, the killing of the collapsed hunter – told
from the operator’s viewpoint, so the act is only heard, not seen.
Crisis, catastrophe, consequence, and conclusion form a diegetic series of narrative
situations offering variable dynamic conditions. In this story, these first include the
participants’ fragile bodily conditions (collapsing); then a fragile psychological condition
(panic); a fragile linguistic sensibility and intelligence; and the fatal mistake rendered
probable by the condition of carrying arms11: (4) refers back to (1) in this respect.
Maybe the experience of life is diegetic in this or a similar sense? A recurrent contextual
rhythm of four canonical phases might then underlie our apparently fluid experiential
time and connect perception and reflection by the same backwards reference we find in
narrative literature and jokes.12 This contextual rhythm, implicit or explicit, might then
form a discreet elementary micro-narrative phenomenological music of time presupposed
by the larger historical contextual frames that interest hermeneutics and the disciplines
of historiography, where the experiential dimension disappears, and where the linguistic
problem of contextual determinations of meaning is therefore less relevant.

THREE TYPES OF MEANING

Making sure is either assertive (stating) or performative (acting). In modern standard
pragmatics, the distinction between performative and assertive meaning is well-known
and accepted as relevant, across various terminologies. Only assertive meaning can
PRAGMATICS AND SEMIOTICS 167

carry truth value. Performative meaning instead can carry validity or ‘speech act force’.
In modern standard semiotics, as in Landowski, supra, this pragmatic distinction is
translated into semiotic terms of modality: epistemic utterance force (truth value) versus
deontic utterance force (performative value), respectively. In French, this translation is
particularly tempting, since the French language only has two pure modal verbs, pouvoir
and devoir, the former basically epistemic, the latter basically deontic – though A. J.
Greimas’s modal syntax of course complexifies the field in many ways. Standard semiology
also essentially distinguishes two semantic sign types, icons (signs of possibility, according
to C. S. Peirce) and symbols (signs of necessity, whether logical or other). Of course, such
dichotomies do not in any way explain the variations in force and value. We would like to
better understand how the dichotomies work, why they work, and whether they account
exhaustively for the types of meaning that exist in the pragmatic world. Terminologies
alone do not satisfy our curiosity.
Symbols are signs of doing, and icons are signs of being: symbolic signs are always
instructions calling for acts, physical or mental (in math, music, social codes, etc.), whereas
iconic signs are manifestations of seeing something, physically or mentally, in certain ways.
In language, therefore, words are symbolic, and sentences are iconic.13 So, lexemic signs
are ‘arbitrary’, conventional and, in a sense, performative, comparable to imperatives,
orders in context (like social graphic signs: flags, emblems, injunctions: No smoking!
Wrong way; No u-turn). Words are instructions for the mind to follow. Sentences are
motivated co-lexemic signs, whose connecting morphemes create a sort of assembled
conceptual predicate to a scene, an image of it, a simulacrum of a representation; only in
ritual uses do sentences become symbols.
Icons are signs of being, that is, of something being seen as something, as a subject
with a predicate that invites the addressee to share the predication and thereby establish
an affective bond with the sender.
When icons and symbols combine, the operation consists in establishing the affective
iconic bond and then superimposing a symbolic instruction to act accordingly, namely
as the predication suggests (in caricatures: dissociate yourself from the iconic subject,
reject it; in advertisements: associate yourself with the subject, buy it). Communication
in the social sphere mainly uses such icono-symbolic messages; the combination is, so
to speak, the basic formula for manipulation. Roland Barthes discusses it in his famous
article ‘Rhétorique de l’image’ on a Panzani advertisement (Barthes 1964). Semiotically
speaking, there is in this advertisement a double superposition, namely a (symbolic) text
IN the image and a text ON the image. Figure 8.2 shows an unfolding of the result, an
image that connotes, has a meta-semiotic comment and contains a text (naming the brand
announced); hence a complex of four semioses.
In this graph, the connotative semiosis taking the iconic semiosis as its expression derives
its abstract content from the metonymic meaning of the iconic content. Composition and
chromatic properties of the image can trigger rich associations. Connotation and metonymy
are closely related. The driving force in such semiosic constellations is always the affective
effect of the iconic parts (the referential image and its metonymic connotations) and the
deontic effect of the symbolic parts (the metatext and the embedded text).
There is, however, a third type of signs, besides symbols and icons, namely diagrams.14
These are neither arbitrary-and-performative nor image-like and similar to their
representational content; they are neither about doing nor about being-and-feeling. They
are instead graphic or gestural signs of minding – diagrams spontaneously use labelled
arrows, boundary lines, bubbles and boxes, imaginary topologies of many sorts, which
168 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 8.2 The rhetoric of image and text.

are schemas, models or ‘theories’ of how things may be connected. Diagrams are signs of
minding, modelling, ideation and surprisingly enough, the most important sign type in the
social world, while also being the most ignored. There are reasons to believe that thinking
itself is diagrammatic, and that the diagrammatic semiosis constitutes the semiotics of
the mind itself, whether the conscious or the unconscious part. Ideas we exchange are
most often neither instructions nor factual or idealized pictures; they may instead be
sketches of a programme, a half-baked plan, a hypothetic construction, building blocks
of a problématique, as the French say, an imaginary architecture to be completed by
self or other. A diagram shows the thinking of something, simply as being thinkable
(German: als denkbar). Disciplines of knowledge are mostly based on signs of this type,
so the language that supports them is also of the third kind: dialogue and discourse, not
just words and sentences. Dialogue and discourse allow us to ‘think aloud’ and thereby
to think together, letting others complete our thought, or trying to complete others’
thought. When we discuss, we negotiate a shared imaginary architecture, corresponding
to a diagram, and sometimes explicitly a drawn diagram on whiteboard or paper that we
can point to.
Symbols, icons and diagrams are three fundamentally different sign types in two
ways. Firstly, by the constitution of their semiosis: the expression-to-content relation
is conventional and non-motivated in the first case, non-conventional and motivated in
the second case, and neither conventional nor motivated in the third case.15 Secondly,
the type of meaning they convey is different: the addressee of symbols is called upon to
act in some manner (the address is deontic: ‘you are suggested to do X’); the addressee
of icons is called upon to perceive something in a specific manner (the address is
alethic, and therefore affective16: ‘you are invited to feel this about X’); and diagrams
invite the addressee to connect ideas, i.e. think (so the address is epistemic: ‘we may
understand the problem X like this’). The sign types are as mutually irreducible as
the meaning types.17 Responses and pragmatic exchanges differ according to these
variations. You may obey or contest in the first case; be captivated or repelled in the
second; and be interested or sceptical in the third. Discourse is possible in all cases, but
PRAGMATICS AND SEMIOTICS 169

dialogue is optimal in the third. Dialogue becomes quarrel if the exchange is entirely
symbolic; by contrast, it vanishes or becomes a mutual echoing, if the exchange is
entirely iconic.
We will of course want to know why this is so. It will therefore be useful to consider
the interrelations of meanings and their signs in social life and in our subjectivity.

POWER AND CRITIQUE

Symbolic meaning rests upon principles of authority, namely that of the instances
that back up the speaker, and entitles the enunciator to use performatives in specified
situations. Authority in this sense includes named institutional identities (ministries,
political parties, corporative, industrial, religious entities, etc.) and traditional
cultural, historical references (memorable events, dates, personal statuses, domains
of knowledge, etc.), and allows speakers and writers to express authoritative ‘values’
instead of their own thinking. All commands, instructions and otherwise imperative
utterances receive their ‘speech act force’ from a presupposed entitlement signed by an
authority, at least in the imaginary mode. The ritualization of commands (uniforms,
gestures, formulaic phrasing) manifest this depersonalization of the performer. I would
like to add that the ritualization and the depersonalization of the symbolic performer
also produces another relevant effect: iconization. The performer plays a theatrical role
and becomes an image of the non-personhood of authority – of its rigidity, the scary
inanimate statue we call power.
Ordinary iconic meaning is not backed by any such authority. It relies only on the
affective meaning of the depicted motif, which means that making an iconic representation
of something is (a symptom of) affirming its affective meaning. Therefore, icons of the
imaginary gestalt of authority, that is, icons of the source of symbolic force, may be the
foundational semiotic function that constitutes power in general.18
The mechanism described can be analysed in terms of enunciation. There is an
elementary structure, in which first person (enunciator, P1) gives third person (object of
attention, P3) to second person (enunciatee, P2). The constitution of power uses three
embedded instances of this structure:
I. A human collective (P1) offers a cult ICON (P3) to itself (P2)
II. The ICON (as P1) then confers authority (A, P3) to a person S1 (P2)
III. This person S1 (as P1) finally gives another person S2 (P2) a symbolic message,
an order (SY)
IV. The order (P1) tells S2 (P2) to act accordingly (P3).
The uncanny statue of power (the icon of authority) stands behind the instance
authorizing S1’s symbolic, performative acts, and it emerges as a compulsive motif of
depiction: portraits of rulers hanging in the offices of their servants, monumental statues
of famous and supposedly great historical persons, placed in the urban landscape,
pompous architecture of banks and businesses, even advertisements linking the power
of brands to images of their ‘greatness’, election posters, etc. – all such semiotic displays,
iconic and symbolic, and eventually the entire social iconicity may consist in offering
competing cults of implied symbolic power holders of all kinds.
Universally, the iconic semiosis may have a symbolic semiosis hidden in its content,
as in Figure 8.2. Thereby, affect and duty – fear and pride – will be connected in the
170 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­FIGURE 8.3 The semiotic constitution of power through enunciation.19

complying social addressee’s mind. Hence the interest of Barthes’s analysis: if social
power is a pragmatic effect of socio-semiotic structures, a possible counter-force may be
sought in critical structural commentary.
Diagrammatic meaning, as it happens, can be critical. It can oppose iconic cult and
symbolic obedience. It always ‘occurs’ to the thinking mind, which ‘gets’ ideas instead
of actively producing them. Thoughts are mental diagrams that form the elementary,
‘occurring’ contents of consciousness, for example as maps of one’s present location,
intuitions about relations between present persons, and about the time frame of present
doings (something is still the case; but something else is already the case). More abstract
diagrammatic contents are supported by worded labels or other markers, imported from
memory, and inserted in the nodes of the intuitive graphics.
A literary drama is slightly more abstract than an episodic experience, essentially
because we are not in it. We are abs-tracted from it, and its understanding becomes an
intellectual challenge to meet by diagramming. Science is built upon elaborate models
of abstract empirical domains of the world; it ‘formalizes’ its intuitive diagrams into
models which then underlie scientific theories. Political life would, on the other hand,
be impossible without diagrams analysing the respective force of competing powers.
The contemporary version of global political life has conferred overwhelming iconic-
symbolic capacities to the digital media and is responsible for creating a dangerous
imbalance between power and critique. Diagramming on screens is mainly seen in critical
programmes, which are becoming rare; one reason is that diagrammatic thinking is slow,
and that digital time is an expensive commodity demanding fast action; another reason
is that global culture may be decreasingly prepared for autonomous alphanumerical,
diagrammatical, mental activity by general education – except for weather or stock
exchange reports. Critical activity is then bound to unfold wildly in the shape of
bodily conflicts and material clashes of collective ‘identities’, each with its protective
iconography.
PRAGMATICS AND SEMIOTICS 171

FIGURE 8.4 A sociogram example.

MUSIC
Rhythmic emission of tonal phrases is probably the earliest semiotic activity of our
species. Before serving as a generalized anaesthetizing tapestry for all sorts of messages in
the media, it was mainly used for celebration. Music for weddings, funerals, coronations,
memorial events of all kinds, was fundamental. It had, and still under certain circumstances,
has the phenomenological capacity to extend the moment of celebration: to let ‘time
stand still’. Concert music is self-celebrating and can be considered as experimental or
laboratory music, as opposed to what I will call natural music, the sort that has been with
us in our different cultures from their beginnings. The pragmatics of music20 is thus a
global, universal and evolutionarily primordial concern. Its generality is the same as that
of language, and curiously, music unfolds a closely similar structural complexity.
Musical semiotics has in fact, like language, an immanent and a transcendent aspect.
In music, certain sounds are performed and perceived as discrete tones, which are parts
of melodic lines (syntagmatic axis) and also parts of tonal scales (paradigmatic axis) based
on octave recognition. This aspect is the ‘phonetics’ of music. The melodic wholes have
necessary rhythmic properties that determine how the involved bodies will move to it and
especially dance, thereby expressing particular emotional states. These states are ‘staged’,
induced by the music or called into being by its experience, and constitute the immanent
semantic meaning of the specific songs played. In modern, academic music, where corporeal
movements are often suppressed, the dancing and its meaning are mentalized and ‘performed’
by the imaginary homunculi in your head, whether you are a musician or a listener.
The transcendent aspect includes the situation connoted by specific musical phrases and
by the tonality chosen and used; scales are often specific for specific contexts: morning
172 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 8.5 The complex semiosis of music.

music, evening music, night music, winter music, spring music, summer music, fall music,
high status scales, low status scales, etc., as if the calendar and the entire social spatio-
temporal world were represented in the inventory of a culture’s acknowledged tonal
scales. Tonality and pragmatics are thus naturally connected.
The analogy between the complex semioses of language and those of music may make
the following diagram easier to interpret.
It is likely that the neuro-biological disposition for language is very similar to the
disposition to make and enjoy music. Music may even have been laying the neural ground
for language.21
In language, the integration of immanent, conceptual meaning in transcendent,
pragmatic, referential meaning is an essential aspect of semantics, as we have seen. In
music, this integration comes to the fore as a duality of feelings of the ‘character’ of a
given work (if written) or song (if not) and the ‘character’ of the situation of performance:
music ‘for’ funerals (sad), weddings (happy), coronations (solemn), military parades
(awe-inducing), etc. – often linked to power displays. If power is rooted in collective,
intersubjective iconicity (subjects seeing and believing what other subjects are seeing and
believing), as suggested above, then the mere sharing of music, both seen and heard, will
reinforce this constant re-founding effect of iterated collective performances that pertain
to the existence of social powers, whether affirmative or contestant.

THE PRAGMATICS OF TROPES

Through it all, Welser-Möst and the [Cleveland] orchestra are dashing partners,
surrounding their guest [Radu Lupu] in vibrant colors without washing out any of his.
On Thursday, they also helped steady him during the Adagio [of Bela Bartok’s Piano
PRAGMATICS AND SEMIOTICS 173

Concerto No. 3], when a ringing cell phone twice poked holes in the silken musical
fabric he was weaving. To listeners, the noise was simply rude. But to Lupu, it must
have been painful, like lightning striking.
(Zachary Lewis, ‘Ensemble offers up an appealing climate’
Cleveland Plain Dealer, 01/17/09)

Clouds occasionally gather in the E-flat trio, though Schubert is resilient enough to
emergefrom minor-key territory and let the sun shine in his inimitable way (Donald
Rosenberg, ‘Trio Conveys Wonder of Schubert’s Work,’ Cleveland Plain Dealer,
27/03/10 [Italics mine in both quotations]).22
Speaking about music as heard is not an easy task. Strikingly often, critics seem to refer
to meteorology rather than to tonality. Above: lightning, climate, clouds, sunshine – but
also vibrant colors, silken . . . fabric, and in all cases, visual, auditive and often also tactile
sensations (silk) that happen in an imaginary space far from the concert hall, and far from
what is written in the composition being played, in the above cases.
You may be thinking of metaphor to explain this phenomenon. Language is not
prepared for describing sensory perceptions that activate deep registers of the soul, so we
use expressions of weather observations and other highly emotional imaginary contexts
to express what we feel. However, metaphor would supply imagery to a topic that
could also be covered literally; here, there is no literal wording of what is experienced
available. In some books, the case is then called catachresis, or necessary metaphor (table
leg, headquarter, etc.). But in metaphor, the image covers, and often hides, the referent
(target), and, so to speak, signifies it. In the concert hall, the sounding tonal events
inversely signify the reported imagery! This imagery seems to be the target of the tones. I
have proposed (Brandt 2013) to treat the phenomenon as ‘inverted metaphor’. Anyway,
the meaning of music is often experienced as related to this imaginary content.23 There
is a mental space of imagery, often cosmic or meteorological, or just narrative (as in
note 16), and there is a different mental space containing the sensory perception; they
connect in interesting ways, not only by the possible mappings between items of each
space, but also by appearing as if they were one and the same blended space: Radu Lupu
weaves silken fabric, Schubert lets the sun shine (in the quotations above). The semiotic
and cognitive theory of blending, or conceptual integration, offers a pragmatically
relevant theory of such experiences and textual occurrences of double-space semantic
descriptions.24
In this theory, there is a semiotic Base space, a situation of experience or communication,
where subjects exchange signifiers, and signifieds therefore emerge in their mind
according to the semioses discussed above. These signifieds are now further observed as
forming spaces around themselves, instead of just appearing as isolated nominal concepts
of thought. It is suggested that we think and produce meaning through processes that
establish, hold and combine mental, semantic spaces where our isolated concepts are
contextualized, allowing them to change and merge, thereby giving rise to new concepts.25
The most interesting aspect may be the discovery (Brandt and Brandt 2005) that
the blending process must be stabilized by a schematization stemming from Base space,
that is, from the reality of the thinking or communicating subject(s). If ‘this surgeon
is a butcher’,26 the evaluative and affective meaning produced stems from the ethical
schema of helping and harming applied to the blend, not from the generic butcher.
Schemas of different kinds are activated by the immediate clash of mapped terms
174 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

(surgeon ≠ butcher!) and are semantically foregrounded in the blend. Each schema
may be either generally cultural or locally social, it may be rooted in history or in
phenomenology, or emerge as part of the present situation, and even be imported from
ongoing dialogue, since the base space it comes from has variable ontological depth;
and this pragmatic anchoring is thus a necessary condition for the meaning production
in the blend to happen.
This metaphor model, which can be generalized to other tropes, first of all to metonymy,
shows that in blending processes, pragmatics is inherent in semantics (see Figure 8.6a–c).
Whereas metaphor has input spaces whose contents are taken from different
experiential domains,27 metonymy uses input from same domain in both mental spaces.
Example: ‘I saw a Banksy in the street today’, meaning a graphical work by Banksy. The
artist referred to is in one space, therefore called Reference Space, in the model, whereas
the drawing is in the space of present perception, the Presentation Space. In the blend, the
artist seems to materialize in the scenario as reincarnated in his artwork. Both metaphor
and metonymy are counterfactual in this sense. But in the latter case, the schematization
is of a particular nature: it expresses a respectful attitude towards an authority of some
kind, positive or negative. Using a part to ‘mean’ a whole, or inversely, or a concept for
a related concept, is invariably an operation schematized as an enunciative indication of
attitude, in this sense. Again, this schema emerges from the speaker’s Base space, not from
the semantic contents involved.
In Figure 8.6a–c, I will first summarize the general model of semio-pragmatic blending,
and then show the blending process supposed to structure our two examples of metaphor
and metonymy.
The field of application of this semio-pragmatic model is much larger than what can
be discussed in this presentation. (The sentence you just read is an example of theoretical
metonymy: what you see represents something that is not presented, but which is thus
represented as worthy of your attention.)

FIGURE 8.6a The blending model.

PRAGMATICS AND SEMIOTICS 175

FIGURE 8.6b Metaphor.

FIGURE 8.6c Metonymy.

We may, however, add the tricky case of elementary irony. Wayne Booth (1974) offers
this example:

A man enters his office totally soaked after having biked from home under a torrential
rain, and greets his colleague with few words, something like . . . ‘Nice weather today’.
Deidre Wilson (2006) has the same sort of examples:
Mary (after a difficult meeting): ‘That went well.’
176 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Classical accounts define irony by formulae like this one:

1. A says p to B
2. A knows that ¬p
3. A intends B to understand p in (1) as meaning ¬p
The lemmas (1) and (2) alone would define p as a lie. But the semantic miracle happens
in (3), which therefore calls for a pragmatic explanation. What needs explaining is also
why someone would think and talk like this at all.
Note that in Booth’s example, it first has to rain heavily, and that in Wilson’s, the
meeting has to be specified as difficult. We therefore need to add two more lemmas to
the formula: (4) B knows that p/¬p is A’s evaluation of the situation X that p/¬p refers
to, and which directly involved A28; and (5) The use of p in (1) and (3) expresses A’s
frustration, to be empathically shared by B, caused by a very high degree of ¬p in X.
The instantiations of the last lemma (5) often manifest a semiotic dimension, a facial
or gestural expression, or a phonetic aspect, a modification of intonation, as if the speaker
were quoting someone. To clarify, let us now project these observations onto the blending
model.
In the base space, we have A, B, the situation X (heavy rain or difficult meeting) and
A’s utterance ‘p’. In the Presentation Space of the network, there is the evaluation p of
X, and in the Reference Space, the contrasting evaluation ¬p of X. In the blend, p and
¬p of X somehow merge, and after an implicit schematization yield the content of the
Meaning Space, ¬p!
There is in fact a schema characteristic of irony; we might call it ‘the road not taken’
(see Figure 8.7a): S, the subject, follows a path that suddenly bifurcates in such a way that
there is a very attractive path and a repulsive path, and the former is barred; therefore, S
must take the latter, while still thinking of the other one and regretting not to have been
able to be on it.
Note that irony can be ‘biting’; this aspect ‘bites’ at the barrier on the p branch:
something went wrong, and the cause is evident, namely that ‘you blew it!’29 In this case,
the addressee is foregrounded as responsible for the barrier. Enunciation is active in this
way particularly in dialogues of quarrelling.
As illustrated in Figure 8.7b, the network activates the schema in the Relevance input
to the blend, where p and ¬p merge, paradoxically, absurdly, but again are disambiguated

FIGURE 8.7a The schema: ‘the road not taken’.

PRAGMATICS AND SEMIOTICS 177

FIGURE 8.7b Irony.

by the schema and ‘resolved’ as an emphatic ¬p! to be shared in the pragmatic Base space
by A and B or else by A and bystanders (C), as in polemical irony, where A and B is are
antagonists.
The blending mechanism is ubiquitous in language and in graphic imagination, such
as cartoons, caricature drawings, advertisements and poetry. In all contexts where the
expression itself is foregrounded, which is the case whenever emotion prevails, our
communicative thinking tends to split the semantic content into two mental spaces
interrelated as a Presentation to a Reference, and, fundamentally, as a foregrounded
predicate to a known subject, in the logical, propositional sense.
In poetry, where the stipulated situational context (X) is aesthetic (that is, determined
as both sacred and intimate30), irony is a possible ‘strategy’ besides all sorts of tropes and
linguistic and graphic operations on the form of the utterance. The primary such operation
is of course the introduction of the phenomenon called verse, whether metrical or ‘free’,
as an expressive gesture of disruption that may be compared to the implicit negation (¬)
in irony. In poetry, a referential content r is intended behind a presentation where r is
disrupted by verse, tropes, figures, graphic gestures, syntactic cut-ups and a more or less
chanting vocal performance: (¬)r. We often oppose the ironic coolness and the pathetic
heat in lyric contexts (poetry and songs); the poetic heat may stem from the fact that the
disruption (¬)r appears in the Presentation space, whereas it appears in the Reference
space of irony.31 The fundamental poetic schema may however be almost the same. The
barrier blocks the enunciator from going ‘directly’ to the core of a content, which is
sacred and intimate; the blocking is a call for circumlocution in a very wide sense.

­CONCLUSION: SEMIOTICS AND PRAGMATICS

As we have seen, meaning is determined by both immanent and transcendent semiotic
structuring: it is both conceptual and contextual. The recursion of semiosis makes it
possible to understand and theorize this open but non-chaotic relation between minimal,
178 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

medial and maximal sign structures and the experiential life world that infuses social
systems with meaning and lets cultural, semiosic and mental content develop as a
continuity. Semiotics and pragmatics are interconnected, and their bonds are indissoluble;
if cut off, pragmatics would become a part of psychology and semiotics a specialty of
linguistics.
The cognitive theory of mental spaces and conceptual blending needed a semiotic and,
as we suggested above, a semio-pragmatic grounding in order to grow out of its initial
format as a philosophical daydream. The model explained here shows how situational
and experiential contributions, namely the schemas that blend activate, intervene in the
sense-making, also called the meaning production, of the processes of creative thinking
and signifying that semiotics studies; the pragmatic interest in signifying behaviour,
on the other hand, would disappear into sociology and anthropology, or collapse into
linguistics, if the semiotic insistence on iconicity, symbolicity, diagrams, enunciation and
deixis did not call it back to the fundamental life of signs in human life.

NOTES
1 An example: The concept of justice means different things in the (transcendent) context of
revenge or retribution and in that of jurisdiction. In itself, immanently, it contains a schema
of balance or equilibrium between ‘weights’ of penalty and wrong-doing.
2 ‘the only way out is through the formulation of a general theory of language which will
postulate a necessary complementarity between syntax and semantics on the one hand, and
pragmatics on the other’ (A. J. Greimas in Perron and Collins 1989: 93). The paper, entitled
‘Pragmatics and Semiotics. Epistemological Observations’, was read as an introductory
lecture to the International Symposium on Pragmatics and Semiotics held at the University of
Perpignan from 17 to 19 November 1983.
3 Meaning is expressed in many ways, and language is only one of them, although presumably
predominant. The realm of available expressive activities in the human world is extensive.
For example, there are certainly reasons for developing a pragmatics of music, as of the vast
family of iconic media and symbolic gestures and objects (think of architecture). Nevertheless,
it may be wise to start out from considering language in particular; very few of the other
forms would have evolved without it.
4 The function that ties a content to an expression, or a signified to a signifier, is an automatic
mental mechanism, not a process of reasoning like the interpretation of a treatise of
philosophy; I use the diagrammatic format of a triangle for a semiosis, so that the ‘coding’
mechanism can occupy an angle. This ‘little machine’ is a cognitive routine intimately related
to the neural processes that make us think. We do not know yet what they are and how they
causally work, but we can approach them by observing as closely as possible what they are
achieving: the different forms of meaning-making that pervade the human world.
5 The principle of semiosic recursion is presented in ‘The Sign Cascade – A Basic Format
of Semiotic Apperception’, in Fraser and Turner (2009). ‘From Linguistics to Semiotics:
Hjelmslev’s fortunate Error’, in Hénault (2019), has a broader unfolding of this glossematic
discovery.
6 Hjelmslev called this constellation a relation between a metalanguage and its object-language.
The ‘meta-semiosis’ that has an ‘object semiosis’ in its content is rather the context of the
embedded semiosis. This constellation is therefore of great interest to pragmatics as a study
of contexts.
7 The model is explained in the chapter ‘The Role of Semio-Syntax’, in Brandt (2019).
PRAGMATICS AND SEMIOTICS 179

8 The (1983) symposium lecture, ‘Pragmatics and Semiotics. Some Semiotic Conditions of
Interaction’, in Perron and Collins (1989). ‘What is being dealt with here is the résemiotization
of context, or, rather, the setting up of a semiotics of situations’ (p. 101).
9 Innumerable attempts have been made, including my own, ‘Forces and Spaces’, in Brandt
(2020).
10 This joke was rated and declared to be preferred over a thousand other jokes by a large
American test population, as reported in the science journal Nature: “ScienceUpdate”,
4 October 2002.
11 This is a particularly fine joke for Americans.
12 Brandt (2020) has a chapter explaining this diegetic model and illustrating it by the reading
of short stories by Maupassant, Borges and Hemingway.
13 This statement may seem a little blunt. What is meant is the following: words are firstly
instructions for the mind to actualize concepts, whereas sentence construes imaginable
scenarios.
14 Peirce’s third semantic sign type is of course the index, unfortunately a poorly elaborated
type. When we perceive one event (smoke) and infer another event (fire), which is the case in
all forms of perceptions, except some aesthetical forms, then the world is not signifying to us
what it intends us to think, as by sending us a signifier to trigger a signified in our mind. The
index is not a sign, since it is not intentional. What is happening instead is that the perception
contains an inference. We have to interpolate a schema between the first categorization event
(smoke) and the second (fire), either a mereological or a causal schema. These schemas are
mental diagrams, and they are of the exact same constitution as our expressive diagrams.
Therefore, the third sign type is the diagram, not the index.
Furthermore, the index is often taken to include the phenomenon of deixis, as if the world
wished to ‘point to’ an object in order to call our attention to its meaning. Deixis is instead
the enunciational aspect of all expressed signs. The frame of the picture; the sign post of the
traffic sign. Only mental diagrams do not have deixis.
15 This is not the standard explanation of the difference. ‘Motivation’ here means orientation
towards a goal known beforehand, in casu imitation of the figurative properties of the target,
or imaging.
16 Our feelings, whether they are the emotions of the moment, the moods of the day or the
passions of our life, react to what we see as true. You do not get angry, if you do not feel
certain you are being insulted – ‘perhaps’ is not enough. Inversely, feelings can trigger beliefs,
which is why manipulation, in the unethical sense, is possible.
17 Diagrams may be the evolutionary origin of symbols and icons; gesture is often neither
conventional-instructive (as greetings are) nor motivated-illustrative (as theatrical showing
is), because its basic function may be inscribing the persons communicating in an imaginary
space of presently shared meaning. We may find the same basic function in art, which is
neither commanding nor illustrating. Roman Jakobson’s poetic function corresponds to the
meaning of this basic diagramming aspect of gesture.
18 However, the ‘source of symbolic force’ still has to qualify for the iconic cult that inscribes
it in the logic of enunciation. I have argued elsewhere (Brandt 2019, 2020) that the origin
authorities of all kinds may be the need for and the belief in protection. Money protects, and
ethnic, religious and other strong collective identities, even mafias and tribes, protect. Money
was named after Juno Moneta, the protective goddess of Rome. Temples became banks.
19 This enunciative cascade would also be illustrated by a Christian priest (S1) who blesses a
believer (S2) in the name of God.
20 See Small (1998).
180 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

21 The strange human habit of singing utterances instead of just humming or musicking by
instruments makes the connection between music and language even stronger. Syllables are
then superimposed on tones, and melodic phrases on grammatical phrases. Semantically,
linguistic meanings and musical, emotional meaning will then cover each other and create
strong impacts. Social life may have started as some sort of ongoing, non-fictive opera.
22 Quoted and discussed in Brandt (2013).
23 A professional violinist recalls her memory of hearing Carl Nielsen’s violin concerto for the
first time at age twelve: ‘The orchestra bangs out an enormous chord, and then the violin
takes over, as if it says: “I will take care of this!”, and it’s like hearing a bird you let out of
the cage and which is now just flapping around in the room’ (Grit D-H Westi, Information
Kultur, 29 May 2020). The music is heard as describing the imaginary scenario, not the other
way around.
24 The theory of conceptual blending was developed by Turner and Fauconnier, who summarized
and illustrated their approach in a very entertaining 2002 book. It was further developed
by Line Brandt (2005, 2013) and the author, who expanded the model semiotically and
pragmatically.
25 See also Oakley (2011).
26 The example discussed in Brandt and Brandt (2005).
27 Experiential domains are phenomenological regions of experiences that human cognition
distinguishes and conceptualizes distinctly. Brandt (2004) has a first account of the
phenomenon, after Sweetser (1990), who introduced the idea of basic semantic domains in
order to account for meaning differences in modality. Domain difference as such was already
presupposed in conceptual metaphor theory (Lakoff and Johnson 1980) but was never
theorized or even discussed as an interesting phenomenon: why do we naturally develop
domains of experience so that moving a categorial concept between domains changes its
meaning?
28 Line Brandt (2013) discusses these and other more complex examples and pragmatically
concludes on Wilson’s analysis of irony, that is, on ironic enunciation, that irony is not a
propositional effect, but that it takes specific situational contexts and specific intersubjective
speaker-hearer relations to achieve ironic effects.
29 Example: A and B are quarrelling. B, A’s wife, underlines her point of view by throwing the
kitchen’s crockery against the walls and on the floor, creating an impressive mess. A: ‘Nice
work, my dear!’.
30 See the analysis of semantic domains in Brandt (2004, 2019, 2000).
31 Poetry as such may therefore be seen as a sort of inverse irony, in this sense.

REFERENCES
Barthes, R. (1964), ‘Rhétorique de l’image’, Communication, 4: 40–51.
Barthes, R. (1985), The Responsibility of Forms, trans. R. Howard, New York: Hill & Wang.
Booth, W. (1974), A Rhetoric of Irony, Chicago: University of Chicago Press.
Brandt, L. (2013), The Communicative Mind: A Linguistic Exploration of Conceptual
Integration and Meaning Construction, Newcastle upon Tyne: Cambridge Scholars.
Brandt, L. and PAa. Brandt (2005), ‘Making Sense of a Blend: A Cognitive Semiotic Approach
to Metaphor’, Annual Review of Cognitive Linguistics, 3 (1): 216–49.
Brandt, PAa. (2000), ‘The Architecture of Semantic Domains: A Grounding Hypothesis in
Cognitive Semiotics’, Revista Portuguesa de Humanidades, 4 (1–2): 11–51.
PRAGMATICS AND SEMIOTICS 181

Brandt, PAa. (2004), Spaces, Domains, and Meaning, Berne: Peter Lang Verlag, Series:
European Semiotics, Number 4.
Brandt, PAa. (2013), ‘Weather Reports: Discourse and Musical Cognition’, in K. Chapin and A.
H. Clark (eds), Speaking of Music: Addressing the Sonorous, New York: Fordham University
Press.
Brandt, PAa. (2019), The Music of Meaning. Essays in Cognitive Semiotics, Newcastle upon
Tyne: Cambridge Scholars.
Brandt, PAa. (2020), Cognitive Semiotics: Signs, Mind and Meaning, London: Bloomsbury.
Fauconnier, G. and M. Turner (2002), The Way We Think: Conceptual Blending and the Mind’s
Hidden Complexities, New York: Basic Books.
Fraser, B. and K. Turner, eds (2009), Language in Life, and a Life in Language: Jacob Mey – a
Festschrift, Bingley, UK: Emerald.
Hénault, A., ed. (2019), Le sens, le sensible, le réel: Essais de sémiotique appliquée, Paris:
Sorbonne University Press.
Lakoff, G., and M. Johnson (1980), Metaphors We Live By, Chicago: University of Chicago
Press.
Oakley, T. (2011), ‘Conceptual Integration’, in J. O. Östman and J. Verschueren (eds),
Handbook of Pragmatics, Amsterdam: John Benjamins.
Perron, P. and F. Collins, eds (1989), Paris School Semiotics: I. Theory, Amsterdam: John
Benjamins.
Saussure, F. de ([1915] 1972), Cours de linguistique générale, Critical edn, ed. T. de Mauro,
Paris: Payot.
Small, C. (1998), Musicking: The Meanings of Performing and Listening, Middletown:
University Press of New England.
Sweetser, E. (1990), From Etymology to Pragmatics, Cambridge: Cambridge University Press.
182
 ­C HAPTER NINE

Gesture Studies
and Semiotics
IRENE MITTELBERG AND JENNIFER HINNELL

INTRODUCTION
As semiotic beings, we build our understanding of the discourses we participate in, and
more generally of the world around us, by integrating signs in different modalities into
dynamic meaningful ensembles. From the first exchanges between infants and caregivers,
dialogue typically involves articulations in multiple modalities such as vocalizations, spoken
or signed language, manual gestures, eye gaze, body posture and facial expressions. While
listening to our conversational partners, we observe their facial expressions; we sense
emotional qualities and inner dispositions in the ways they articulate themselves – both
verbally and gesturally, and we might respond to them – both verbally and gesturally – or
hold ourselves back. Put differently, spontaneous interactive discourse typically consists
of concerted multimodal semiotic acts of contextualized meaning-making. In this chapter,
we place the human body and its communicative behaviour at the centre of studying
multimodal processes of semiosis (e.g. Peirce 1875: CP 1.337; 1907: 5.472, 5.484). We
characterize various ways in which the human body functions as sign and sign-creator,
bringing together semiotic and related accounts that help us understand how interlocutors
gesturally indicate objects, ideas, locations, or events, and enact physical habits such as
interpersonal interaction, movement patterns or object manipulation, as well as more
abstract, yet deeply embodied, schemata of experience.
Gestures here are broadly understood as discourse-embedded, kinesic actions that are
performed with the hands and arms, head, shoulders, torso, or entire body, and have
semiotic function(s). Imagine a conversation in which a friend explains to you that her
plan for the weekend is to work. While she speaks, she performs a typing action in
mid-air, simulating typing on an imagined keyboard. From her gesture you will likely
infer that she will spend her weekend typing on an actual keyboard at an actual desk:
i.e. you will infer that, here, ‘working’ means on a computer. You may also imagine the
written text that will result from the work, as well as other ideas and states (e.g. your
friend’s future mental or emotional state when she submits her final manuscript) resulting
from her weekend of writing. This decontextualized example nonetheless shows how a
gesture can evoke not only an implied object or tool, but also the associated pragmatically
structured context of experience.
In this chapter, we focus on aspects of the study of gesture most highly relevant
to semiotics. After an overview of pioneering research in modern gesture studies, we
184 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

provide a semiotic characterization of gesture, especially as compared to more highly

codified linguistic signs. Drawing on Peirce, we then demonstrate how in gestural signs
basic semiotic modes such as iconicity, indexicality and conventionality/habit interact in
modality-specific ways. We further highlight the inherently metonymic nature of gestures
and their tendency to schematize experience. We close the chapter with a forward-
looking perspective on the field and an overview of new technologies being put to use in
semiotic gesture analyses.

BACKGROUND: THREE WAVES OF MODERN

GESTURE STUDIES
Over the past five decades, modern gesture studies has developed into a flourishing
interdisciplinary field that intersects semiotics, linguistics, anthropology, philosophy,
psychology and neurology, among other disciplines (e.g. Müller et al. 2013, 2014).
While gesture was acknowledged as part of the act of oratory by Greek and Roman
rhetoricians (Hall 2004), it wasn’t until the seventeenth century that a truly scholarly
interest in gesture emerged that motivated a descriptive, naturalistic account of gesture
(though still tied to rhetorical purposes as well). For example, Figure 9.1 features an

FIGURE 9.1 A seventeenth-century illustration of a gesture signalling antithesis (Bulwer 1644).

GESTURE STUDIES AND SEMIOTICS 185

illustration by seventeenth-century physician and philosopher John Bulwer of a gesture

signalling antithesis.1
In this section, we present the past century as three ‘waves’ that constitute modern
gesture studies. These waves are intended to be understood as time periods with unifying
developments, rather than as discrete periods. (For a more complete history of gesture
studies, see Kendon 2008, 2017; Müller 1998a; Bressem 2013a.)

The first wave: Founding treatises

Founding treatises that could be considered a ‘first wave’ of the modern field include
those of Wilhelm Wundt (1921, 1973), George Herbert Mead (1934, 1938), David Efron
([1941] 1972), Charles Morris (1946), Leroi-Gourhan (1964), Desmond Morris (1979),
and Paul Ekman and Wallace Friesen (1969) inter alia. Here we briefly introduce the
foremost contributions of several of these scholars.
German psychologist Wilhelm Wundt (1921, 1973) viewed language as an interactive
practice in which non-verbal behaviour played a role. His semiotic classification system
emerged from his studies of the hand movements of deaf people, and co-speech gestures
of Indigenous people of North America, Cistercian monks and Neapolitan speakers and
featured a tertiary division between demonstrative/pointing, descriptive and symbolic
gestures. Wundt’s classifications were foundational to later typologies (e.g. Efron’s and
Ekman and Friesen’s).
Efron’s ethnographic research ([1941] 1972) was part of the small ripple in the Zeitgeist
that ran counter to linguistic structuralism, which viewed language as an abstract system
and had little interest in gesture as a linguistic phenomenon. Efron addressed the question
of whether different cultures use gesture differently. In his study of gestures of traditional
and assimilated Italian and Yiddish speakers in New York, Efron found distinctive gestural
characteristics between the traditional groups, but also concluded that the traditional
gestures disappeared upon the speakers’ assimilation into the wider community. He thus
suggested that gestural actions are socially shared beyond the individual and that different
cultures and social groups have different patternings. Efron also presented a classification
system ([1941] 1972), later refined by Ekman and Friesen (1969).
Ekman and Friesen (1969) developed their classification system by investigating the
full repertoire of bodily movement beyond the hands (facial expressions, posture shifts,
etc.). It was based on three fundamental considerations: ‘how a [non-verbal] behaviour
became part of the person’s repertoire, the circumstances of its use and the rules which
explain how the behaviour contains or conveys information’ (1969: 49). They further
distinguished between affect – displays of emotions largely in facial expressions; adaptors,
e.g. scratching an itch or adjusting one’s glasses; regulators – now known as interactive
gestures (Bavelas et al. 1995); emblems, such as the ‘victory’-sign; and illustrators – which
portray speech content. Later typologies (see below) relied heavily on these distinctions.

The second wave: 1970s–90s

In what could be considered a second – and pivotal – wave, the pioneering research of
linguistic anthropologist Adam Kendon (e.g. 1972, 1980, 1988, 2004), psycholinguist
David McNeill (e.g. 1985, 1992, 2000, 2005) and others (e.g. Calbris 1990) has greatly
shaped our understanding of manual gestures as an integral part of thought processes,
utterance formation and communicative interaction.
186 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Kendon presents comparative semiotic studies of the role of the body in interaction,
or, to use his term, visible action as utterance (2004). His research brings together a
structuralist approach to the form and function of kinesic utterances; ethnographic
methods inspired by Efron ([1941] 1972) and Wundt (1921); and a focus on social
interaction (Goffman 1955; Schegloff 1984; Seyfeddinipur and Gullberg 2014: 2). His
studies of Indigenous sign languages of Australia (1988), gestures of Neapolitans in Italy
(1995, 2004) and gesture families (2004) are among his most significant contributions and
led to many original observations regarding different kinds of multimodal language (i.e.
co-speech gestures and signed languages) (see also subsection entitled ‘Multi-functionality
and Polysemy of Co-speech Gestures’ below and contributions in Seyfeddinipur and
Gullberg’s (2014) Festschrift in honour of Kendon’s work).
McNeill’s research into the interrelation between speech and gesture and on the
cognitive function of gesture has formed the basis of much of his work. McNeill suggests
that speech and gesture are ‘part of the same psychological structure’ (1985: 350 see
also 2005 on ‘growth points’, dynamic units of online verbal thinking that combine
imagery and linguistic content). Furthermore, McNeill’s (1992) Peirce-inspired gesture
typology now represents one of the most widely used categorization systems in gesture
research. In it, he differentiates beats – rhythmic gestures aligned with speech prosody
and used mainly as emphasis markers; deictics – pointing gestures; emblems – codified
gestures with a stable form-meaning relation, e.g. the ‘victory’ or ‘okay’ signs; iconics –
which depict a physical action or a concrete object; metaphorics, which portray abstract
entities; and cohesive gestures – repetitions of a gesture form or location across a stretch
of discourse. Moving away from assigning gestural signs to a single category, McNeill
(2005) later prefers to speak of mixing dimensions, such as iconicity and indexicality (see
also the fourth main section below and Duncan, Cassell and Levy’s 2007 Festschrift in
honour of McNeill’s work).
Geneviève Calbris (1990) offered a detailed semiotic analysis of French gestures
based on her elaborate coding system. Her account of meaning is truly multimodal;
she examines the temporal and semantic coordination of meaningful movements of
the hands, head, torso and facial mimics. She also highlighted gestures’ capacity to
abstract from the concrete and to concretize the abstract (e.g. spatial representations
of time), and how the motivation for gestural forms – i.e. through iconicity and
analogy – involves conventionalization, including both cultural practices and cognitive
schemata (Calbris 2011).

The third wave: Gesture studies as a multi-disciplinary field

In a third wave, since the late 1990s, modern gesture studies has exploded into a multi-
and interdisciplinary field addressing questions in semiotics, psychology, linguistics,
anthropology, philosophy, neuroscience and many more disciplines using a wide variety
of methodologies. Here we briefly introduce research strands that emphasize the semiotics
of gesture.
Linguistic anthropologists Charles Goodwin (1981, 2007, 2018), John Haviland
(1993, 2000), and Nick Enfield (2009, 2011, 2013) as well as interactional linguist
Jürgen Streeck (1993, 2009) have focused on the situated nature of meaning, i.e. that
conversation is anchored in the material world and is co-constructed dynamically by
participants. Importantly, their work is also grounded in the study of a multitude of
non-European languages, such as Haviland’s (1993) work on pointing gestures of the
GESTURE STUDIES AND SEMIOTICS 187

Guugu Yimithirr Indigenous community in Australia and Enfield’s (2009) collection of

studies on Lao speakers of Southeast Asia. Streeck’s ecological, conversation-analytic
approach employs microscopic analyses of interactions – e.g. car mechanics talking while
repairing a vehicle – to study how we make meaning in and gather meaning from our
environments, how we share meaning with others and how we organize our interactions
(2009: 3, 1993; LeBaron and Streeck 2000). Streeck also proposed six ‘gesture ecologies’
or ‘patterns of alignments between human actors, their gestures and the world’ (2009: 7)
(see discussion on ‘Gestural Practices of Sign Formation’ below and Streeck, Goodwin
and LeBaron 2011).
Cornelia Müller’s research, rooted in linguistic discourse analysis, has influenced
the field in many ways. Müller (1998a, 2017a) showed how Bühler’s ([1934] 1982)
Organon model of communication with its three functions – expressive, referential and
appellative – illuminates the multi-functionality of gestures. She also introduced a widely
used set of gestural modes of representation (Müller 1998a, b, 2014; see below). Her
work on multimodal metaphor has demonstrated how gestures may bring out the dynamic
nature of metaphor (Müller 2017b; Cienki and Müller 2008) and how speech, gesture
and expressive movement are temporally orchestrated in multimodal interaction and film
(Müller and Kappelhoff 2018). Furthermore, Müller (2017a, 2018) has offered insights
into motivated and conventionalized facets of both gesture and sign, e.g. by introducing
the notion of recurrent gestures (see also Ladewig 2011, 2014; Bressem 2014; Bressem
and Müller 2014).
Besides the individual scholars we have mentioned so far, there are also groups of
gesture researchers whose work is anchored in a shared theoretical perspective. Here
we name two schools that are particularly akin to semiotics. Cognitive Linguistics (CL)
was one of the first linguistic fields to embrace gesture studies, building on a shared
interest in thought processes, embodiment and multimodality (e.g. Johnson 1987; Cienki
1998, 2013; Müller 1998a; 2017b; Sweetser 1998, 2007, 2012; Taub 2001; Parrill
and Sweetser 2004; Wilcox 2004; Gibbs 2005; Núñez and Sweetser 2006; Cienki and
Müller 2008; Feyaerts, Brône and Oben 2017; Wilcox and Occhino 2017; Talmy 2018).
There is some theoretical overlap between CL and a cognitive semiotic perspective on
gesture, which similarly centralizes embodied meaning-making in the broader context of
cognitive, social and neurobiological processes, but has a stronger anchorage in semiotic
theory (e.g. Sonesson 2014; Zlatev 2015, this volume; Mittelberg 2006, 2008, 2013a;
Wolf et al. 2017, 2018).
What unites these three waves is the acknowledgement that human language, whether
spoken or signed, is embodied, dynamic and multimodal and needs to be analysed and
theorized as such (see also Liebal et al. 2013 for a multimodal approach to human primate
communication). New interdisciplinary research strands are evolving as quickly as new
technologies: e.g. the use of 3D motion capture systems, large-scale multimedia corpora
and brain imaging to study gesture. Could these methodologies be leading us to a fourth
wave? We return to this question to conclude the chapter.

GESTURES ACTING AS SIGNS

A question central to gesture research, and to this chapter, concerns the cultural, cognitive,
affective, physical, material and interpersonal factors that motivate the formation of
individual gestural signs and gesture patterns, thereby lending a certain regularity to forms
and functions of human communicative behaviours. In comparison to the Saussurian
188 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

understanding of language and linguistic signs (Saussure [1916] 1986), most of these
regularities do not rely on highly conventionalized and socially agreed-upon codes. Thus,
one of the most basic questions from a semiotic perspective is how a gesture acts as a sign.
Like any material sign carrier in any semiotic process, in order for a gesture to function
as a sign, it needs to be perceived and interpreted by a human mind or some other cognitive
system (e.g. a machine). According to prominent American philosopher Charles Sanders
Peirce (1839–1914), a ‘sign [. . .] is something that stands to somebody for something in some
respect or capacity’ (c.1897: CP 2.228; see fifth main section for semiotic theory). These basic
principles underlying semiotic processes require us to consider: In what ways and capacities
can coverbal gestures stand for something and be interpreted as such by an addressee? Do they
always clearly stand for something? And what role do the concurrent linguistic signs play?

Some prerequisites for verbo-gestural sign processes

We begin by considering some factors that condition how gestures enter multimodal
sign processes for both an addresser and their addressee. When interlocutors speak
in face-to-face interaction, they ensure that they can be heard and can hear their
interlocutors. A not-too-noisy environment and intact audio channel are prerequisites
for the successful exchange of (verbal) linguistic signs. By contrast, with the exception
of signers, interlocutors usually do not check whether they can see each other’s gestures
(e.g. when video-conferencing) and they gesture even when their interlocutors cannot
see them (e.g. on the telephone; Bavelas et al. 2008). Regarding gesture interpretation,
it is thus important to take into account that many gestures are not consciously or not
fully perceived by the addressee(s) of a multimodal utterance. People’s attention, rather,
seems to narrow in on certain articulations (e.g. speech, head movement, gaze, gesture) in
certain moments and then to change focus again to seize on others. Despite this gradient
nature of gesture perception, interlocutors are clearly attending to them to some degree,
as gestures enhance the understanding of oral discourse and the learning of new concepts,
among other functions (e.g. McNeill 2000; Goldin-Meadow 2003).
Gesture production shows a great deal of variation depending on discourse genres,
personal styles, cultural practices and other factors (e.g. Müller et al. 2013, 2014).
Speakers can be more or less aware of the gestures they make while talking. The fact that
speakers are often not aware of their gestural behaviour can reveal less monitored aspects
of multimodal semiosis during communication (Sweetser 2007). As meaningful bodily
actions influenced by cognitive and affective states, gestures provide valuable insights
into the physical grounding, emotional dimensions, and socio-cultural situatedness of the
semiotic processes humans rely on for thinking, imagination and communication (e.g.
Streeck, Goodwin and LeBaron 2011; Müller 2017b).
There are situational contexts in which gestures may become the focus of joint attention
and joint intentionality (e.g. Tomasello 1999) and thus the focus of the communicative
action of all participants involved, which makes it very likely that they fulfil semiotic
functions for both the gesturer and the addressee(s). Such contexts include expert
explanations (e.g. Streeck 2009) and language teaching (e.g. McCafferty and Stam 2008).
Among the different kinds of gestural signs, pointing gestures, or deictics (McNeill 1992),
in particular, need to be fully perceived by the addressee(s). Speakers use such indexical
gestures to purposely direct their addressee’s attention and to indicate a specific object,
location or event, or less specific targets such as broad directions (e.g. north, south) or
regions in space (e.g. a mountain range) (Kita 2003; Fricke 2007; Goodwin 2007; see
also discussion on ‘Contiguity – Indexicality – Metonymy’ below).
GESTURE STUDIES AND SEMIOTICS 189

Emblematic gestures (emblems, McNeill 1992, or quotable gestures, Kendon 2004)

are also gestural signs that interlocutors need to attend to for a communicative exchange
to be successful. Within the spectrum of less versus more conventionalized, or ‘language-
like’, gestural behaviour, emblems exhibit a high degree of conventionalization on the
basis of well-formedness conditions and culturally codified meanings (see McNeill (2005)
for the so-called ‘Kendon-continuum’ encompassing gesticulation-pantomime-emblems-
sign language). An example of an emblem is the ‘thumbs-up’ gesture signifying (in many
Western cultures) approval and positive appraisal, formed with an outstretched thumb
extended about 90 degrees away from the fingers, which are curled into a fist (shown in
Figure 9.2). While it can signal approval without relying on the semantics of the concurrent
speech, the interpreter still needs to consider the immediate discourse and situational
context to understand what the speaker-gesturer is approving of. In Figure 9.2, we see
a thumbs-up emblem produced with both hands by US Vice President Kamala Harris to
give positive feedback to her student audience.2
In light of the highly conventionalized and codified nature of emblems, they seem
to be, strictly speaking, the only type of gesture that can be adequately described with

FIGURE 9.2 Thumbs-up emblem signalling approval. US Vice President Kamala Harris gives
a thumbs up to students at Miller Elementary School, Dearborn, Michigan, Monday, May 6,
2019. Copyright: Associated Press 2019.
190 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Saussure’s model of the linguistic sign ([1916] 1986). That is, reference is afforded via
coded form-meaning pairings, i.e. between a signifier that is unequivocally assigned to a
signified as in the thumbs-up gesture signifying approval. For the same reason, emblems
(but not all gestures) may count as symbols in Peirce’s understanding of the term (see
below on ‘Gesture and semiotic theory’). While popular understandings of the term
gesture are often associated with emblems, they make up only a sub-group of gestures
and subsequently also of gesture research (for cross-cultural uses of emblems, see, e.g.
Calbris 1990; Brookes 2004; Kendon 2004; Müller and Posner 2004; for an overview
of research on emblems, see Teßendorf 2013). This chapter is mainly concerned with
spontaneously produced co-speech gestures that show a larger and more varied range of
forms, functions and degrees of conventionalization.

Multi-functionality and polysemy of co-speech gestures

Gestures typically fulfil more than one communicative function and therefore often defy
neat categorization. Hierarchical understandings of semiotic processes (Jakobson 1960;
Peirce (1903: EP 2.289–99); Bühler [1934] 1982) have been adapted to account for
the multi-functionality of gestures, which often combine in varying degrees iconicity,
indexicality and conventionality (Peirce c.1895: CP 2.302; McNeill 1992, 2005),
and pragmatic and interactive functions described beyond Peirce-based paradigms
(Bavelas et al. 1995; Müller 1998a; Kendon 2004). Depending on local pragmatic
forces and contextual factors, one of the modes comes to the fore, thus determining
the predominant function of a given gestural sign. The non-dominant functions still
contribute to the overall meaning of the utterance, e.g. referential functions may
layer with the stance of the speaker towards the content of the utterance (see Müller
(1998a, 2014), drawing on Bühler; Mittelberg (2013a) and Mittelberg and Waugh
(2009), drawing on Jakobson and Peirce; and Debras (2017) and Hinnell (2020) on the
multimodal expression of stance).
The following example from a teaching context illustrates how gestures act as signs in
multimodal utterances. When saying ‘there is the main verb’, the speaker in Figure 9.3
(adapted from Mittelberg 2013a), a linguistics professor, is drawing the students’
attention to the verb form ‘taught’ written on the blackboard behind him by creating
a gestural index with his right arm and hand. Here the linguistic index ‘there’ and the
gestural index pointing at the written form ‘taught’ jointly establish reference; they are
semantically integrated (e.g. Cooperrider et al. 2021). Meanwhile, the speaker’s left hand
exhibits basic form features of a receptacle representing ‘the main verb’ at a higher level
of abstraction. Without considering the speech content, we would not know that the
cupped hand does not portray a physical object, but, rather, a grammatical category.
By representing an abstract entity through a physical structure, this form qualifies as a
metaphoric gesture (McNeill 1992; Sweetser 1998; Cienki and Müller 2008; see also
section ‘Gesture and Semiotic Theory’ below).
Notably, these gestures exhibit both similar and different semiotic characteristics
and functions within the multimodal utterance. Both are polysemous sign carriers
that depend on linguistic and other contextual information to assume a specific,
contextualized meaning; yet, different degrees of context-dependence are at work.
The meaning of indexical signs is categorically highly context-sensitive, a fact well
established for pronouns, demonstratives and other function words in language (or
shifters, Jakobson 1957). Pointing gestures similarly depend more fully on the linguistic
GESTURE STUDIES AND SEMIOTICS 191

FIGURE 9.3 Pointing gesture (‘there is’) and metaphoric container gesture (‘the main verb’).

and extralinguistic context and cause the attendee’s attention to shift from the hand
itself to the entity pointed at. By contrast, iconic and metaphoric gestures typically
exhibit form features which, taken by themselves, evoke content, including embodied
schemata (see Johnson 1987; Cienki 1998, 2013; Mittelberg 2006, 2018). Focusing
only on the gestural part of the utterance above, we see that the gestural index refers
to something external to the speaker’s body, while the container gesture also is what it
signifies, a physical structure in the form of a receptacle (e.g. Merleau-Ponty 1962: 216;
Mittelberg 2013b: 340). In the speech content, we observe another difference between
these two multimodal sign processes. Whereas the referent of the linguistic deictic
expression ‘there is’ can only be identified with a simultaneous gestural index, the noun
phrase ‘the main verb’ could be understood without the container gesture illustrating
it. Additional semiotic support comes from the written form ‘taught’ that represents an
exemplar of main verbs. All in all, the speech content, the visual information on the
board and the two coinciding gestures contextualize each other, thus co-constituting a
multimodally performed semiotic act.
Thus far, we have introduced the semiotic complexity that characterizes dynamic
multimodal sign processes. Coverbal gestures are typically polysemous and multi-
functional signs that assume their local meaning based on their own semiotic characteristics
and in conjunction with concurrently uttered speech and other contextual factors. While
questions of referentiality (e.g. Deacon 1997) are crucial to understanding how gestures
act as signs, this chapter cannot provide a full treatment of this complex issue (but see
‘Gesture and Semiotic Theory’ below).
192 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

BODILY SEMIOTIC ACTION AND

MULTIMODAL LANGUAGE
Co-speech gestures are often described as part of visual communication or visual
language (e.g. Sandler, Gullberg and Padden 2019). While they clearly constitute the
visible, observable part of language use in interaction, a semiotic characterization of
gesture needs to account for the fact that they are, compared to static visual signs such as
drawings, paintings or hieroglyphs, more than merely visual in nature. Here, we address
the role kinesic action plays in the semiotics of gesture and discuss connections to signed
languages, thus broadening the scope of multimodal language.

Kinesic, visual and imagined facets of gestural signs

Kendon’s (2004: 7) term visual action as utterance highlights the fact that gestures
are dynamic bodily actions. While utterance stresses gestures’ affinity with verbal
expression, here the term suggests that gestures can also be seen as utterances in their
own right (Seyfeddinipur and Gullberg 2014). Drawing on Jakobson’s notion of motor
signs (1972: 474), Mittelberg (2019a) recently introduced the notion of visuo-kinetic
signs to capture the fact that gestures – as well as the various articulations in signed
languages – are conditioned by the kinesic and sensory affordances of the speaker’s body
and its interactions with the environment. In this understanding of gestures, semiosis
(Peirce 1907: CP 5.472, 5.484) and embodiment (Varela et al. 1991) are intimately
linked: experientially grounded embodied schemata and force dynamics (Johnson 1987)
are assumed to underpin both the mindful production and interpretation of bodily signs,
thus allowing for (inter-)subjective understanding (see Mittelberg 2013a on the exbodied
mind). Honing in further on bodily articulations, Boutet, Morgenstern and Cienki (2016)
proposed a fine-grained kinesiological account of gestures in their examination of gestures
reflecting grammatical aspect.
Gestures’ deep grounding in physical actions and social interactions explains why
speakers readily and effortlessly use their hands to hold or move around real or virtual
objects (see, e.g. Kita 2000; studies in Streeck, Goodwin and LeBaron 2011). Participants
further co-construct contextures of action with their conversational partners (Goodwin
2011, 2018). In principle, however, a gesture is a gesture – i.e. a reduced and more
schematic bodily action with semiotic function – precisely because gesturers typically only
pretend to be manipulating objects, touching surfaces, tapping on someone’s shoulder,
etc. (see Clark (2016) on depictions; Hostetter and Alibali (2008) on simulated actions
in gesture and Müller (1998a/b, 2014) on acting). When speakers do ‘pull’ real persons,
physical objects or tools into their gestural actions, these elements become part of the
respective visuo-kinetic sign process and thus themselves fulfil semiotic functions: they
stand for someone or something else. Although the verbo-gestural co-expression of
propositional content has always been a focus of gesture research (e.g. McNeill 1992;
Kendon 2000, 2004), the action affinity of gestures has also offered insights into how
gestures may do something in their own right. Examples include brushing away an
argument (e.g. Bressem and Müller 2014), fending off a rhetorical attack (e.g. Wehling
2017) or inviting an interlocutor to take the floor (e.g. Holler and Bavelas 2017).
Despite gestures’ visual and bodily mediality, a challenging factor in gesture
interpretation and analysis resides in the fact that the ‘semiotic material’ we are looking
at often provides rather sparse and short-lived imagery (e.g. Arnheim 1969). Gestures
GESTURE STUDIES AND SEMIOTICS 193

often consist only of quickly performed motion-onsets or furtive schematic figurations

abstracted from, for example, the full action routine, object or spatial constellation
(e.g. Müller 1998a, 2014; Mittelberg and Waugh 2014). Moreover, while a gestural
form description starts with the observable physical components such as body posture,
hand shape, palm orientation, finger configuration, and the position and action of
gesturing hands (e.g., Bressem 2013b), gestural sign carriers also tend to involve facets
that are not directly observable, yet are also signifying. These may include the imaginary
interlocutors, objects or surfaces mentioned above, but also invisible movement traces
or points set in gesture space, all of which need to be metonymically inferred from the
visible, physical elements and actions (Mittelberg 2019a; see also below). For instance,
the typing gesture described earlier necessarily implies an imagined keyboard, which
is not itself gestured. A thorough gesture analysis thus needs to include elements of
multimodal sign processes that are not visual, or visible, but still contribute to a gesture’s
form, meaning, pragmatic function and kinesthetic feel.

Gestural practices of sign formation

We now turn our attention to how communicating hands create gestural signs. Rudolf
Arnheim gives a succinct impression of the ‘intelligent’ abstractions speakers perform
with their gestures:

By the very nature of the medium of gesture, the representation is highly abstract.
What matters for our purpose is how common, how satisfying and useful this sort of
visual description is nevertheless. In fact, it is useful not in spite of its spareness but
because of it. Often a gesture is so striking because it singles out one feature relevant to
the discourse. [. . .] The gesture limits itself intelligently to emphasizing what matters.
(Arnheim 1969: 117)

Spontaneously produced gestures like the ones described by Arnheim, particularly those
that portray an aspect of an object, action or idea, may take shape through employing
different kinds of semiotic practices. Kendon (2004), for example, distinguishes
between different techniques of representation, namely modelling a body part to stand
for something else, enacting certain features of an action pattern, or depicting objects
through hand movements recognized as sketching or sculpting the shape of something.
Müller (1998a, b) introduced four modes of representation in gesture, drawing on tools,
media and mimetic techniques stemming from the visual arts: drawing (e.g. tracing the
outlines of a picture frame); molding (e.g. sculpting the form of a crown); acting (e.g.
pretending to open a window); and representing (e.g. a flat open hand stands for a piece
of paper; Müller 2014).
Some gesture scholars specifically differentiate between gestures that carefully depict
a particular space or referent object from those gestures that seem to reflect a concept
or an idea, for example, via metaphor (e.g. Calbris 1990; McNeill 1992; Fricke 2012).
Streeck (2009: 151–2) distinguishes between depicting (e.g. via a gesture portraying a
physical object) and ceiving (e.g. via a gesture conceptualizing a thematic object, such
as an experience or idea), among others, attributing the latter mode to a more self-
absorbed, conceptually driven way of finding a gestural image for an emerging idea. Clark
(2016) presented a detailed multimodal account of three main modes of communication
including depicting, describing (with arbitrary symbols such as words) and indicating
194 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

(using indices such as pointing and placing). See Ferrara and Hodge (2018) for a review
of the techniques addressed in this section.

Comparative semiotics: Co-speech gesture and sign language

Reflecting on semiotic action and the multimodal nature of language leads us to consider,
though briefly, how co-speech gestures relate to signed languages. In a historical context
in which sign languages were long not accorded the same status as spoken languages,
original accounts of signed languages comprised phonological descriptions in the
structuralist approach that characterized linguistics at the time (e.g. Stokoe [1960] 2005
on ASL). The theoretical ‘rapprochement’ (Wilcox and Occhino 2017: 111) between the
study of signed languages and gesture studies has occurred over the last twenty-five years
(Kendon 1988, 2008; Armstrong, Stokoe and Wilcox 1995), a period in which cognitive
approaches have come to play a greater role in studies of signed languages (Wilcox and
Occhino 2017). This, in turn, has brought into focus the semiotic mechanisms that play a
role in both gesture and signed languages.
Gestures and signs in signed languages share similarities, yet also differ, in how they
act as signs (e.g. Goldin-Meadow 2003; Liddell 2003; McNeill 2005; Sweetser 2009;
Perniss, Thompson and Vigliocco 2010; Goldin-Meadow and Brentari 2017; Müller
2018; Perniss 2018). Regarding their semiotic status, signed languages compare to
spoken languages in that they are highly conventionalized sign systems, each with a coded
phonology, lexicon and grammar (e.g. Wilcox and Occhino 2017). With respect to their
semiotic resources, gesture and signs share largely the same set of kinesic articulators and
the same primary lieu of meaning-making, namely, the body-centred, three-dimensional
signing/gesture space. However, in contrast to spoken discourse and concurrent ‘co-
speech’ gestures, sign languages are entirely visuo-kinetic multimodal sign systems. The
hands do most of the semiotic work, but in a composite manner with other articulations
such as facial expressions. In every instance of signed interaction, several symbolic signs
thus need to be simultaneously visually perceived by the addressee and semantically
and grammatically integrated. The study of emerging sign languages, or homesign (e.g.
Morford and Kegl 2000) can help us better understand how physical semiotic resources
are dynamically coded by their users and thus develop into a multimodal language.
Sign languages also comprise gestural strata and components (Liddell 2003; Kendon
2008; Wilcox and Occhino 2017). For example, some signs enter a sign language as a
gestural form, e.g. as the substrate for lexical signs (Janzen and Shaffer 2002), which
can grammaticalize to grammatical signs, such as is the case with modals (Wilcox and
Shaffer 2006). By the same token, co-speech gestures may exhibit various degrees of
grammaticalization (e.g. Fricke 2012; Mittelberg 2017b; Zima and Bergs 2017; Müller
2017a, 2018; Harrison 2018; Hinnell 2019, 2020; Ladewig 2020).
There are also less codified articulations in sign language that may be ascribed a
gestural quality. Constructed action, for example, is a widely used strategy for reporting
narratives and for event enactment in which signers ‘encode different perspectives,
either simultaneously or sequentially relaying information about referents by pivoting
their gaze, shoulders, or bodies or by changing facial markers’ (Kurz, Mullaney and
Occhino 2019: 90). Some sign linguists have also argued that certain sign forms are
blends of conventional signs and gestural locations, given that the locations are not
codified and are therefore part of gesture (e.g. a point in space to index a discourse
referent) (Liddell 2003).
GESTURE STUDIES AND SEMIOTICS 195

We now turn to the basic semiotic modes that are at work in both co-speech gestures
and signed language.

GESTURE AND SEMIOTIC THEORY

In this section, we hone in on one of the fundamental semiotic theories applied to gesture
and multimodal sign processes, namely, Peirce’s semiotic. We also draw connections
to Saussure and Jakobson, as well as to the fields of cognitive linguistics and cognitive
semiotics, which share basic assumptions with Peirce regarding embodied patterns of
experience and expression. A Peircean perspective on multimodal interaction can account
for both highly symbolic sign systems, such as spoken and signed languages, and less
codified, dynamic visuo-spatial modalities, such as coverbal gestures. In Peirce’s pragmatist
doctrine of signs, semiosis and cognition are tightly intertwined: ‘we think only in signs’
(Peirce c.1895: CP 2.302). Given the central question in gesture research of how gestures
partake in expressive and cognitive action during communication, Peirce’s semiotic has
informed work done in several intersecting fields. These include cognitive semiotics
(e.g. Mittelberg 2006, 2008, 2013a, 2019a, b; Sonesson 2007, 2014), linguistics (e.g.
Fricke 2007, 2012; Andrén 2010; Bressem 2014), anthropology (e.g. Haviland 1993,
2000; Enfield 2009, 2011, 2013), philosophy (Maddalena 2015), psycholinguistics (e.g.
McNeill 1992, 2005), psychology (Clark 1996, 2016), and social neuroscience (e.g. Wolf
et al. 2017).

PEIRCE’S TRIADIC SIGN MODEL APPLIED TO GESTURE

Peirce’s widely cited definition of the sign reads as follows:

A sign, [in the form of a] representamen is something which stands to somebody for
something in some respect or capacity. It addresses somebody, that is, creates in the
mind of that person an equivalent sign, or perhaps a more developed sign. That sign
which it creates I call the interpretant of the first sign. The sign stands for something,
its object. It stands for that object, not in all respects, but in reference to some sort of
idea, which I sometimes called the ground of the representamen.
(Peirce c.1897: CP 2.228; italics in original)

The term Representamen refers to the material form the sign takes, e.g. a spoken word
or a gesture; Interpretant refers to the response evoked in the mind of the sign receiver.
Sign interpreters are seen as agents actively involved in meaning-making. Whether
one recognizes a gesture as a sign and what one sees in it depends on one’s semiotic
history, notably, socio-cultural knowledge, code(s), and other patterns of perception and
interpretation, including both coded and less symbolic gestural practices (e.g. Mittelberg
2006, 2019b; Fricke 2007; Enfield 2013; Wolf et al. 2017).
Peirce’s wide understanding of a semiotic Object encompasses both physical objects/
actions and abstract entities/processes, including qualities, feelings, relations, concepts,
mental states and possibilities. As is true for any semiotic material or system, in gesture,
too, the medium-specific affordances of the hands, arms and other physical articulators
determine what may be represented or indicated, and how (see discussion on ‘Gestural
Practices of Sign Formation’ above). Imitating manual typing on a keyboard comes
196 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

naturally. Whereas a hand can readily become a small receptacle (e.g. Figure 9.3), huge
Objects such as buildings need to be cognitively and physically brought down to human
scale in order to describe them with gestures. Some Objects thus do not lend themselves
well to depiction or enactment via gesture; colours, for example, may be gesturally
represented in terms of their vividness rather than actual colour (for how Peirce’s notions
of the immediate and dynamic Object apply to gesture, see, e.g. Mittelberg (2006, 2013a)
and Enfield (2013)).
Peirce’s concept of the Ground highlights the fact that sign vehicles represent Objects
not in their entirety, but only with respect to some perceptually salient and/or pragmatically
relevant aspect. By entering the semiotic process, these abstracted and foregrounded
features of the Object function as the Ground of the Representamen (Peirce c.1897: CP
2.229; Sonesson 2007: 40). Accordingly, gestural sign processes have been devised based
on different semiotic grounding mechanisms following Peirce (e.g. Mittelberg 2013a;
Mittelberg and Waugh 2014; Sonesson 2014). In what follows, we will discuss how the
basic Representamen-Object correlations icon, index and symbol (Peirce 1893: CP 2.275)
have been described in relation to coverbal gestures.

Similarity – iconicity – metaphor

Similarity generally builds the basis for iconic signs. According to Peirce (1903: CP
2.276), ‘icons have qualities which “resemble” those of the objects they represent, and
they excite analogous sensations in the mind’. Although there is a visual bias in the term
icon, it encompasses a multimodal understanding of iconicity and thus also includes those
‘sensations’ that cause something to feel, taste, look, smell, move, or sound like something
else. A gesture with a highly iconic ground gives a partial image of its Object based on a
perceived or construed similarity (e.g. Andrén (2010), Lücking (2013), Mittelberg (2014)
and Sonesson (2014) for more theoretical insights into gestural iconicity). In gesture,
similarity may reside in structural resemblance, which has also been described in terms of
isomorphism (e.g. Kita 2000; Mittelberg 2006; Fricke 2007). Similarity may also motivate
how speakers foreground an expressive quality of a movement or a phase of an action
routine (e.g. Müller 1998b; Mittelberg 2019a; also below). For example, in McNeill’s
(1992: 12) well-known example of a speaker retelling the action of a cartoon character
in conjunction with the utterance He grabs a big oak tree and he bends it way back, the
gesturer first performs a grabbing action and then a pulling backward action with his right
hand and arm. As McNeill (2005: 6–7) points out, ‘the gesture has clear iconicity – the
movement and the handgrip; also a locus (starting high and ending low) – all creating
imagery that is analogous to the event being described in speech at the same time’. In
the McNeillean tradition, one focus of analysis has been on how speech and gesture
encode different aspects of motion events, e.g. path and manner of movement, and how
these strategies reveal patterns that correlate with typological differences (e.g. studies in
Duncan, Cassell and Levy 2007; for an overview of work on iconic or representational
gestures, see Mittelberg and Evola 2014).
Iconic gestures also reflect the viewpoint strategies speakers adopt when describing
events or experiences (e.g. McNeill 1992). A speaker can describe a scene from the inside,
imitating the actions of a character (character viewpoint) or adopt an external viewpoint on
the event and describe, for example, the motion path of a character (observer viewpoint).
Speakers may also express multiple viewpoints on the same experience by employing
several bodily articulators (see Calbris (1990) on body segments; Dudis (2004) on body
GESTURE STUDIES AND SEMIOTICS 197

partitioning in ASL; Parrill (2009) on dual viewpoint; and Stec (2012) for an overview).
From a semiotic perspective, viewpointed iconic gestures are indexically conditioned, as
they reveal the conceptual anchorage and/or physical perspective from which the gestural
sign is created (e.g. Sweetser 2012; Mittelberg 2017a).
Peirce’s distinction between three subtypes of icons (i.e. hypoicons; Peirce 1903: CP
2.276) – images, diagrams (i.e. icons of relations) and metaphors (implying a parallelism;
Peirce c.1897: CP 2.228; Mittelberg 2014) – captures the multi-dimensional nature
of gestural imagery. Image iconicity resides, for instance, in iconic gestures depicting
a character’s actions, as in McNeill’s tree example above. Full body enactments of an
action or object – e.g. a child ‘becoming’ a helicopter, with his torso representing the
body of the helicopter and his arms the rotating blades (Bouvet 1997: 17) – show more
iconic form substance than a virtual outline of an object, such as a picture frame sketched
in the air (Müller 1998a,b). To further illustrate these tendencies, consider the gestural
portrayal of a painting by Paul Klee in Figure 9.4 (‘Dance of a Mourning Child’, adapted
from Mittelberg 2013b). As the video still in the centre image reveals, the speaker adopts
character viewpoint when imitating the figure in the painting by mirroring its posture,
tilted head and downward gaze.3 Here, the speaker’s entire body functions as an image
icon with a high degree of iconicity. By contrast, the motion-capture plot on the right
(Natural Media Lab, RWTH University, Aachen) highlights the balanced image-iconic
structures the speaker created with both hands. Visualizing and freezing the motion traces
produces static iconic signs of the eye slits (at the very top of the mocap plot) and the
heart-shaped mouth, which the speaker drew on her own face, as well as the stretched-out
arms and the skirt flowing around her legs (for a detailed image schema analysis of the
painting and the multimodal description, see Mittelberg 2013b).
Diagrammatic iconicity underpins gestures that trace connections between two
or more locations in gesture space to schematically represent, for instance, kinship
relations (Enfield 2009), syntactic structure (Mittelberg 2008) or a travel itinerary, as
in Figure 9.5. Consisting of visualized motion traces, this gestural diagram is a digitally
engendered iconic sign of a diagrammatic gesture (Schüller and Mittelberg 2016). The
speaker on the right summarizes the itinerary of a train trip through Europe she and her

FIGURE 9.4 Painting (P. Klee, Dance of a Mourning Child 1922 as stimulus; speaker’s body as
icon of figure in painting [video still]; iconic figurations [MoCap plot]).
198 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 9.5 Motion-capture gesture diagram of travel itinerary: Stockholm, Prague, Kiev, etc.

conversation partner agreed upon: ‘I think then we’ll go like this, you know, from (-)
there to there across down and then to there and then back home again, no?’4 Given the
high number of indexical function words in the verbal utterance, this synopsis can only
be fully understood in the context of the incrementally emerging diagram, the preceding
multimodal discourse and aspects of the common ground shared by the interlocutors (e.g.
Holler and Bavelas 2017; for a detailed analysis, see Mittelberg and Rekittke 2021).
An example of metaphor iconicity motivating a gesture is the open cupped hand
discussed earlier (Figure 9.3) that represents an abstract category in the form of a small
container. In Peircean terms, this metaphoric gesture represents a parallelism between an
abstract semiotic object and a physical Representamen (c.1897: CP 2.228). According to
Müller and Cienki (2009), this gesture is an instance of a monomodal metaphor, for the
concurrent linguistic expression ‘main verb’ is not metaphorical, but technical in nature
(Mittelberg 2008). Put into cognitive linguistic terms, the speaker here enacts the primary
metaphor categories are containers, which is deeply rooted in experience (Grady 1997).
Here, the embodied image schema container (Johnson 1987) provides structure that
motivates the gestural form. (For more research on metaphor in gesture, mainly from
a cognitive linguistic perspective, see, e.g., Calbris 1990, 2011; Cienki 1998; Sweetser
1998, 2007; Parrill and Sweetser 2004; Núnez and Sweetser 2006; Cienki and Müller
2008; Mittelberg 2008; Müller and Cienki 2009; Müller 2017b; Pagán Cánovas et al.
2020; for metaphor in signed languages: see, e.g., Taub 2001; Wilcox and Occhino 2017.)

Contiguity – indexicality – metonymy

Contiguity relations are born out of a factual (physical or causal) connection between
the semiotic Object and the sign carrier, notably physical contact, but also temporal and
spatial proximity or distance. Contiguity underpins signs with a predominant indexical
GESTURE STUDIES AND SEMIOTICS 199

Ground (Peirce 1901: CP 2.306; Sonesson 2014: 1992). According to Peirce (c.1897:
CP 2.228), ‘an Index is a sign which refers to the Object that it denotes by virtue of
being really affected by that object’. As demonstrated above (see discussion on ‘Gestures
Acting as Signs’, also Figure 9.3), the spatial orientation of pointing gestures is indeed
affected by the location of their target Object. It is through the very act of pointing that
the Object is established via a visual vector. Whereas the highly indexical gesture shown
in Figure 9.3 clearly points at the intended referent located nearby, thus instigating a
contiguity relation to it, other instances of pointing may be more ambiguous, evoking
several possible meanings (e.g. Goodwin 2014), including abstract pointing (McNeill,
Cassell and Levy 1993) and pointing at previously produced signs (e.g. Fricke 2007).
Form variants of the indicating hand and arm have been shown to fulfil various pragmatic
functions (cf. Clark 2003; Enfield, Kita and De Ruiter 2007; Talmy 2018; Hassemer and
McCleary 2020). While pointing gestures can be observed across languages and cultures,
field work in various parts of the world has demonstrated that pointing practices and
conventions are not universal, but rather vary considerably and also include non-manual
pointing (cf. Haviland 2000; Kita 2003; Wilkins 2003; Kendon 2004; Enfield 2009;
Cooperrider, Slotta & Nuñez 2018; Cooperrider et al. 2021).
We emphasize here that besides pointing, all gestures are, in principle, indexical to a
greater or lesser extent, in that their meaning and pragmatic functions always depend on
the speaker’s body’s spatio-temporal anchorage in a real or imagined context (Sweetser
2012). Indexicality thus conditions any gestural act of meaning-making. It also resides
in interactive gestures (Bavelas et al. 1995) and environmentally coupled gestures
(Goodwin 2007). Furthermore, indexicality tends to interact with iconicity, not only
regarding viewpoint (as discussed above), but also in gestural diagrams (Mittelberg and
Rekittke 2021), gestural enactments of contrast (Hinnell 2019, 2020; see also below) and
metonymy in gesture (Mittelberg and Waugh 2014), for instance.
Jakobson’s (1956) balanced theory of metaphor and metonymy directly builds on
Peirce’s notions of similarity and contiguity and shows how these fundamental modes
of association and signification interact in linguistic and visual signs. Jakobson’s work,
particularly his distinction of inner and outer contiguity (Jakobson and Pomorska 1983),
is well suited to illuminate the experientially motivated and inherently metonymic nature
of gestures (addressed in the third section above; see also Mittelberg and Waugh (2009,
2014). Here we only briefly treat the main principles.
Jakobson understood inner contiguity (giving rise to internal metonymy) to underlie
part-whole relationships (e.g. of a physical structure such as the human body). The
cupped hand shown in Figure 9.3 metonymically represents a solid container. Brief action
onsets or schematic movements may further evoke the full action routine in question. For
example, Mittelberg (2019a) describes a speaker performing a swimming gesture that is
metonymic in more than one respect: a stylized and reduced iconic movement of her arms
evokes a full-fledged swimming action, with the leg movements left out. Outer contiguity
(giving rise to external metonymy) holds among elements that are spatially, temporally or
pragmatically correlated within a situational or semiotic context (Jakobson 1956). Let us
consider Figure 9.6, which shows the same linguist as in Figure 9.3 continuing to explain
the relation between main verbs and auxiliaries. Here, he stresses that the auxiliaries
‘have, will, being and been [. . .] must all belong to some subcategory’.
To interpret this gesture, we must first see the speaker-gesturer as a dynamic image
icon of someone holding something (based on internal metonymy) and then draw on
the outer contiguity relation (contact/adjacency) between the vertical palms of the
200 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 9.6 Metonymy: Contiguity relations between hands and the virtual object they seem
to be holding. Metaphor: Imaginary object represents the abstract entity (‘subcategory’).

hands (i.e. the source of the metonymic mapping in cognitive linguistic terms) and the
virtual object between them (i.e. the metonymic target). Here we witness an instance
of cross-modal metonymic inferencing, for the verbal expression ‘subcategory’ causes
our attention to shift from the manual action of holding something (not referred to
verbally) to the imaginary object being held. This gesture thus enacts a frame-metonymy
(Dancygier and Sweetser 2014) and, more specifically, the metonymic mapping action-
for-object involved in action or presentation-for-presented (e.g. Panther and Thornburg
2003). Further examples of external metonymy in visuo-kinetic signs include speakers
presenting reified discourse contents to be imagined on a palm-up open hand or between
a hand’s index and thumb (Mittelberg 2006). (Re-)creating contact with the environment,
these kinds of gestures exhibit comparably low degrees of indexicality, yet ‘point’ to the
virtual object, tool, surface or space ‘at hand’. Depending on their respective shape, size
and affordances, the (imaginary) elements resonate in a gesture’s particular hand shape
and action pattern. Hence, such signs hence are – to return to Peirce’s notion of an
index – dynamically affected by the Object.
Finally, to arrive at the abstract grammatical category of the multimodal explanation
in Figure 9.6, it is crucial to realize that metonymy leads into metaphor (Mittelberg and
Waugh 2009, 2014) in such gestural actions that intuitively draw on familiar scenes of
GESTURE STUDIES AND SEMIOTICS 201

experience (Fillmore 1982). In short, the imaginary object functions as both the target of
the metonymic mapping as explained above and the source of the interlaced conceptual
metaphor ideas are objects or categories are containers (Lakoff and Johnson 1980;
Dancygier and Sweetser 2014). Hence, the metaphoric target, i.e. the subcategory,
is construed as a tangible object that can be intersubjectively shared with the student
audience (see Mittelberg 2019a; for comparable processes in signed languages see, e.g.,
Taub 2001; Wilcox and Occhino 2017).

Conventionality – symbolicity – conceptual schemata

According to Peirce (1903: CP 2.249), ‘[a] symbol is a sign which refers to the Object that
it denotes by virtue of a law’. Strictly speaking, only codified emblematic gestures are, as
noted earlier, truly symbolic signs, which is also why Saussure’s ([1916] 1986) model of
the linguistic sign seems far too rigid when it comes to gesture. Rather, different levels and
sources of both conventionality and iconicity tend to interact in communicative actions
of the human body, thus driving the emergence and routinization of motivated signs
(e.g. Calbris 1990 on cultural clichés in French gestures; see also Posner 2004). These
gesture-based insights are valuable in the context of the long-standing debate regarding
the structural integration of iconicity in predominantly symbolic linguistic signs. As
Jakobson ([1966] 1990) claimed in his work based on Peirce, and as subsequent work on
spoken and signed languages has clearly shown, iconicity is a general feature of language
at all levels of linguistic structure (Jakobson and Waugh [1979] 2002; Perniss, Thompson
and Vigliocco 2010; also above). As far as symbolic indices in gesture are concerned, they
occur mainly in the form of pointing, which, as Peirce (c.1903: CP 2.262) maintained,
may build a constitutive part of a (symbolic) proposition (e.g. Stjernfelt 2014).
Compared to law-like conventionality, Peirce’s notion of habit (e.g. Peirce 1902:
CP 2.170; Nöth 2016) seems particularly suited to accounting for gradually routinized
correlations between recurring gestural forms and their functions (Mittelberg 2006).
Peirce’s universal categories of perception and experience have been shown to illuminate
the multi-dimensionality of gesture (e.g. Fricke 2007, 2012) and, particularly, the habit-
based emergence of different kinds of gestural patterns and schematicity: Mittelberg
(2019b) distinguishes between gestural enactments of habits of feeling (Firstness), habits
of acting (Secondness) and habits of thinking (Thirdness) (see also Sonesson 2014; Nöth
2016). Accordingly, Thirdness-laden properties of gestures should allow interpreters to
discern some of their semantic/pragmatic functions without considering the concurrent
speech (see also Wolf et al. (2017) on perceived conventionality).
Embodied image schemas and conceptual metaphor (e.g. Johnson 1987) mediate
between experience, thought and expression and have thus been shown to motivate the
schematic form and meaning of many iconic and metaphoric gestures (e.g. Cienki 1998,
2013; Mittelberg 2008, 2013a/b, 2018, 2019b; Ladewig 2011; Wehling 2017). Besides
the basic schemata container (Figure 9.3) and object (Figure 9.6) illustrated above, and
balance, to be discussed below, further schemata prone to manifest – and interact – in
gesture include path, boundary and cycle, the spatial relation schemata up-down, near-
far and left-right (Lakoff and Johnson 1999) and force schemata such as resistance and
blockage (Johnson 1987; Talmy 1988). Compared to these abstract schemata structuring
certain gestures in adult speakers, the mimetic schemas introduced by Zlatev (2005) – such
as jump, kick, grasp, push and hit – are anchored in specific physical actions. They play
a fundamental role in the cognitive and linguistic development of children and ‘help
202 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

explain most literally the grounding of both communication and thought through action
and imitation’ (Zlatev 2014: 2; see also Andrén 2010; Cienki 2013).
It is precisely because most gestural signs are not fully coded, but combine different
semiotic modes to quite varying degrees, that one of the central goals of modern gesture
research has been to identify gestural forms that exhibit relatively high degrees of
patterning and conventionalization. Gestures that are frequently used across individual
speakers, speech genres and situational contexts belong to gesture families (e.g. Kendon
2004), gesture ecologies (Streeck 2009) and recurrent gestures (e.g. Müller 2004, 2017a;
Ladewig 2011, 2014; Bressem and Müller 2014). Besides their tendency to develop
pragmatic functions, some of these routinized forms may become grammatical markers
and/or feed into multimodal constructions (e.g. studies in Zima and Bergs 2017; Hinnell
2018, 2019, 2020).
This section has focused on how semiotic theory and gesture analysis can illuminate
one another. In what follows, we provide insights into how theoretical questions and
empirical work can advance each other in additional ways and on a larger scale.

RECENT EMPIRICAL RESEARCH STRANDS

As discussed above, we are now in what we have identified as a third wave of modern
gesture studies. In this section, we introduce technologies that are being used to explore
semiotically oriented research questions in multimodal communication, such as large-
scale multimedia archives, fMRI imaging, and 3D motion capture and a sampling of
research projects using these technologies.
Empirical gesture research depends on the time-intensive task of producing annotated
speech and gesture transcripts that allow for the analysis of the synchrony of co-occurring
linguistic and gestural forms. First, one transcribes the speech and segments the gestural
utterance into phrases and phases (e.g. McNeill 1992; Kendon 2004). Then, gestures are
normally annotated according to their form and functions as per the specific research
questions driving the study (see, e.g., Kendon (2004) and Bressem (2013b) for gesture
annotation; Hinnell (2018, 2020) for detailed annotations of speech, gesture and other
bodily articulators such as head movement). Video-annotation tools such as ELAN allow
users to create and search utterance transcriptions and multiple annotation layers.5 These
annotation methods form the basis for a range of methodologies, from experimental to
corpus-based work we have referred to throughout this chapter.
Due to theoretical perspectives informed by cognitive linguistics that focus on actual
language usage in interaction (Feyaerts, Brône, and Oben 2017), as well as the availability
of larger, text-searchable, multimedia databases, gesture research has recently featured
a growth in multimodal corpus studies. Platforms such as Red Hen (Steen and Turner
2013; www.redhenlab.org) and the TV News Archive (www.archive.org/tv) allow
researchers to harness data from publicly broadcast television and feature searchable text
from television closed-captioning. Once a targeted search – e.g. for a specific linguistic
construction – is complete, data are manually annotated for linguistic and gestural form
and function, and then analysed using quantitative and/or descriptive statistical methods
(e.g. Joo, Steen and Turner 2017; Hinnell 2018, 2019, 2020; Pagán Cánovas et al. 2020;
Woodin et al. 2020). The availability of interactional, multimedia data has resulted in
studies featuring much larger data sets. Such studies address issues that are inherently
semiotic, e.g. patterned communicative behaviour and the conventionalization of signs,
GESTURE STUDIES AND SEMIOTICS 203

including the degree and the loci of conventionality that designate components of a
gestural sign as such and of multimodal sign processes.
As an example, in a recent corpus-based study aligned with scholarship on the central
role of the body and our experience in the world in driving linguistic and conceptual
structure (e.g. in cognitive linguistics (Johnson 1987) and semiotics (Pelkey 2017)),
Hinnell (2019) investigates how speakers of North American English build semiotic
environments around the construal of contrast (recall the gesture of antithesis in
Figure 9.1). Using data from Red Hen, the study documents how speakers mark contrast
in gesture and head movement alongside spoken utterances such as on the one hand/on
the other hand, the conjunction versus, comparative expressions (better than/worse than),
antonym pairs such as David/Goliath and rich/poor, opposition as negation (should I/
shouldn’t I?), and others. Hinnell argues that both iconicity and indexicality motivate the
multimodal forms that characterize her data. Iconicity is seen in the symmetrical nature
of the gestures (and head movements) and the use of lateral space, which embodies the
balance image schema that is the basis of contrast (Johnson 1987). Indexicality emerges
in the handshape – frequently, but not always, a point with an extended index finger (see
Figure 9.7) – and in the practice of indicating towards opposite sides of the lateral gesture
space to reference opposing discourse objects. In Figure 9.7, the first line (S) signifies the
speech utterance and the second (G) gesture description. Target utterances in the text
search are underlined.
Brain imaging studies (see review in Özyürek 2014) and 3D motion capture (MoCap)
also facilitate empirical investigations of semiotic and related theories. Functional magnetic
resonance imaging (fMRI) has been used to investigate the neuroscientific basis for Peirce’s
universal categories (Wolf et al. 2017, 2018) and metonymy in gesture (Joue et al. 2018).
As a digital medium, MoCap is semiotic in its very nature, given its capacity to render
gesture both visually and algorithmically. Video recordings are made of dialogues in which
speakers wear markers, e.g. on their hands, arms, shoulders, hips, and face (for marker-less
tracking, see Trujillo et al. 2019; also Pouw et al. 2019 on video-based motion tracking).
After data processing, the motion trace of a gesture as well as the skeletal morphology and
position of a person’s body can be viewed without seeing the person’s physical body (as
one can in video recordings), as in Figure 9.4. MoCap can serve both computational and
linguistic semiotic purposes. For example, Mittelberg and colleagues (2015; building on

FIGURE 9.7 Embodied contrast: On the one hand/on the other hand, bilateral point sequence.
204 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Hinnell 2018) recorded a series of conversations among speakers of American English in a

motion capture lab. Gestures marking specific event construals were identified and manually
annotated. The MoCap profiles of identified gestural forms were then used to derive kinetic
patterns and improve spatiotemporal similarity searching in 3D data (Schüller et al. 2017).
Semiotic understandings of gesture have an important role to play in the development
of artificial intelligence (AI) and human-computer interaction (HCI). In order to render
human-like dialogue in HCI applications, the quality of multimodal utterances would,
presumably, need to approach human-like behaviour (see Hinnell 2020). Wicke and
Veale (2021), for example, have developed an embodied framework for the multimodal
performance and interpretation of narratives within robotic storytelling. (See Bressem
(2013a), Joo, Steen and Turner (2017) and Kopp (2017) for applications of gesture
studies to AI and vice versa).
Returning to the wave metaphor, we suggest that such methodological and
technological advances, which, importantly, are driving interdisciplinary collaborations,
are also driving a fourth wave of gesture studies. To have substantial and sustainable
impact, we propose that the new technologies be developed with an aim of making them
available beyond individual research programmes and that results should ideally feed
back into semiotic theory-building.

CONCLUSION
In this chapter, we have aimed to show that investigating gestural phenomena through a
semiotic lens allows for deep insights into the specific semiotic nature of gesture. What we
can draw from the expansive body of work discussed in this chapter is that gesture – at
first glance a basic and natural human communicative resource – can teach us a great
deal about semiosis in general, as well as about embodiment and multimodality. Gesture
indeed reveals essential aspects about iconicity, metaphor, indexicality, metonymy,
conventionality/habit, etc., and about the formation and functioning of signs. It is likely
precisely because gestures are so essential to us as embodied social beings that they can
reveal primordial, including preverbal, facets of how we experience our inner and outer
world, express ourselves and understand others.
Looking ahead, semiotic perspectives in gesture studies continue to demonstrate
the potential to advance our understanding of the intricate bodily-based sign processes
that underlie human cognition and communicative interaction. In order to do justice
to the noted semiotic complexity of co-speech gestures, and multimodal language more
broadly, the evolution of both theoretical and empirical work must feature focused
studies within the intersecting disciplines, as well as interdisciplinary research initiatives
that see humanities scholars, data scientists and many others pursue truly collaborative
innovation.

NOTES
1 This image is in the public domain and was accessed online: https://publicdomainreview.org/
collection/chirologia-or-the-natural-language-of-the-hand-1644
2 US Vice President Kamala Harris gives thumbs up to students at Miller Elementary School,
Dearborn, Michigan, Monday, 6 May 2019. Retrieved 4 June 2021. Copyright: Associated
Press 2019. Source: https://www.taiwannews.com.tw/en/news/3697138
GESTURE STUDIES AND SEMIOTICS 205

3 Transcript of verbal transcription (Figure 9.4): ‘I thought it was a girl, because, uhm she
looked like she was wearing some sort of short flowing skirt uhm, and . . . uh her head was
turned to this side if I were mirroring what she was doing and her arms were like this. Uhm,
and . . . uh.. her mouth was almost in the shape of a heart and . . . uhm, I kept trying to see if
her eyes were open or closed, and it looked like they were just slits.’
4 German original: ‘Nee, ich denke wir fahren dann so weißte von da nach da rüber runter und
dann nach da und dann wieder nach Hause, nein?’
5 https://tla.mpi.nl/tools/tla-tools/elan/ (ELAN 2020).

REFERENCES
Andrén, M. (2010), Children’s Gestures from 18 to 30 Months, PhD.diss., Lund University,
Lund.
Armstrong, D. F., W. C. Stokoe and S. Wilcox (1995), Gesture and the Nature of Language,
Cambridge: Cambridge University Press.
Arnheim, R. (1969), Visual Thinking, Berkeley: University of California.
Bavelas, J., J. Gerwing, C. Sutton and D. Prevost (2008), ‘Gesturing on the Telephone:
Independent Effects of Dialogue and Visibility’, Journal of Memory and Language, 58 (2):
495–520.
Bavelas, J. B., N. Chovil, L. Coates and L. Roe (1995), ‘Gestures Specialized for Dialogue’,
Personality and Social Psychology Bulletin, 21 (4): 394–405.
Boutet, D., A. Morgenstern and A. Cienki (2016), ‘Grammatical Aspect and Gesture in French:
A Kinesiological Approach’, Russian Journal of Linguistics, 20 (3): 132–51.
Bouvet, D. (1997), Le Corps et la Métaphore dans les Langues Gestuelles: À la Recherche des
Modes de Production des Signes, Paris: L’Harmattan.
Bressem, J. (2013a), ‘20th Century: Empirical Research of Body, Language, and
Communication’, in C. Müller et al. (eds), Body – Language – Communication: An
International Handbook on Multimodality in Human Interaction, vol. 1, 393–415, Berlin:
De Gruyter Mouton.
Bressem, J. (2013b), ‘A Linguistic Perspective on the Notation of Form Features in Gestures’,
in C. Müller et al. (eds), Body – Language – Communication: An International Handbook on
Multimodality in Human Interaction, vol. 1, 1079–98, Berlin: de Gruyter Mouton.
Bressem, J. (2014), ‘Repetitions in Gesture’, in C. Müller et al. (eds), Body – Language –
Communication: An International Handbook on Multimodality in Human Interaction, vol.
2, 1037–958, Berlin: de Gruyter Mouton.
Bressem, J. and C. Müller (2014), ‘A Repertoire of German Recurrent Gestures with Pragmatic
Functions’, in C. Müller et al. (eds), vol. 2, 1575–91, Berlin: de Gruyter Mouton.
Brookes, H. (2004), ‘A Repertoire of South African Quotable Gestures’, Journal of Linguistic
Anthropology, 14 (2): 186–224.
Bühler, K. ([1934] 1982), Sprachtheorie: Die Darstellungsfunktion der Sprache, Stuttgart/New
York: Fischer.
Calbris, G. (1990), The Semiotics of French Gestures, Bloomington: Indiana University Press.
Calbris, G. (2011), Elements of Meaning in Gesture, trans. M. M. Copple, Amsterdam: John
Benjamins.
Cienki, A. (1998), ‘Metaphoric Gestures and Some of Their Relations to Verbal Metaphoric
Expressions’, in J.-P. Koenig (ed.), Discourse and Cognition: Bridging the Gap, 189–204,
Stanford, CA: CSLI Publications.
206 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­Cienki, A. (2013), ‘Image Schemas and Mimetic Schemas in Cognitive Linguistics and
Gesture Studies’, Review of Cognitive Linguistics, 11 (2): 417–32. https://doi.org/10.1075/
rcl.11.2.13cie
Cienki, A. and C. Müller, eds (2008), Metaphor and Gesture, Amsterdam: John Benjamins.
Clark, H. H. (1996), Using Language, Cambridge: Cambridge University Press.
Clark, H. H. (2003), ‘Pointing and Placing’, in S. Kita (ed.), Pointing: Where Language, Culture,
and Cognition Meet, 243–68, Mahwah, NJ: L. Erlbaum Associates.
Clark, H. H. (2016), ‘Depicting as a Method of Communication’, Psychological Review, 123
(3): 324–47.
Cooperrider, K., J. Slotta and R. Núñez (2018), ‘The Preference for Pointing with the Hand Is
Not Universal’, Cognitive Science, 42 (4): 1375–90.
Cooperrider, K., J. Fenion, J. Keane, D. Brentari and S. Goldin-Meadow (2021), ‘How
Pointing Is Integrated into Language: Evidence from Speakers and Signers’, Frontiers in
Communication, 6: 567774.
Dancygier, B. and E. E. Sweetser (2014), Figurative Language, Cambridge: Cambridge
University Press.
Deacon, T. (1997), The Symbolic Species: The Co-evolution of Language and the Brain, New
York and London: Norton.
Debras, C. (2017), ‘The Shrug: Forms and Meanings of a Compound Enactment’, Gesture, 16
(1): 1–34.
Dudis, P. (2004), ‘Body Partitioning and Real Space Blends’, Cognitive Linguistics, 15:
223–38.
Duncan, S., J. Cassell and E. T. Levy (eds) (2007), Gesture and the Dynamic Dimension of
Language, Amsterdam: John Benjamins.
Efron, D. ([1941] 1972), Gesture, Race and Culture: A Tentative Study of the Spatio-temporal
and ‘Linguistic’ Aspects of the Gestural Behavior of Eastern Jews and Southern Italians in
New York City, Living under Similar as well as Different Environmental Conditions, The
Hague: Mouton.
Ekman, P. and W. Friesen (1969), ‘The Repertoire of Nonverbal Behavior: Categories, Origins,
Usage, and Coding’, Semiotica, 1 (1): 49–98.
ELAN (Version 6.0) [Computer software] (2020), Nijmegen: Max Planck Institute for
Psycholinguistics, The Language Archive. Retrieved from https://archive.mpi.nl/tla/elan.
Enfield, N. J. (2009), The Anatomy of Meaning: Speech, Gesture, and Composite Utterances,
Cambridge: Cambridge University Press.
Enfield, N. J. (2011), ‘Elements of Formulation’, in J. Streeck, C. Goodwin and C. LeBaron
(eds), Embodied Interaction: Language and the Body in the Material World, 59–66,
Cambridge: Cambridge University Press.
Enfield, N. J. (2013), ‘A “Composite Utterances” Approach to Meaning’, in C. Müller et al.
(eds), vol. 1, 689–707, Berlin: de Gruyter Mouton.
Enfield, N. J., S. Kita and J. P. De Ruiter (2007), ‘Primary and Secondary Pragmatic Functions
of Pointing Gestures’, Journal of Pragmatics, 39 (10): 1722–41.
Ferrara, L. and G. Hodge (2018), ‘Language as Description, Indication, and Depiction’,
Frontiers in Psychology, 9: 1–15.
Feyaerts, K., G. Brône and B. Oben (2017), ‘Multimodality in Interaction’, in B. Dancygier
(ed.), The Cambridge Handbook of Cognitive Linguistics, 135–56, Cambridge: Cambridge
University Press.
Fillmore, C. J. (1982), ‘Frame Semantics’, in The Linguistic Society of Korea (eds), Linguistics
in the Morning Calm, 111–37, Seoul: Hanshin.
GESTURE STUDIES AND SEMIOTICS 207

­Fricke, E. (2007), Origo, Geste und Raum – Lokaldeixis im Deutschen, Berlin and New York:
de Gruyter.
Fricke, E. (2012), Grammatik multimodal: Wie Wörter und Gesten zusammenwirken, Berlin: de
Gruyter.
Gibbs, R. W., Jr. (2005), Embodiment and Cognitive Science, Cambridge: Cambridge
University Press.
Goffman, E (1955), ‘On Face-work: An Analysis of Ritual Elements in Social Interaction’,
Psychiatry: Interpersonal and Biological Processes, 18: 213–31.
Goldin-Meadow, S. (2003), Hearing Gesture: How Our Hands Help Us Think, Cambridge,
MA: Harvard University Press.
Goldin-Meadow, S. and D. Brentari (2017), ‘Gesture, Sign, and Language: The Coming of Age
of Sign Language and Gesture Studies’, Behavioral and Brain Sciences, 40: e46.
Goodwin, C. (1981), Conversational Organization: Interaction between Speakers and Hearers,
New York: Academic Press.
Goodwin, C. (2007), ‘Environmentally Coupled Gestures’, in S. D. Duncan, J. Cassell and E. T.
Levy (eds), Gesture and the Dynamic Dimension of Language, vol. 1: Gesture Studies 1,
195–212, Amsterdam: John Benjamins.
Goodwin, C. (2011), ‘Contextures of Action’, in J. Streeck, C. Goodwin and C. LeBaron (eds),
Embodied Interaction: Language and the Body in the Material World, 182–93, Cambridge:
Cambridge University Press.
Goodwin, C. (2014), ‘The Intelligibility of Gesture within a Framework of Co-operative
Action’, in M. Seyfeddinipur and M. Gullberg (eds), From Gesture in Conversation to Visible
Action as Utterance, 199–218, Amsterdam/Philadelphia: John Benjamins.
Goodwin, C. (2018), Co-operative Action, Cambridge: Cambridge University Press.
Grady, J. (1997), Foundations of Meaning: Primary Metaphors and Primary Scenes, PhD diss.,
University of California, Berkeley, CA.
Hall, J. (2004), ‘Cicero and Quintilian on the Oratorical Use of Hand Gestures’, The Classical
Quarterly, 54 (1): 143–60.
Harrison, S. (2018), The Impulse to Gesture: Where Language, Minds, and Bodies Intersect,
Cambridge: Cambridge University Press.
Hassemer, J. and L. McCleary (2020), ‘The Multidimensionality of Pointing’, Gesture, 17 (3):
417–63.
Haviland, J. B. (1993), ‘Anchoring, Iconicity, and Orientation in Guugu Yimithirr Pointing
Gestures’, Journal of Linguistic Anthropology, 3 (1): 3–45.
Haviland, J. B. (2000), ‘Pointing, Gesture Spaces, and Mental Maps’, in D. McNeill (ed.),
Language and Gesture, 13–46, Cambridge, MA: Cambridge University Press.
Hinnell, J. (2018), ‘The Multimodal Marking of Aspect: The Case of Five Periphrastic
Auxiliary Constructions in North American English’, Cognitive Linguistics, 29 (4): 773–806.
Hinnell, J. (2019), ‘The Verbal-kinesic Enactment of Contrast in North American English’, The
American Journal of Semiotics, 35 (1–2): 55–92.
Hinnell, J. (2020), ‘Language in the Body: Multimodality in Grammar and Discourse’, Ph.D.
diss., University of Alberta, Edmonton.
Holler, J. and J. Bavelas (2017), ‘Multi-modal Communication of Common Ground: A Review
of Social Functions’, in R. B. Church, M. W. Alibali and S. D. Kelly (eds), Why Gesture:
How the Hands Function in Speaking, Thinking and Communicating, 213–40, Amsterdam:
John Benjamins.
Hostetter, A. B. and M. W. Alibali (2008), ‘Visible Embodiment: Gestures as Simulated Action’,
Psychonomic Bulletin & Review, 15: 495–514.
208 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­Jakobson, R. (1956), ‘Two Aspects of Language and Two Types of Aphasic Disturbances’, in
L. R. Waugh and M. Monville-Burston (eds), Roman Jakobson – On Language, 115–33,
Cambridge: Harvard University Press.
Jakobson, R. ([1957] 1984), ‘Shifters, Verbal Categories, and the Russian Verb’, in L. Waugh
and M. Halle (eds), Russian and Slavic Grammar: Studies, 1931–81, 41–58, Berlin:
Mouton.
Jakobson, R. (1960), ‘Closing Statement: Linguistics and Poetics’, in T. A. Sebeok (ed.), Style in
Language, 350–77, Cambridge: MIT Press.
Jakobson, R. ([1966] 1990), ‘Quest for the Essence of Language’, in L. Waugh and M.
Monville-Burston (eds), Roman Jakobson: On Language, 407–21, Cambridge: Belknap
Press.
Jakobson, R. (1972), ‘Motor Signs for “Yes” and “No”’, Language in Society, 1 (1): 91–6.
Jakobson, R. and K. Pomorska (1983), Dialogues, Cambridge: Harvard University Press.
Jakobson, R. and L. R. Waugh ([1979] 2002), The Sound Shape of Language, Berlin: de Gruyter
Mouton.
Janzen, T. and B. Shaffer (2002), ‘Gesture as the Substrate in the Process of ASL
Grammaticization’, in R. P. Meier, K. Cormier and D. Quinto-Pozos (eds), Modality and
Structure in Signed and Spoken Languages, 199–223, Cambridge: Cambridge University
Press.
Joue, G., L. Boven, K. Willmes, V. Evola, L. R. Demenescuz, J. Hassemer, I. Mittelberg, K.
Mathiak, F. Schneider and U. Habel (2018), ‘Handling or Being the Concept: an fMRI-
study on Metonymy Representations in Coverbal Gestures’, Neuropsychologia, 109:
232–44.
Johnson, M. (1987), The Body in the Mind: The Bodily Basis of Meaning, Imagination, and
Reason, Chicago: University of Chicago Press.
Joo, J., F. F. Steen and M. Turner (2017), ‘Red Hen Lab: Dataset and Tools for Multimodal
Human Communication Research’, Künstliche Intelligenz, 31: 357–61.
Kendon, A. (1972), ‘Some Relationships between Body Motion and Speech’, in A. W. Siegman
and B. Pope (eds), Studies in Dyadic Communication, 177–216, New York/Toronto/Oxford/
Sydney/Braunschweig: Pergamon.
Kendon, A. (1980), ‘Gesture and Speech: Two Aspects of the Process of Utterance’, in M. R.
Key (ed.), Nonverbal Communication and Language, 207–88, The Hague: Mouton.
Kendon, A. (1988), Sign Languages of Aboriginal Australia: Cultural, Semiotic and
Communicative Perspectives, Cambridge: Cambridge University Press.
Kendon, A. (1995), ‘Gestures as Illocutionary and Discourse Structure Markers in Southern
Italian Conversation’, Journal of Pragmatics, 23 (3): 247–79.
Kendon, A. (2000), ‘Language and Gesture: Unity or Duality’, in D. McNeill (ed.), Language
and Gesture: Window into Thought and Action, 47–63, Cambridge: Cambridge University
Press.
Kendon, A. (2004), Gesture: Visible Action as Utterance, Cambridge: Cambridge University
Press.
Kendon, A. (2008), ‘Some Reflections on the Relationship between “Gesture” and “Sign”’,
Gesture, 8 (3): 348–66.
Kendon, A. (2017), ‘Pragmatic Functions of Gestures: Some Observations on the History of
Their Study and Their Nature’, Gesture, 16 (2): 157–75.
Kita, S. (2000), ‘How Representational Gestures Help Speaking’, in D. McNeill (ed.), Language
and Gesture, vol. 2: Language, Culture and Cognition, 162–85, Cambridge: Cambridge
University Press.
GESTURE STUDIES AND SEMIOTICS 209

­Kita, S., ed. (2003), Pointing: Where Language, Culture, and Cognition Meet, Mahwah, NJ: L.
Erlbaum Associates.
Kopp, S. (2017), ‘Computational Gesture Research: Studying the Functions of Gesture in
Human-Agent Interaction’, in R. B. Church, M. W. Alibali and S. D. Kelly (eds), Why
Gesture: How the Hands Function in Speaking, Thinking and Communicating, 267–84,
Amsterdam: John Benjamins.
Kurz K. B., K. Mullaney and C. Occhino (2019), ‘Constructed Action in American Sign
Language: A Look at Second Language Learners in a Second Modality’, Languages, 4 (4):
90.
Ladewig, S. H. (2011), ‘Putting the Cyclic Gesture on a Cognitive Basis’, CogniTextes, 6: 1–23.
Ladewig, S. H. (2014), ‘Recurrent Gestures’, in C. Müller et al. (eds), Body – Language –
Communication: An International Handbook on Multimodality in Human Interaction, vol. 2,
1558–74, Berlin: de Gruyter Mouton.
Ladewig, S. H. (2020), Integrating Gestures: The Dimension of Multimodality in Cognitive
Grammar, Berlin: de Gruyter.
Lakoff, G. and M. Johnson (1980), Metaphors We Live by, Chicago: Chicago University Press.
Lakoff, G. and M. Johnson (1999), Philosophy in the Flesh: The Embodied Mind and Its
Challenge to Western Thought, New York: Basic Books.
Lausberg, H. (2019), The NEUROGES Analysis System for Nonverbal Behavior and Gesture:
The Complete Research Coding Manual Including an Interactive Video Learning Tool and
Coding Template, Berlin: Lang.
LeBaron, C. and J. Streeck (2000), ‘Gestures, Knowledge, and the World’, in D. McNeill (ed.),
Language and Gesture, 118–38, Cambridge: Cambridge University Press.
Leroi-Gourhan, A. (1964), Le Geste et la Parole: Techniques et Langage, Paris: Éditions Albin
Michel.
Liddell, S. K. (2003), Grammar, Gesture, and Meaning in American Sign Language, Cambridge:
Cambridge University Press.
Liebal, K., B. Waller, A. Burrows and K. Slocomb (2013), Primate Communication: A
Multimodal Approach, Cambridge: Cambridge University Press.
Lücking, A. (2013), Ikonische Gesten: Grundzüge einer Linguistischen Theorie, Berlin: de
Gruyter.
Maddalena, G. (2015), The Philosophy of Gesture: Completing Pragmatists’ Incomplete
Revolution, Montreal/Kingston: McGill-Queen’s University Press.
Mandel, M. (1977), ‘Iconic Devices in American Sign Language’, in L. Friedman (ed.), On the
Other Hand: New Perspectives on American Sign Language, 57–107, New York: Academic
Press.
McCafferty, S. and G. Stam, eds (2008), Gesture: Second Language Acquisition and Classroom
Research, New York/London: Routledge.
McNeill, D. (1985), ‘So You Think Gestures Are Nonverbal?’ Psychological Review, 92 (3):
350–71.
McNeill, D. (1992), Hand and Mind: What Gestures Reveal about Thought, Chicago: Chicago
University Press.
McNeill, D., ed. (2000), Language and Gesture, 2nd edn, Cambridge: Cambridge University
Press.
McNeill, D. (2005), Gesture and Thought, Chicago: University of Chicago Press.
McNeill, D., J. Cassell and E. T. Levy (1993), ‘Abstract Deixis’, Semiotica, 95 (1/2): 5–19.
Mead, G. H. (1934), Mind, Self and Society from the Standpoint of a Social Behaviorist,
Chicago: University of Chicago Press.
210 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­Mead, G. H. (1938), The Philosophy of the Act, ed. C.W. Morris, Chicago: University of
Chicago Press.
Merleau-Ponty, M. (1962), Phénoménologie de la perception, Paris: Gallimard.
Mittelberg, I. (2006), Metaphor and Metonymy in Language and Gesture: Discourse Evidence
for Multimodal Models of Grammar, PhD diss., Ithaca: Cornell University.
Mittelberg, I. (2008), ‘Peircean Semiotics Meets Conceptual Metaphor: Iconic Modes in
Gestural Representations of Grammar’, in A. Cienki and C. Müller (eds), Metaphor and
Gesture, 115–54, Amsterdam: John Benjamins.
Mittelberg, I. (2013a), ‘The Exbodied Mind: Cognitive-semiotic Principles as Motivating
Forces in Gesture’, in C. Müller et al. (eds), vol. 1, 755–84, Berlin: de Gruyter Mouton.
Mittelberg, I. (2013b), ‘Balancing Acts: Image Schemas and Force Dynamics as Experiential
Essence in Pictures by Paul Klee and Their Gestural Enactments’, in B. Dancygier, M.
Borkent and J. Hinnell (eds), Language and the Creative Mind, 325–46, Stanford: CSLI.
Mittelberg, I. (2014), ‘Gestures and Iconicity’, in C. Müller et al. (eds), vol. 2, 1712–32, Berlin:
de Gruyter Mouton.
Mittelberg, I. (2017a), ‘Experiencing and Construing Spatial Artifacts from within: Simulated
Artifact Immersion as a Multimodal Viewpoint Strategy’, Cognitive Linguistics, 28 (3):
381–415.
Mittelberg, I. (2017b), ‘Multimodal Existential Constructions in German: Manual Actions
of Giving as Experiential Substrate for Grammatical and Gestural Patterns’, Linguistics
Vanguard, 3 (1): 20160047.
Mittelberg, I. (2018), ‘Gestures as Image Schemas and Force Gestalts: A Dynamic Systems
Approach Augmented with Motion-capture Data Analysis’, Cognitive Semiotics, 11 (1): 1–2.
Mittelberg, I. (2019a), ‘Visuo-kinetic Signs Are Inherently Metonymic: How Embodied
Metonymy Motivates Forms, Functions, and Schematic Patterns in Gesture’, Frontiers in
Psychology, 10: 1–18.
Mittelberg, I. (2019b), ‘Peirce’s Universal Categories: On Their Potential for Gesture Theory
and Multimodal Analysis’, Semiotica, 228: 193–222.
Mittelberg, I. and L. R. Waugh (2009), ‘Metonymy First, Metaphor Second: A Cognitive-semiotic
Approach to Multimodal Figures of Thought in Co-speech Gesture’, in C. J. Forceville and
E. Urios-Aparisi (eds), Multimodal Metaphor, 329–58, Berlin: de Gruyter Mouton.
Mittelberg, I. and L.-M. Rekittke (2021), ‘Frames und Diagramme im Dialog: Indexikalische
und Ikonische Zeichenmodi im Verbal-Gestischen Ko-Konstruieren von Reiserouten’,
Zeitschrift für Semiotik, 43(1–2): 81–116.
Mittelberg, I. and L.R. Waugh (2014), ‘Gestures and Metonymy’, in C. Müller et al. (eds), vol.
2, 1747–66, Berlin: de Gruyter Mouton.
Mittelberg, I. and V. Evola (2014), ‘Iconic and Representational Gestures’, in C. Müller, et al.
(eds), vol. 2, 1732–46, Berlin: de Gruyter Mouton.
Mittelberg, I., J. Hinnell, C. Beecks, M. Hassani and T. Seidl (2015), ‘Emergent Grammar
in Gesture: A Motion-capture Analysis of Image-schematic Aspectual Contours in North
American English Speaker-gesturers’, 13th International Cognitive Linguistics Conference,
Newcastle, 20–25 July.
Morford, J. and J. A. Kegl (2000), ‘Gestures as Precursors to Linguistic Constructs: How
Input Shapes the Form of Language’, in D. McNeill (ed.), Gesture and Language, 158–387,
Cambridge: Cambridge University Press.
Morris, C. W. (1946), Newcastle Signs, Language, Behaviour, New York: Prentice Hall.
Morris, D. (1979), Gestures: Their Origins and Distributions, London: Cape.
GESTURE STUDIES AND SEMIOTICS 211

­Müller, C. (1998a), Redebegleitende Gesten: Kulturgeschichte – Theorie – Sprachvergleich,

Berlin: Spitz Verlag.
Müller, C. (1998b), ‘Iconicity and Gesture’, in S. Santi et al. (eds), Oralité et Gestualité:
Communication Multimodale et Interaction, 321–8, Montréal/Paris: L’Harmattan.
Müller, C. (2004), ‘Forms and Uses of the Palm Up Open Hand: A Case of a Gesture Family’,
in C. Müller and R. Posner (eds), The Semantics and Pragmatics of Everyday Gestures,
233–56, Berlin: Weidler.
Müller, C. (2014), ‘Gestural Modes of Representation as Techniques of Depiction’,
in C. Müller, et al. (eds), Body – Language – Communication: An International
Handbook on Multimodality in Human Interaction, vol. 2, 1687–702, Berlin: de Gruyter
Mouton.
Müller, C. (2017a), How Recurrent Gestures Mean: Conventionalized Contexts-of-use and
Embodied Motivation, Gesture, 16 (2): 277–304.
Müller, C. (2017b), ‘Waking Metaphors: Embodied Cognition in Multimodal Discourse’,
in B. Hampe (ed.), Metaphor: Embodied Cognition and Discourse, 297–316, Cambridge:
Cambridge University Press.
Müller, C. (2018), ‘Gestures and Sign: Cataclysmic Break or Dynamic Relations?’ Frontiers in
Psychology, 9: 1651.
Müller, C. and A. Cienki (2009), ‘Words, Gestures and Beyond: Forms of Multimodal
Metaphor in the Use of Spoken Language’, in C. Forceville and E. Urios-Aparisi (eds),
Multimodal Metaphor, 297–328, Berlin: de Gruyter Mouton.
Müller, C. and H. Kappelhoff (2018), Cinematic Metaphor, Berlin: de Gruyter.
Müller, C. and R. Posner, eds (2004), The Semantics and Pragmatics of Everyday Gesture: The
Berlin Conference, Berlin: Weidler Verlag.
Müller, C., A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill and S. Teßendorf, eds (2013),
Body – Language – Communication: An International Handbook on Multimodality in
Human Interaction, vol. 38/2, Berlin: de Gruyter Mouton.
Müller, C., A. Cienki, E. Fricke, S. Ladewig, D. McNeill and J. Bressem, eds (2014), Body –
Language – Communication: An International Handbook on Multimodality in Human
Interaction, 38/2, vol. 2, Berlin: de Gruyter Mouton.
Nöth, W. (2016), ‘Habits, Habit of Change, and the Habit of Habit Change According to
Peirce’, in D. West and M. Anderson (eds), Consensus on Peirce’s Concept of Habit, 35–63,
Dordrecht: Springer.
Núñez, R. E. and E. Sweetser (2006), ‘With the Future behind Them: Convergent Evidence
from Aymara Language and Gesture in the Crosslinguistic Comparison of Spatial Construals
of Time’, Cognitive Science, 30 (5): 401–50.
Özyürek, A. (2014), ‘Hearing and Seeing Meaning in Speech and Gesture: Insights from Brain
and Behaviour’, Philosophical Transactions of the Royal Society B: Biological Sciences, 369
(20130296): 1–10.
Pagán Cánovas, C., J. Valenzuela, D. Alcaraz Carrión, I. Olza and M. Ramscar (2020),
‘Quantifying the Speech-Gesture Relation with Massive Multimodal Datasets: Informativity
in Time Expressions’, PLoS ONE, 15 (6): e0233892.
Panther, K.-U. and L. L. Thornburg, eds (2003), Metonymy and Pragmatic Inferencing,
Amsterdam: John Benjamins.
Parrill, F. (2009), ‘Dual Viewpoint Gestures’, Gesture, 9 (3): 271–89.
Parrill, F. and E. Sweetser (2004), ‘What We Mean by Meaning: Conceptual Integration in
Gesture Analysis and Transcription’, Gesture, 4 (2): 197–219.
212 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­Peirce, C. S. ([1866–1913] 1931–58), The Collected Papers of Charles Sanders Peirce, vols. 1–6,
C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed), Cambridge: Harvard University
Press. Cited as CP.
Peirce, C. S. ([1893–1913] 1998), The Essential Peirce: Selected Philosophical Writings, Peirce
Edition Project (eds), vol. 2, Bloomington: Indiana University Press. Cited as EP 2.
Pelkey, J. (2017), The Semiotics of X: Chiasmus, Cognition, and Extreme Body Memory,
London: Bloomsbury.
Perniss, P. (2018), ‘Why We Should Study Multimodal Language’, Frontiers in Psychology, 9:
1109.
Perniss, P., R. Thompson and G. Vigliocco (2010), ‘Iconicity as a General Property of
Language: Evidence from Spoken and Signed Languages’, Frontiers in Psychology, 1: 227.
Posner, R. (2004), ‘Everyday Gestures as a Result of Ritualization’, in M. Rector, I. Poggi and
N. Trigo (eds), Gestures: Meaning and Use, 217–30, Porto: Edições Universidade Fernando
Pessoa.
Pouw, W., J. P. Trujillo, and J. A. Dixon (2020), ‘The quantification of gesture–speech
synchrony: A tutorial and validation of multimodal data acquisition using device-based and
video-based motion tracking’, Behavior Research Methods, 52: 723–740.
Sandler, W., M. Gullberg and C. Padden, eds (2019), ‘Visual Language’, Frontiers in Psychology
and Frontiers in Communication, 10: 1765, Lausanne: Frontiers Media.
Saussure, F.de ([1916] 1986), Course in General Linguistics, trans. R. Harris, Chicago: Open Court.
Schegloff, E. A. (1984), ‘On Some Gestures’ Relation to Talk’, in J. M. Atkinson and J.
Heritage (eds), Structures of Social Action: Studies in Conversation Analysis, 266–96,
Cambridge: Cambridge University Press.
Schüller, D. and I. Mittelberg (2016), ‘Diagramme von Gesten: Eine Zeichentheoretische
Analyse Digitaler Bewegungsspuren’, Zeitschrift für Semiotik, 34 (3–4): 3–32.
Schüller, D., C. Beecks, M. Hassani, J. Hinnell, B. Brenger, T. Seidl and I. Mittelberg (2017),
‘Automated Pattern Analysis in Gesture Research: Similarity Measuring in 3D Motion
Capture Models of Communicative Action’, Digital Humanities Quarterly, 11 (2): 1–14.
Seyfeddinipur, M. and M. Gullberg, eds (2014), From Gesture in Conversation to Visible Action
as Utterance, Amsterdam and Philadelphia: John Benjamins.
Sonesson, G. (2007), ‘The Extensions of Man Revisited: From Primary to Tertiary
Embodiment’, in J. M. Krois, M. Rosengren, A. Steidele and D. Westerkamp (eds),
Embodiment in Cognition and Culture, 27–53, Amsterdam: John Benjamins.
Sonesson, G. (2014), ‘Some Issues in the Semiotics of Gesture: The Perspective of Comparative
Semiotics’, in C. Müller et al. (eds), Body – Language – Communication: An International
Handbook on Multimodality in Human Interaction, vol. 2, 1989–99, Berlin: de Gruyter
Mouton.
Stec, K. (2012), ‘Meaningful Shifts: A Review of Viewpoint Markers in Co-speech Gesture and
Sign Language’, Gesture, 12 (3): 327–60.
Steen, F. and M. Turner (2013), ‘Multimodal Construction Grammar’, in M. Borkent, B.
Dancygier and J. Hinnell (eds), Language and the Creative Mind, 255–74, Stanford, CA:
Centre for the Study of Language and Information.
Stjernfelt, F. (2014), Natural Propositions: The Actuality of Peirce’s Doctrine of Dicisigns,
Boston: Docent Press.
Stokoe, W. C. ([1960] 2005), ‘Sign Language Structure: An Outline of the Visual
Communication Systems of the American Deaf’, Journal of Deaf Studies and Deaf Education,
10 (1): 3–37.
Streeck, J. (1993), ‘Gesture as Communication I: Its Coordination with Gaze and Speech’,
Communication Monographs, 60 (4): 275–99.
GESTURE STUDIES AND SEMIOTICS 213

­Streeck, J. (2009), Gesturecraft: The Manu-facture of Meaning, Amsterdam and Philadelphia:

John Benjamins.
Streeck, J., C. Goodwin and C. LeBaron, eds (2011), Embodied Interaction: Language and Body
in the Material World, Cambridge: Cambridge University Press.
Sweetser, E. E. (1998), ‘Regular Metaphoricity in Gesture: Bodily-based Models of Speech
Interaction’, in Actes du 16e Congrès International des Linguistes (CD-ROM), Oxford:
Elsevier.
Sweetser, E. E. (2007), ‘Looking at Space to Study Mental Spaces: Co-speech Gesture as a
Crucial Data Source in Cognitive Linguistics’, in M. Gonzalez-Marquez, I. Mittelberg, S.
Coulson and M. J. Spivey (eds), Methods in Cognitive Linguistics, 201–24, Amsterdam: John
Benjamins.
Sweetser, E. E. (2009), ‘What Does It Mean to Compare Language and Gesture? Modalities
and Contrasts’, in J. Guo, E. Liven, N. Budwig, S. Ervin-Tripp, K. Nakamura and S.
Özcaliskan (eds), Crosslinguistic Approaches to the Psychology of Language: Research in the
Tradition of Dan Isaac Slobin, 357–66, New York: Psychology Press.
Sweetser, E. (2012), ‘Introduction – Viewpoint and Perspective in Language and Gesture’, in B.
Dancygier and E. E. Sweetser (eds), Viewpoint in Language: A Multimodal Perspective, 1–22,
Cambridge: Cambridge University Press.
Talmy, L. (1988), ‘Force Dynamics in Language and Cognition’, Cognitive Science, 12: 49–100.
Talmy, L. (2018), The Targeting System of Language, Cambridge, MA: MIT Press.
Taub, S. (2001), Language from the Body: Iconicity and Metaphor in American Sign Language,
Cambridge: Cambridge University Press.
Teßendorf, S. (2013), ‘Emblems, Quotable Gestures, or Conventionalized Body Movements’,
in C. Müller et al. (eds), Body – Language – Communication: An International Handbook on
Multimodality in Human Interaction, vol. 1, 82–100, Berlin: de Gruyter Mouton.
Tomasello, M. (1999), The Cultural Origins of Human Cognition, Cambridge, MA and
London: Harvard University Press.
Trujillo, J. P., J. Vaitonyte, I. Simanova and A. Özyürek (2019), ‘Toward the Markerless and
Automatic Analysis of Kinematic Features: A Toolkit for Gesture and Movement Research’,
Behaviour Research Methods, 51: 769–77.
Varela, F., E. Thompson and E. Rosch (1991), The Embodied Mind: Human Cognition and
Experience, Cambridge: MIT Press.
Wehling, E. (2017), ‘Discourse Management Gestures’, Gesture, 16 (2): 245–76.
Wicke, P. and T. Veale (2021), ‘Creative Action at a Distance: A Conceptual Framework for
Embodied Performance with Robotic Actors’, Frontiers of Robotics and AI, 8: 662182,
online: https://doi.org/10.3389/frobt.2021.662182/
Wilcox, S. (2004), ‘Cognitive Iconicity: Conceptual Spaces, Meaning, and Gesture in Signed
Languages’, Cognitive Linguistics, 15 (2): 119–48.
Wilcox, S. and B. Shaffer (2006), ‘Modality in American Sign Language’, in W. Frawley (ed.),
The Expression of Modality, 207–37, Berlin: de Gruyter Mouton.
Wilcox, S. and C. Occhino (2017), ‘Signed Languages’, in B. Dancygier (ed.), Cambridge
Handbook on Cognitive Linguistics, 99–17, Cambridge: Cambridge University Press.
Wilkins, D. P. (2003), ‘Why Pointing with the Index Finger Is Not a Universal (in Sociocultural
and Semiotic Terms)’, in S. Kita (ed.), Pointing: Where Language, Culture, and Cognition
Meet, 171–216, Mahwah, NJ: L. Erlbaum Associates.
Wolf, D., L.-M. Rekittke, I. Mittelberg, M. Klasen and K. Mathiak (2017), ‘Perceived
Conventionality in Co-speech Gestures Involves the Fronto-temporal Language Network’,
Frontiers in Human Neuroscience, 11: 1–19.
214 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­Wolf, D., I. Mittelberg, L. M. Rekittke, S. Bhavsar, M. Zvyagintsev, A. Haeck, F. Cong, M.

Klasen and K. Mathiak (2018), ‘Interpretation of Social Interactions: Functional Imaging
of Cognitive-semiotic Categories during Naturalistic Viewing’, Frontiers in Human
Neuroscience, 12: 296.
Woodin G., B. Winter, M. Perlman, J. Littlemore and T. Matlock (2020), ‘“Tiny Numbers”
Are Actually Tiny: Evidence from Gestures in the TV News Archive’, PLoS ONE, 15 (11):
e0242142.
Wundt, W. (1921), Völkerpsychologie: Eine Untersuchung der Entwicklungsgesetze von Sprache,
Mythos und Sitte, vol. 1: Sprache, Mythus und Sitte, 4th edn, Stuttgart: Kröner.
Wundt, W. (1973), A Psychology of Gesture, trans. J. S. Thayer, C. M. Greenleaf and M. D.
Silberman, The Hague: Mouton.
Zima, E. and A. Bergs, eds (2017), ‘Towards a Multimodal Construction Grammar’, Linguistics
Vanguard [Special Issue], 3 (s1).
Zlatev, J. (2005), ‘What’s in a Schema? Bodily Mimesis and the Grounding of Language’, in B.
Hampe (ed.), From Perception to Meaning: Image Schemas in Cognitive Linguistics, 313–43.
Berlin: Mouton de Gruyter.
Zlatev, J. (2014), ‘Image Schemas, Mimetic Schemas, and Children’s Gestures’, Cognitive
Semiotics, 7 (1): 3–29.
Zlatev, J. (2015), ‘Cognitive Semiotics’, in P. P. Trifonas (ed.), International Handbook of
Semiotics, 1043–67, Dordrecht: Springer.
 CHAPTER TEN

Multimodality and
Semiotics
DAVID MACHIN AND ARIEL CHEN

INTRODUCTION
Social semiotics and multimodality are terms that have come to be used interchangeably.
Both refer to a form of analysis applied to all forms of communication which has been
developed mostly based on principles from the Systemic Functional Linguistics of
Michael Halliday. Halliday (1978) called his own approach a ‘social semiotic’ model of
language. The basic idea was that rather than modelling grammar, as had been the trend
in linguistics, language was seen rather as a set of available tools, seen as flexible choices,
to be used and combined in different ways in contexts. The task was therefore to identify
what systems of choices were available for accomplishing different things by speakers in
contexts. A number of scholars, such as Bob Hodge and Gunther Kress (1988), began to
argue that this same model had the potential to be applied to all kinds of communication,
such as in the use of images and visual design. The term ‘multimodality’ came into use to
capture this extra-linguistic application of social semiotics.
The term multimodality must be understood in the sense that it was designed to
signal to linguists that meaning-making was always accomplished through instances of
communication that comprised more than language alone. Communication was never
‘mono-modal’ but always involved things like images, fonts, types of paper, gestures,
sound and music. Meaning-making was always ‘multimodal’. Any analysis which was
monomodal would therefore neglect much of what was being deployed to make meaning,
for example, in the case of linguistic analyses of advertisement posters. Here it would miss
the communicative role of images, icons and symbols. Even, it was argued by Kress and
Van Leeuwen (2001), communication was becoming more multimodal. In other words,
as design became more sophisticated, aided by advancing technologies, language was used
more and more in combination with other modes and in turn the use of these other modes,
such as typefaces, colours and images, was becoming more sophisticated and important.
So, linguistics was in danger of being left behind. It had to become multimodal.
Of course, to scholars with any knowledge of semiotics, or working in the field of media
studies, photography studies or film studies, the idea that communication is multimodal
would be rather obvious and there is at least a century worth of studies and theories of the
visual and film. But multimodality comprised a significant breakthrough within linguistics,
where social semiotics has been applied to all manner of communicative forms, such as
images, film, monuments, photojournalism and art. One of its major shortcomings has
216 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 10.1 UFIT protein shake (left) and Slimfast protein shake (right).

been, perhaps, that the linguistics who have taken up this social semiotic approach to all
forms of communication have engaged little with existing literature in semiotics, visual
studies, media studies, film studies, photography or graphic design. And one problem is
that this model has been treated as a singular grand theory of all things and there has been
little real critical interrogation of its concepts and principles and what kinds of limitations
they may have (Ledin and Machin 2018). At the time of writing it was the scholarship
in multimodality which was looking outwards also to link with other theories of the
visual, with semiotics and more sociological understandings of communication which
were looking much more robust. Nevertheless, social semiotics, certainly, in itself has
much to offer our understandings of communication, and multimodality has developed
into a range of exciting subfields each with its own emphasis, procedures and concerns
(Jewitt, Bezemer and O’Halloran 2016).
Overall, social semiotics and multimodality encourage us to think about how we
communicate by drawing on inventories of existing choices of signs, and that these are
used by people in a motivated way in contexts. And this is not neutral but related to
power, persuasion and the negotiation of social relations. Multimodality can be very
successful in drawing out the ideas and values which may be more buried in instances of
MULTIMODALITY AND SEMIOTICS 217

communication. In this chapter we show how this does indeed have something highly
useful to contribute to the wider field of semiotics.
Two books, Reading Images (Kress and Van Leeuwen 1996) and The Language of
Displayed Art (O’Toole 1994), have been most influential, laying the foundations of
multimodality. In fact, these two books pick up in different aspects of Halliday’s social
semiotics in his systemic functional grammar and this can be seen echoed in work in the
area to this day. As critical linguists Kress and Van Leeuwen were more interested in the
functional side of communication. They were interested in motivated sign-making and
ideology and did indeed engage with other works in semiotics, particularly Roland Barthes
(1977). So here communication was always to be placed in social and political context.
O’Toole, on the other hand, was more interested in the systemic side of communication.
Here the aim is more to model visual communication in the fashion that linguists model
grammar in language. In this chapter we draw on the former, more functionally and
socially oriented approach in multimodality.
In this chapter we now go on to lay out some of these basic principles and tools of
multimodality. To illustrate how these work we show how they can be used as a research
tool using the example of food packaging. Here we look at the way that foods marketed
for fitness and health are very much gendered, even though they may contain the same
ingredients. Multimodality here can help us to show how gender is communicated as well
as fitness and healthiness itself. It helps us to draw out what kinds of ideas and values are
associated with these. For example, in the case of the two fitness drinks in Figure 10.1,
these contain similar ingredients. We do not need to look very hard to see that they are
gendered, i.e. marketed differently for men and women. What multimodality can help
us do is to focus in on details that we may miss as we describe how exactly this has been
accomplished. And this focus on details can in turn help us to draw out the less obvious
ways that ideas, values and identities are communicated. And indeed, it is an interesting
thing, ideologically, that healthiness and fitness are so gendered.

SEMIOTIC RESOURCES AND CHOICES

One basic idea in multimodality is that all acts of communication are accomplished
through drawing on a set of available choices which users can apply to, and assemble
appropriately, for specific contexts. These choices can be thought of as ‘semiotic
resources’ since they are available to us pre-loaded with meaning potential to be used for
our purposes in different instances. These choices are loaded with meaning potentials in
the sense that a word like ‘youth’ has the potential to describe ‘young people’, or in the
case of ‘youth of today’ be combined to suggest something more negative. Here the word
‘youth’ as opposed to ‘young’ carries, therefore, slightly different meaning potentials. In
this way, the making of communication should be seen not as necessarily rigid, but as a
process where communicators use and combine choices to suit their needs in contexts
to achieve certain goals. And the choices they make are not simply about describing or
representing the world but about how we make the world as we create priorities, plans,
definitions and build institutions. There is no correct or neutral way to represent a thing,
process or person, only available semiotic resources that are preloaded with meanings
and which carry traces of the socio-political moment they were created (Kress 2010).
Multimodal analysis, therefore, is interested in the available choices of the resources,
what meanings these resources can be used to produce and then how they also shape how
218 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

we make our societies. New and fresh semiotic production is going on all the time in the
processes of creativity, as meaning potentials are deployed in different ways. Yet these
potentials nevertheless are always ideologically loaded.
The notion of choice and semiotic resources can be applied to all modes of
communications, for example, the design of a webpage, an exhibition space, a sculpture
or a food package. In the case of designing food packages, it will involve not only
choosing the words that are placed on the package, but also the material, the texture
of the packaging, the shape of the package, the image, the colour and fonts and so on.
We will go into details with these inventories in the second part of the chapter. The idea
here is that if we could break down instances of communication in ways which would
create inventories of available semiotic resources, we can give a more detailed, systematic
and predictive form of analysis. Doing a multimodal analysis, therefore, is concerned
with documenting semiotic resources used in the object we analyse. After realizing what
options communicators use, we then can ask how these options are used, why these
options are used and what the consequences of using these options are.
As an example, we can see in our two fitness drinks in Figure 10.1 that there are
different semiotic choices between the two in terms of the thickness and curvature of
fonts, in the colours, the shape of the bottles, the use of iconography. We return to this
shortly to show how we can be more systematic about how we go about making such
descriptions.

SEMIOTIC MATERIALS, AFFORDANCES

AND CANONS OF USE
Besides the choice of semiotic resources, multimodality is also interested in the forms of
communication, what we call semiotic materials. Semiotic materials carry, or are formed
by, the semiotic choices. A food package, a monument, a document, a menu, a Facebook
page are all semiotic materials. A semiotic material can accomplish a particular kind of
communication act but not others. So semiotic materials all have affordances. Monuments
provide some affordances which a piece of paper does not. Crucially, a semiotic material
itself also comes loaded with ideas and assumptions, and shapes communication, social
organization and social interactions. A monument and a piece of A4 paper shape what
communication gets done and also the kinds of social interactions that take place. If we
want to commemorate a leader who fought for social justice, a piece of A4 paper may
not work so well as a large bronze figure in a town square. In the same way if we wanted
to send our grandmother a message that we missed her, the A4 sheet would work better
than the monument. Or a semiotic material like Facebook might do it. In each case a
particular kind set of ‘moves’ in communication are laid out by the semiotic material.
These allow certain things but not others, therefore templatizing communication and
social interaction. Semiotic materials relate the bonds within which communication
occurs as well as shaping the kinds of interpersonal and social relationships which can
take place. Semiotic materials, like monuments, a sheet of A4, a Facebook page, create a
kind of roadmap which we can fill with our semiotic choices.
This templatization through semiotic materials means that the semiotic choices we
make are done and encountered through what we can call ‘canons of use’. This term is
useful as it allows us to capture how semiotic materials are always part of social practices.
When we encounter semiotic choices in a material object, such as food-packaging, as
MULTIMODALITY AND SEMIOTICS 219

in our two examples in Figure 10.1, we do not have to make sense of meaning-making
through the details of the semiotic choices such as fonts, colours and iconography, but
understand this already as an instance of food packaging which is used to shape and load
products with ideas and values.
Canons of use relate to the traditions of use of a semiotic material as well as the kind of
semiotic choices that we find with them. Photographs, for example, have different canons
of use in photojournalism, in advertising, personal social media accounts, in galleries. In
each case they come with different established meanings, and behind each lies different
processes of production and consumption. So, in each case the use of the same kinds of
semiotic choices may even take on slightly different kinds of meaning.
In this model then there are three levels. There is the micro-level. This is the semiotic
material, or ‘text’, which we encounter in situations, like the monument, the A4 sheet,
a Facebook page or a fitness drink bottle. This semiotic material, the text, is part of
culture, of a tradition of shared ideas, beliefs and behaviours within social groupings. This
would include all kinds of artefacts and technologies, such as computers, cameras, fitness
equipment and other products.
There is the meso-level, which are the canons of use. This accounts for the meaning-
making potential for each semiotic material in contexts. So a photographer, or food
packaging designer, will use the cultural resources of their moment in time, such as a
digital camera, design software, to make meaning aligned with the existing canons of use,
or the typical and established ways of using photography or food packaging in a culture.
This leads to the macro-level where the broader available semiotic choices carrying
meaning potentials, like words, colours, textures, iconography, will be deployed. To a
large extent this will be routine. For example, a photojournalist may use a certain kind
of image and style to connote ‘war and suffering’, say an image of a dirty faced child,
standing alone outside a bombed-out house. A food-packaging designer may communicate
fitness and strength, as we see in the left-hand image in Figure 10.1, by using bold fonts,
saturated colours and a bottle whose shape appears stable, rather than tall and elegant.
Or they may do so by using a slimmer, more gently curved font, a curved bottle, images
of fruits and a busy list of ‘healthy’, properties, such as ‘gluten free’ which, if you are not
gluten intolerant, will have no health benefits and means in fact much processing (De
Giorgio, Volta, and Gibson 2016).
Put simply materiality and communication become interdependent (Bateman et al.
2017). And communication is not simply here about representation, but rather comes
to be infused into our material world such that it forms and gives meaning to the world
in which we live (Hjelmslev 1961). And as we can see this materiality houses ideas and
values. News photographs claim to represent wars, although they tell us nothing of the
political and economic motivations which cause them. Notions of health and physical
fitness become presented and shaped to serve food marketing.
Multimodality will carry out detailed analyses of the semiotic choices found on
instances of communication such as photographs or food packaging. But this analysis
must be placed in the context of the forms of activities, ideas and values of which they are
part. And we must analyse those semiotic choices within their typical pattern of use in a
canon of use. A word, a colour, a form of typeface may have slightly different meaning
potential depending on where it is deployed. And this is an important methodological
point since one weakness with the more systemic based work in multimodality is that it
seeks to create one-size-fits-all models of semiotic choices regardless of context. It also
means that we must be aware of how a semiotic material, like a food package, tends to be
220 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

used typically as a canon of use, but also that designers are using the cultural resources of
their time, which afford certain kinds of communication and uses of semiotic resources
(cf. Bezemer and Kress 2016).
In the case of food packaging technology has brought about huge changes, for example,
in regard to how liquids can be stored and presented. Before such technology, package
designers would not be able to choose to use paper base packages to communicate a
more environmentally friendly image of a milk product, for example. As well as this
we must also grasp how there are constraints on the uses of materials and semiotic
choices. Food-packaging regulation itself shapes this. The EU, regulation No 1169/2011
maps out how some specific semiotic resources can be used and leaves other semiotic
resources relatively free in their use. For example, a list of ingredients is mandatory
information that is required to be placed on the packaging in a specific way. There is
also regulation No 432/2012 which defines when and which specific health claims can be
used on food packages. These regulations are established to ensure the communication
on food packages, according to the regulation, protects and does not mislead consumers.
However, the use of other semiotic resources like the shapes, textures and colours is left
with a larger degree of flexibility in their usage. This flexibility gives food packages the
affordance to suggest ideas and values of a product to consumers. Looking at the two
fitness products in Figure 10.1, research shows there is no evidence that these whey-
based drinks will actually contribute to muscle building nor to fitness per se (Hansen
et al. 2016). But if we look closely, as we do shortly, no such claim is actually made. It is
communicated symbolically through fonts, colours and through buzzwords.

DISCOURSES AND DISCURSIVE PRACTICE

This particular social semiotic and multimodal approach also draws on Foucault’s (1972)
notion of discourse. Here discourses are like models that consist of our understandings
of how things work in society. One discourse that has been extensively studied is the
discourse of gender. In this discourse, people believe that, for example, there are certain
characteristics, kinds of appearance and behaviours that belong to or are more appropriate
for one gender but not others. Such discourse can appear simply natural and inevitable.
But discourse is also about power and ideology and the struggle for the definition of
reality (Flowerdew and Richardson 2017). So, multimodality here would be interested
in analysing instances of communication to reveal how semiotic choices carry certain
discourses and ideologies, legitimize and naturalize certain power interests.
The notion of discourse is highly useful for approaching the analysis of semiotic
resources in the case of our two fitness drinks found in Figure 10.1. In the two decades
of the twenty-first century these has been a rising popularity of high protein diets, driven
by people’s desire to maximize their exercise outcomes (Hartmann and Siegrist 2016).
This must in itself be placed in a surge in interest in gyms and body management (Chen
and Eriksson 2019). In supermarkets, we have seen an increasing number of products
being marketed highlighting their high protein content and suggesting that this relates
to a healthy and active lifestyle. Research shows that consumers believe protein has the
capacity to increase training outcomes, reduce risk of injuries and illness, improve health
and help control body weight (Heikkinen et al. 2011; Royne et al. 2014). However,
nutritional scientists have reported that there is no conclusive evidence of a physiological
effect of a high protein diet (Pasiakos et al. 2015; Churchward-Venne et al. 2016; Hansen
MULTIMODALITY AND SEMIOTICS 221

et al. 2016). Some researchers even argue that protein supplementation is unnecessary
for healthy adults participating in general fitness programmes (Kreider et al. 2010). The
myth of protein in relation to fitness points to discourses that are carefully constructed
for specific consumer target groups in the interests of increasing sales for food producers
(Chen and Eriksson 2019).
What these observations show is the need to place the analysis of a set of semiotic
choices within a set of discourses prevalent at any given time. So the designers here, or the
marketing team, are using the technological and cultural resources of their time, aligned
with existing canons of use for food packaging in a particular culture to load their protein
products with a set of ideas and values related to muscle building and fitness. Clearly here
semiotic materials relate to activities within cultures and to ideas and behaviours within
social groupings.
The term discursive practices (see Fairclough 1992) is also useful here to think about
how instances of communication, semiotic materials and the semiotic choices deployed to
form them, such as in the case of the fitness drinks, belong always in a broader framework
of behaviours, forms of social interactions, social processes which are realizations of
discourses. Put simply, how the world represented through communication is related to
and influences how we think, talk, act and live. In this viewpoint, the protein discourse
discussed earlier, which is loaded with commercial interests, plays a role in how we shop,
eat, see our and others’ bodies and value our and others’ way of living. The production
part of making such drinks both draws on these discourses and helps to shape, drive and
legitimize them.
A multimodality project is, therefore, a project of revealing discourses, the kind of
discursive practice that is involved and the ideology they serve. Semiotic materials should
be thought about as how the use of them has become established in social and political
contexts in order to serve particular interests. The choices in semiotic resources should be
examined through how they are related to ideology.

­ NALYSING FOOD PACKAGING AS PART OF

A
A DISCOURSE OF GENDERED HEALTH AND FITNESS
What should be clear at this point is that social semiotics and multimodality are only
effective and useful when applied to specific instances of communication understood
within context. And it must be paired with theories and practices from other disciplines
(Van Leeuwen 2005). For example, we are applying multimodality to the specific issue
of how food packaging communicates ideas about gender and fitness; this must be placed
against existing scholarship in these areas. This is necessary to help us to formulate the
focus of the research question (Bateman et al. 2017). In this case, we want to understand
more about the two fitness drink packages in Figure 10.1, relating to how whey protein
drink is marketed in two different ways targeting men and women. We want to know
what discourses are used and how these are communicated. So, in this case we also need
to draw knowledge from scholarship in packaging design, marketing, food studies and
gender studies. Here we are able to create more context.
Food studies tells us that food should be understood not merely through its function
to sustain life, nor about culinary pleasure, but also through what it symbolizes socially
and culturally (Falk 1994; Beardsworth and Keil 1997). What is considered healthy food
can in itself differ across cultures and also across socioeconomic groups, for example,
222 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

where middle-class consumers favour natural, handmade and organic foods as these
bring associations of simplicity and authenticity rather than anonymous industrial
production (Ledin and Machin 2018). Studies also show that food preferences can be
highly gendered (Madden and Chamberlain 2004) and we see how food advertising is
involved in (re)producing these ideas (see, e.g., Bordo 1993; Labre 2005; Gough 2007).
In regard to healthy eating, while it is known that this is a growing market for men, this
tends to relate to fitness and specifically the idea of developing muscles, bulking up and
physical power/performance (Gough 2007). Researchers have long shown that physical
form can substitute actual bodily health, where musculature can symbolize strength and
vitality, even where it comes from diet and training regimes which place stress on the
body (Markula 1997).
On the other hand, media advertising of healthy food to women carries a different
emphasis on bodily appearance (Schneider and Davis 2010). Here good food choices tend
to be related to low-fat food and/or food that allows women to stay slender and therefore
sexy and desirable or for them to maintain energy levels to be capable caregivers (Madden
and Chamberlain 2004). More recent healthy lifestyle trends, such as ‘fitspiration’,
have shifted to also emphasize strength and empowerment over being skinny (Boepple
et al. 2016; Tiggemaann and Zaccardo 2018). It is found that images associated with
‘fitspiration’ contain mainly healthy food such as low-fat/calories salad and smoothies, as
well as protein-type supplements and overwhelmingly represent one body shape, thin and
toned (Boepple et al. 2016; Tiggemaann and Zaccardo 2018).
Scholars have documented the way that muscle became gendered as part of the rise
of ideals of femininity as soft, non-aggressive and passive (Ian 1991). But in the second
decade of the twenty-first century we have seen increasing representations of women in
advertising which carries a rhetoric of empowerment (Toffoletti 2014). Rather than being
passive objects to be looked at they have become independent, dynamic, assertive and have
their own sexual desires (Gonick 2006; Gill 2008). This has been called a kind of ‘power
femininity’ (Lazar 2006: 505). Some have related this to the rise of neoliberalism which
demands that the self must be go-getting, focused, successful, where any form of passivity
is a sign of unproductivity (Walkerdine 2003). Fitness regimes and the drive for a stronger
more muscular body in the case of women can be seen as part of this striving for self-
improvement (Scott 2011). For Dawson (2015) this is about self-management, improving
outputs. Cohen and Colino (2014) and Elliot (2013) also help us to think about how the
appearance of more young women with toned and muscular bodies in the media also relates
to a culture of consumerism which demands the constant ‘new’ and reinvention, where the
self becomes one site where this takes place. And this is an era where we are judged chiefly
on our consumption practices (Bauman 2011) whether we eat well, if we shop ethically, etc.
Scholars have also shown how food products marketed as healthy often simply
offer a hodgepodge of health buzzwords which are often contradictory and confusing
for consumers (see, e.g., Lewis 2008; O’Niell and Silver 2017) and may even be used
to brand foods that are otherwise pretty unhealthy (Chen and Eriksson 2019). While
claims are regulated, buzzwords and packaging designs can, nevertheless, be used to
signify healthiness and simplicity (Eriksson and Machin 2019). Research has shown
that packaging designs can be highly influential to consumers’ perceptions and buying
behaviour (Underwood, Klein and Burke 2001; Clement, Kristensen and Gronhaug
2013; Clement et al. 2017) and it has been shown that gender coding is one way this is
accomplished through design features, including colour, shape, texture, logos, packaging
and brand names (Alreck 1994).
MULTIMODALITY AND SEMIOTICS 223

Through this brief literature review we have been able to provide some of the context
against which we can now go on to carry out the textual part of multimodal analysis. We
know something about the production and discourses loaded onto these products. And
we know something about the social practice, ideas and values of which they are apart.
These are gender-targeted notions of fitness, in a cultural context where fitness and body
management are part of neoliberal ideas and values associated with self-improvement.
This has been particularly transforming for women’s notions of the body with the rise of
the valuing of musculature, strength and power. We now go on to look closely at some
different health and fitness food packages to see what kinds of semiotic resources they
carry, so that we can consider what kind of discourse they communicate.

ANALYSING PACKAGING: HOW GENDER HAS BEEN

INFUSED IN THE DISCOURSE OF FITNESS FOOD
ON FOOD PACKAGES
In what follows we present an analysis of the packages under a number of headings each
of which presents tools of observation typically used in multimodality. We look at shape,
size, textures, colour, fonts, iconography. Each of these can be seen as a set of meaning
potentials. So what kind of shape or texture is chosen and what does it communicate? In
multimodality it is usual to present the available repertoire of semiotic choices as a list of
alternatives, such as hard-soft, rough-smooth, angular-curved where each tends to come
with meaning potentials. These are based on distinctions, identified by a commutation
test (Ledin and Machin 2020). So, in any instance of a semiotic choices we can change
them to see what happens. In practice this leads to inventories of opposites which sit on a
continuum such as light-dark, saturated dilute, rough-smooth. Such inventories have their
basis in what are called ‘system networks’, in Hallidayan linguistics. These are the systems
and layers of choices available to speakers as they build up meaning (Halliday 1978). Here
we do not have the space to present full inventories, nor to carry out a complete analysis
of all the qualities and features of food packages such as shape, materials, composition,
colour, iconography. But we are able to provide an example of how this works looking
at typeface and textures.
To some extent this involves an artificial way of going about analysis as the qualities
and features these tools address are never encountered individually as we look at or
interact with the packages. We meet them as semiotic materials in contexts where they
are already meaningful. But this kind of step-by-step analysis allows us a more systematic
level of observation.

WRITING AND TYPOGRAPHY

An obvious difference on the two drinks bottles in Figure 10.1 is the fonts that are used.
At first glance we might say that the fonts on the bottle marketed at women are simply
more feminine. But in multimodal analysis we want to ask how they communicate this.
And sometimes looking a little more systematically reveals not exactly what we expected.
So, in multimodality we would have a kind of inventory of meaning potentials for
typefaces. These are based on what we call ‘experiential associations’, in other words
associations in our everyday world. So wider objects tend to be more stable than narrow
ones. Angular objects are harsher than curved ones, etc. Here is such an inventory for
224 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

typefaces. See Ledin and Machin (2020) and Van Leeuwen (2006) for a full account of
how such inventories are created.
1. Weight – bold can mean substantial, stable, strong or overbearing. Slight can mean
timid, insubstantial or subtle
2. Expansion – typefaces can take up space confidently or can look cramped, or they
can appear gentle and unassuming
3. Curvature – angular can mean technical harsh or masculine. Curved can mean
organic, nurturing feminine
4. Orientation – tall can mean aspirational or arrogance. Squat can mean down to
earth, stable or inertia
5. Regularity – irregular can mean fun, chaos, non-conformist; regular can mean
order, serious, formal, conformist
Applying these to the two fitness drinks we see that the UFIT package caries heavy and
wide fonts for the brand name. This can suggest that the product is solid and substantial.
The font is also rounded, with no sharp edges, which can point to a smooth and silky
texture of the drink or also suggest the roundedness of chubby muscles. The word ‘fit’ is
also quite tall suggestion aspirational and certainly not as heavy as the number ‘fifty’. So
as well as bulk this suggests something about mobility. The Slimfast drink has slimmer
fonts overall, pointing to lightness rather than bulk, especially for the qualities like
‘High in Fibre’. We find curvature on the word ‘Slimfast’, but also angularity on the
world ‘Vitality’ and on some of the drink qualities. Here this gives a sense of something
technical of ‘information’. What we see is that on packages different font qualities can
be used for different sections. So, curvature may be used in a brand word to suggest
something emotional, expressive, but angular fonts to symbolize facts. We also find
more font regularity on the Ufit package, where we find different sizes of the same
typeface. The Slimfast package uses a range of different fonts and also different weights
of the same font in the same word, such as in ‘Vitality’, here suggesting something
dynamic, fun even.
In terms of fonts then we see some typical older notions of gender. The men’s product
is about bulking up; the women’s is about slimming and energy. We also see that the
women’s product presents a range of health-related buzzwords, whereas the men’s does
not. And it also carries these buzzwords using more formal, technical typefaces.
What we begin to see from this analysis is that, at the level of font at least, we find well-
documented gendered discourses of the meaning of fitness. For men it is about bulk, or at
least power, for women, more about slimming and having energy, as well as actual health
properties being important – here where fonts present them as information. Nothing on
the men’s product carried this level of formality.

TEXTURES
We now move onto a different semiotic resource used by packaging designers and that
is texture. We also introduce two new fitness foods where the same basic content is
marketed differently for men and women. There are snack bars seen in Figures 10.2 and
10.3. Both of these are made from plastic; yet, very different textures have been used
on each. Textures too can be broken down into a number of meaning potentials. For a
MULTIMODALITY AND SEMIOTICS 225

more detailed account of textures see Abousnnouga and Machin (2013), Djonov and Van
Leeuwen (2011) and Ledin and Machin (2018).
1. Rigidity – surfaces can be hard or give to the touch. Tough, uncompromising solid,
versus gentle, yielding, accommodating
2. Relief – surfaces can be flat or have contours, ridges, or this can simply be rough or
smooth. This can mean natural versus unnatural/processes versus handmade
3. Regularity – surfaces can be irregular which can mean interest, complexity,
individually made, while regular surfaces can be controlled, regulated, clean and
certain, or even dull and conformist
4. Natural-artificial – the organic, unprocessed, personal versus processed, or technical
impersonal
5. Sticky – this can mean dirty, but also the ability to grip as in rubber floors, so
comfort. Unsticky can be clean but also uncomforting as in a marble floor
6. Liquidity – wetness can mean life and vitality or putridness. Dryness can mean
cleanness but also lack of vitality.
The packages in Figures 10.2 and 10.3, the SCI-MX and Bounce, both use basically
plastic. However, the SCI-MX has a metallic and glossy surface, accomplished with
layering of polymer film and metal. It appears metallic, shiny, polished and technical and
is smooth to the touch. In contrast The Bounce package has a slightly rough, or grainy,

FIGURE 10.2 SCI-MX protein.

226 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 10.3 Bounce protein energy ball.

texture suggesting something more natural, less processed and polished. The surface of
the SCI-MX package is without relief and is regular also pointing to control, consistency,
processing. The Bounce package, in contrast, fits loosely round the product and is
slightly thicker, though also comprising polymer, metal layering, which creates relief and
irregularity, suggesting less consistent, less polished processing. Both products are high in
fat and sugars, the latter of which provides the energy content. Yet we can see that one
foregrounds something technical and scientific, the other the natural and the handmade.
Returning to the fonts we can also see that the SCI-MX bar carries more angular and
squat fonts compared to the rounded, even hand-written, personalized and slightly taller
fonts on the Bounce bar. Of course, the very names of the products themselves bring
different associations, of the scientific, versus fun.
We can also think about textures in relation to the drink bottles in Figure 10.1.
Sometimes men’s fitness drinks have stickiness with polystyrene grip-type bands a round
them to suggest a kind of functionality and activeness, suggestive of the kinds of grips
found on power tools. Looking at the two bottles we can see the men’s product has a more
MULTIMODALITY AND SEMIOTICS 227

regular surface with some contouring around the middle suggesting a kind of grip zone.
The women’s product has more variety in relief with its slimmer mid-section and curved
top. Both bottles have a rigid texture, unlike some drink bottles for natural products
which may use thinner plastics to suggest being closer to the product. Here rigidity can
be important to suggest the strength and dynamism provided by the product. A fitness or
bulking drink in a very thin plastic which gave to the touch may not be quite appropriate.
Other drinks bottles also use textures which carry liquidity to suggest vitality, or may have
textured paper round them to communicate naturalness as in organic yoghurt drinks.
At the level of texture, we can also see then that health-related foods are gendered in
quite traditional terms. But for the snack bars it is slightly different. Here the men’s product
is linked with science and technology. The women’s product is more personalized and
natural. There are differences between the drink bottles and the snacks, but in both cases we
find that for the women’s products the list of buzzwords appears with more formal fonts.
Of course, gender is no single consume category in marketing terms and these products
will be aimed at different platforms. So the female target group for the Bounce bar may
not be the same as that for the Slimfit drink. But nevertheless, here we find gendering in
both places. And, ultimately, we might want to ask what this means when ‘health’ is so
communicated to men and women, and to different market platforms in different ways.

CONCLUSION
The analysis of these food-packaging examples only begins to scratch the surface; and,
of course, there are still other aspects of even these food packages that we could have
looked at, such as colour, iconography, shape or materials. But here the aim was to purely
introduce how multimodality goes about analysis.
Multimodality looks to carry out high detailed and systematic analyses of instances of
communication. But this must always be done with context in mind, so, understanding
the nature of the semiotic material we are analysing the canons of use to which they are
associated, and also the discourse and discursive practices of which they are a part. All
of these help us to make sense of how and why semiotic choices are made as they are
designed. To do this we need to provide some sociological depth by considering relevant
literature which has already built up established knowledge.
Multimodality can be applied for the analysis of many instances of communication
using different semiotic materials as data, such as film, documents, data presentation
and monuments (Ledin and Machin 2018). But analysis should be driven by a research
question, which is clear and also clarified by reference to wider literature on the topic. And
multimodality is best used in relation to the wider project of drawing out what kinds of ideas
and values are buried within the communication and the semiotic materials around us, like
news websites, social media platforms, the design of buildings and food marketing. In this
chapter this very superficial analysis has already pointed to the way that food-marketing
companies seek to colonize and shape discourse and practices of taking care of our bodies.

REFERENCES
Abousnnouga, G. and D. Machin (2013), The Language of War Monuments, London:
Bloomsbury.
Alreck, P. (1994), ‘Commentary: A new Formula for Gendering Products and Brands’, Journal
of Product & Brand Management, 3: 6–18.
228 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Barthes, R. (1977), Image, Music, Text, London: Fontana.

Bateman, J., J. Wildfeuer and T. Hiippala (2017), Multimodality: Foundations, Research and
Analysis: A Problem-Oriented Introduction, Berlin: de Gruyter Mouton.
Bauman, Z. (2011), Collateral Damage: Social Inequalities in a Global Age, Cambridge and
Malden: Polity Press.
Beardsworth, A. and T. Keil (1997), Sociology on the Menu, New York: Routledge.
Bezemer, J. and G. Kress (2016), Multimodality, Learning and Communication: A Social
Semiotic Framework, London: Routledge.
Boepple, L., R. Ata, R. Rum and K. Thompson (2016), ‘Strong Is the New Skinny: A Content
Analysis of Fitspiration Websites’, Body Image, 17: 132–5.
Bordo, S. (1993), Unbearable Weight: Feminism, Culture, and the Body, Berkeley: University of
California Press.
Chen, A. and G. Eriksson (2019), ‘The Mythologization of Protein: A Multimodal Critical
Discourse Analysis of Snacks Packaging’. Food, Culture & Society, 22 (4): 423–45.
Churchward-Venne, T., A. Holwerda, S. Phillips and L. Van Loon (2016), ‘What Is the
Optimal Amount of Protein to Support Post-exercise Skeletal Muscle Reconditioning in the
Older Adult?’ Sports Medicine, 46 (9): 1205–12.
Clement, J., T. Kristensen and K. Grønhaug (2013), ‘Understanding Consumers’ In-store Visual
Perception: The Influence of Package Design Features on Visual Attention’, Journal of
Retailing and Consumer Services, 20 (2): 234–9.
Clement, J., V. Smith, J. Zlatev, K.Gidlöf and J. Van de Weijer (2017), ‘Assessing Information
on Food Packages’, European Journal of Marketing, 51 (1): 219–37.
Cohen J. and S. Colino (2014), How Strong Is the New Skinny: How to Eat, Live and Move to
Maximize Your Power, New York: Harmony.
Dawson, M. (2015), ‘CrossFit: Fitness Cult or Reinventive Institution’, International Review for
Sociology of Sport, July: 1–19.
De Giorgio, R., U. Volta and P. Gibson (2016), ‘Sensitivity to Wheat, Gluten and FODMAPs in
IBS: Facts or Fiction?’, Gut, 65: 169–78.
Djonov, E. and T. Van Leeuwen (2011), ‘Bullet Points, New Writing, and the Marketisation
of Public Discourse: A Critical Multimodal Perspective’, in E. Djonov and S. Zhao (eds),
Critical Multimodal Studies of Popular Discourse, 232–50, London: Routledge.
Elliot, A. (2013), Reinvention, London: Routledge.
Eriksson, G. and D. Machin (2019), ‘Discourses of “Good Food”: The Commercialisation of
Healthy and Ethical Eating’, Discourse, Context & Media, 33: 1–7.
­Fairclough, N. (1992), Discourse and Social Change, London: Polity Press.
Falk, P. (1994), The Consuming Body, London: Sage.
Flowerdew, J. and J. Richardson (2017), The Routledge Handbook of Critical Discourse Studies,
London: Routledge.
Foucault, M. (1972), The Archaeology of Knowledge, New York: Pantheon Books.
Gill, R. (2008), ‘Empowerment/Sexism: Figuring Female Sexual Agency in Contemporary
Advertising’, Feminism & Psychology, 18 (1): 35–60.
Gonick, M. (2006), ‘Between “Girl Power” and “Reviving Ophelia”: Constituting the
Neoliberal Girl Subject’, NWSA Journal, 18 (2): 1–23.
Gough, B. (2007), ‘“Real Men Don’t Diet”: An Analysis of Contemporary Newspaper
Representations of Men, Food and Health’, Social Science & Medicine, 64 (2): 326–37.
Halliday, M. (1978), Language as Social Semiotics: The Social Interpretation of Language and
Meaning, London: Edward Arnold.
MULTIMODALITY AND SEMIOTICS 229

Hansen, M., J. Bangsbo, J. Jensen, M. Krause-Jensen, B. Bibby, O. Sollie, U. Hall and K.

Madsne (2016), ‘Protein Intake during Training Sessions Has No Effect on Performance and
Recovery during Strenuous Training Camp for Elite Cyclists’, Journal of the International
Society of Sports Nutrition, 13: 9–19.
Hartmann, C. and M. Siegrist (2016), ‘Benefit Beliefs about Protein Supplements: A
Comparative Study of Users and Non-users’, Appetite, 103: 229–35.
Heikkinen, A., A. Alaranta, I. Helenius and T. Vasankari (2011), ‘Dietary Supplementation
Habits and Perceptions of Supplement Use among Elite Finnish Athletes’, International
Journal of Sport Nutrition and Exercise Metabolism, 21 (4): 271–9.
Hjelmslev, L. (1961), Prelegomena to a Theory of Language, Madison: University of Wisconsin
Press.
Hodge, B. and G. Kress (1988), Social Semiotics, Cambridge: Polity Press.
Ian, M. (1991), ‘From Abject to Object: Women’s Bodybuilding’, Postmodern Culture, 1 (3):
1–17.
Jewitt, C., J. Bezemer and K. O’Halloran (2016), Introducing Multimodality, London and New
York: Routledge.
Kreider, R., C. Wilborn, L. Taylor, B. Campbell, A. Almada, R. Collins, M. Cooke, C. P.
Earnest, M. Greenwood, D. S. Kalman, C. M. Kerksick, S. M. Kleiner, B. Leutholtz, H.
Lopez, L. M. Lowery, R. Mendel, A. Smith, M. Spano, R. Wildman, D. S. Willoughby, T. N.
Ziegenfuss and J. Antonio (2010), ‘ISSN Exercise & Sport Nutrition Review: Research &
Recommendations’, Journal of the International Society of Sports Nutrition, 7 (7): 2–43.
Kress, G. (2010), Multimodality: A Social Semiotic Approach to Contemporary Communication,
London: Routledge.
Kress, G. and T. Van Leeuwen (1996), Reading Images: The Grammars of Visual Design,
London: Routledge.
Kress, G. and T. Van Leeuwen (2001), Multimodal Discourse: The Modes and Media of
Contemporary Communication, London: Arnold Publishers.
Labre, M. (2005), ‘Burn Fat, Build Muscle: A Content Analysis of Men’s Health and Men’s
Fitness’, International Journal of Men’s Health, 4 (2): 187–200.
Lazar, M. (2006), ‘“Discover the Power of Femininity!”: Analysing Global “Power Femininity”
in Local Advertising’, Feminist Media Studies, 6 (4): 505–18.
Ledin, P. and D. Machin (2018), Doing Visual Analysis: From Theory to Practice, London: Sage.
Ledin, P. and D. Machin (2020), Introduction to Multimodal Analysis, London: Bloomsbury.
Lewis, T. (2008), Smart Living: Lifestyle Media and Popular Expertise, New York: Peter Lang.
­Madden, H. and K. Chamberlain (2004), ‘Nutritional Health Messages in Women’s Magazine:
A Conflicted Space for Women Readers’, Journal of Health Psychology, 9 (4): 583–97.
Markula, P. (1997), ‘Are Fit People Healthy? Health, Exercise, Active Living and the Body in
Fitness Discourse’, Waikato Journal of Education, 3: 21–39.
O’Neill, K. and D. Silver (2017), ‘From Hungry to Healthy: Simmel, Self-cultivation and
Transformative Experience of Earring for Beauty’, Food, Culture & Society, 20 (1): 101–32.
O’Toole, M. (1994), The Language of Displayed Art, London: Leicester University Press.
Pasiakos, S., T. McLellan and H. Lieberman (2015), ‘The Effects of Protein Supplements on
Muscle Mass, Strength, and Aerobic and Anaerobic Power in Healthy Adults: A systematic
Review’, Sports Medicine, 45 (1): 111–31.
Royne, M., A. Fox, G. Deitz and T. Gibson (2014), ‘The Effects of Health Consciousness
and Familiarity with DTCA on Perceptions of Dietary Supplements’, Journal of Consumer
Affairs, 48 (3): 515–34.
230 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Schneider, T. and T. Davis (2010), ‘Fostering a Hunger for Health: Food and the Self in ‘The
Australian Women’s Weekly’’, Health Sociology Review, 19 (3): 285–303.
Scott, S. (2011), Total Institutions and Reinvented Identities, Basingstoke: Macmillan.
Tiggemann, M. and M. Zaccardo (2018), ‘“Strong Is the New Skinny”: A Content Analysis of
#fitspiration Image on Instagram’, Journal of Health Psychology, 23 (8): 1003–11.
Toffoletti, K. (2014), ‘Iranian Women’s Sports Fandom: Gender, resistance, and Identity in the
Football Movie Offside’, Journal of Sport & Social Issues, 38: 75–92.
Underwood, R., N. Klein and R. Burke (2001), ‘Packaging Communication: Attentional Effects
of Product Imagery’, Journal of Product & Brand Management, 10 (7): 403–22.
Van Leeuwen, T. (2005), Introducing Social Semiotics, London: Routledge.
Van Leeuwen, T. (2006), ‘Towards a Semiotics of Typography’, Information Design Journal +
Document Design, 14 (2): 139–55.
Walkerdine, V. (2003), ‘Reclassifying Upward Mobility: Femininity and the Neo-liberal
Subject’, Gender and Education, 15 (3): 237–48.
 CHAPTER ELEVEN

Discourse Analysis and

Semiotics
KAY L. O’HALLORAN AND SABINE TAN

INTRODUCTION
Semiotics provides the foundations for discourse analysis, most notably multimodal
discourse analysis which is concerned with the meaning arising from the integration of
language, images and other resources in semiotic artefacts and processes (e.g. O’Halloran
2021). In particular, Michael Halliday’s social semiotic approach to the study of language
(e.g. Halliday 2009) provided key theoretical concepts which Kress and van Leeuwen
(2006, 2020) and O’Toole (2011) further developed for the study of images and displayed
art. From these early beginnings, multimodal discourse analysis was developed and
applied to other forms of semiosis such as music (van Leeuwen 1999) and mathematics
(O’Halloran 2015), leading to the field which is generally known as ‘multimodality’
(Jewitt, Bezemer, and O’Halloran 2016; Bateman, Wildfeuer, and Hiippala 2017).
Although there are other approaches to the study of multimodal phenomena (e.g.
cognitive approaches, mediated discourse analysis and interactional analysis), semiotics
provides the basis for the most widespread approaches. In this chapter, key concepts
in semiotics which provide the underlying foundations for multimodal discourse
analysis (e.g. system, metafunction, intersemiosis and resemiotization) are explored and
illustrated. In addition, recent research efforts aimed at developing digital approaches to
multimodal discourse analysis and multimodal mixed methods approaches to big data are
explored. The discussion aims to show how semiotics has provided the basis for various
cross-disciplinary movements with the potential to develop into the multimodal sciences
in the future (O’Halloran et al. 2020).

SEMIOTIC APPROACHES TO DISCOURSE ANALYSIS

The term ‘discourse analysis’ covers a broad terrain, from linguistically orientated
approaches which account for how language is used to achieve certain objectives, to
sociolinguistic approaches which investigate social, political and psychological features of
texts (see Kress 2011). In these traditions, the focus is spoken and written language and
non-verbal communication, driven by the need to understand how knowledge and power
are constructed and enacted through language use (e.g. Tannen, Hamilton and Schiffrin
2015). However, since the mid-1990s when the term ‘multimodality’ first started to
appear (e.g. Jewitt et al. 2016; Machin 2013, 2016), multimodal discourse analysis,
232 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

which is concerned with the meanings arising from the integration of language and other
semiotic resources (image, sound, gesture, etc.), has become a well-established research
tradition (e.g. Kress 2011; Tan, O’Halloran and Wignell 2020; O’Halloran 2021). As
van Leeuwen (2015: 447) explains, given the increasing multimodality of contemporary
communication, ‘[c]learly discourse can no longer be adequately studied without paying
attention to multimodality’. Following van Leeuwen (2015), this chapter is concerned
with the ways in which multimodality is conceptualized and studied by linguists and
semioticians.
Social semiotics provides the theoretical foundations for the most widespread
approaches to multimodal discourse analysis today, although other approaches to the
study of multimodal phenomena exist, such as cognitive approaches (e.g. Forceville
1996; Forceville and Urios-Aparisi 2009), mediated discourse analysis (e.g. Scollon 2001;
Scollon and Scollon 2004) and interactional analysis (e.g. Norris 2004; Norris and Jones
2005). The term ‘social semiotic’ is derived from Halliday (1978) but its origins can be
traced back to two different, but complementary linguistic and philosophic branches of
semiotics developed in the late nineteenth and early twentieth centuries by the American
philosopher Charles S. Peirce and the Swiss-French linguist Ferdinand de Saussure.
These works in turn inspired the ground-breaking work in semiotics, literary criticism
and social theory in the mid-twentieth century by the Russian formalists and the Paris
schools of semiotics, as exemplified, for example, by the works of Russian philosopher
and literary critic Mikhail Bakhtin, and French philosopher Roland Barthes (e.g. see van
Leeuwen 2015). Social semiotics is aligned with cultural semiotics developed by Lotman
and the Tartu-Moscow School (e.g. Lotman and Uspensky 1978; Lotman [1984] 2005)
where culture is understood as interacting systems of meaningful signs (e.g. Halliday
and Hasan 1985).
The concept of multimodality arose in the pioneering works of Gunther Kress and
Theo van Leeuwen, who introduced the term in their books Multimodal Discourse: The
Modes and Media of Contemporary Communication (2001) and Reading Images:
The Grammar of Visual Design (2006, 2020). Drawing on the combined principles
of social theory and social semiotics, and supplemented with insights drawn from art
history, film, iconography, structural semiotics and cognitive psychology, their work
provided the foundations for much multimodal research in the 1980s and 1990s.
O’Toole (2011) developed a complementary social semiotic approach, drawing on
insights from film studies, iconography and art history. O’Toole applied Halliday’s
social semiotic model to the analysis of a variety semiotic artefacts including displayed
art, paintings, sculpture and works of architecture, demonstrating how the approach
can be applied across the arts.
These and other approaches to multimodality inspired by social semiotics and critical
discourse analysis (e.g. van Leeuwen 2005, 2008; Machin and Mayr 2012; Machin 2013,
2016) provided the theoretical basis for what became known as multimodal discourse
analysis (MDA) (e.g. Kress 2011; van Leeuwen 2015; Jewitt, Bezemer and O’Halloran
2016; Bateman, Wildfeuer and Hiippala 2017). From these early beginnings, the social
semiotic approach to multimodal discourse analysis has evolved to become an ever-
growing platform for modelling, analysing and interpreting multimodal phenomena (e.g.
O’Halloran 2021), ultimately leading to approaches referred to as systemic functional
multimodal discourse analysis (SF-MDA) (e.g. O’Halloran 2008, 2021; O’Halloran, Tan
and Wignell 2019; Tan, O’Halloran and Wignell 2020) and multimodal critical discourse
analysis (MCDA) (e.g. van Leeuwen 2012; Machin 2013). The principles behind these
DISCOURSE ANALYSIS AND SEMIOTICS 233

approaches are elaborated below, with a focus on the social semiotic approach which
provides the foundations and inspiration for multimodal discourse analysis (MDA).

MULTIMODAL DISCOURSE ANALYSIS

Multimodal discourse analysis (MDA) largely derives from Halliday’s social semiotics,
where language, images and other semiotic resources are viewed as resources for
meaning-making. Originally developed and applied to the study of language, it formed
the basis for what is now known as System Functional Theory (SFT). Essentially, SFT is
regarded as a ‘theory of meaning as choice’ where language and other semiotic resources
are conceptualized as networks of interrelated systems of meanings which are organized
according to the functions that these semiotic resources serve in social contexts (e.g.
Halliday 1978; Halliday and Hasan 1985). The approach is closely aligned with critical
discourse analysis approaches to multimodality (e.g. Machin 2013; van Leeuwen 2015).
Among the key concepts from SFT is the notion that the communicative function of
any text or artefact is the result of specific choices which realize three strands of meaning
simultaneously. The three strands of meaning, conceptualized in SFT as ‘metafunctions’
(e.g. Halliday and Matthiessen 2014), are:
1. ideational meaning, which consists of experiential and logical meaning for
construing our experience of the world;
2. interpersonal meaning for enacting social relations, expressing attitudes and
creating a stance towards these expressions;
3. textual (or compositional) meaning for organizing these meanings into coherent
messages relevant to their context.
In language, the three strands of meaning are mapped on the structure of a clause by
specifying the grammatical systems through which these metafunctions are realized. For
example, experiential meaning is realized through the grammatical system of Transitivity
which accounts for the different types of processes that are found in a language, and
the structures through which they are expressed (Halliday and Matthiessen 2014:
211–335). Interpersonal meaning, in turn, is realized through (a) the system of Mood,
which realizes ‘meaning as an exchange’ (Halliday and Matthiessen 2014: 135–9), with
choices for giving and demanding information (statements, questions), and for giving
and demanding goods and services (offers, commands) and (b) the system of Modality
(that is, expressions of probability, usuality, obligation and inclination). Lastly, textual
meaning is realized through the thematic structure of the clause which organizes the
flow of information with clearly defined points of departure. Martin and colleagues (e.g.
Martin 1992; Martin and White 2005; Martin and Rose 2007) extended the systemic
functional approach to account for metafunctionally based systems of meaning at the
discourse level (i.e. paragraph and text).
A similar approach has been taken to develop metafunctionally based systems for images
and other semiotic resources (e.g. Kress and van Leeuwen 2006, 2020; O’Toole 2011).
Kress and van Leeuwen (2006, 2020) adopt a (top-down) contextual approach to the study
of images and visual design with particular emphasis on ideological meaning, while O’Toole
(2011) employs a (bottom-up) approach to derive metafunctionally based frameworks
which can be applied for the analysis of displayed art (e.g. paintings, architecture and
sculpture). Regardless of the orientation, the metafunctional principle plays a critical role
234 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

in determining the underlying organization of semiotic resources and for investigating the
ways in which semiotic choices combine and interact in a multimodal text or artefact.
Another key concept adopted from SFT is the notion of stratification, which posits that
the three strands of meaning are also realized through choices made at different ranks
or constituent levels according to the context. Language is seen as a stratified semiotic
system ‘involving three cycles of coding at different levels of abstraction’ (Martin and
White 2005: 8). In this model, the three metafunctions are realized at through choices in
discourse semantics and lexicogrammar, and through choices in phonology (for spoken
language) and graphology (for written text, e.g. as realized through font size, colour
and style). Following O’Toole (2011) who adopted this approach, O’Halloran (2008)
proposed a framework for the analysis of multimodal texts and images where language
is organized according to the constituent ranks of word, word group/phrase, clause and
clause complex, and images are organized according to the ranks of Member (Part),
Figure, Episode, Scene and Work.
SFT concepts have been adapted and extended for a wide range of multimodal texts
and artefacts, including visual art, music, children’s toys and games (e.g. Kress and
van Leeuwen 2001, 2006, 2020; O’Toole 2011; van Leeuwen 1999, 2005), popular
discourse (e.g. Djonov and Zhao 2013) and media practices (e.g. Djonov 2007; Knox
2007; Bednarek and Caple 2012; Stöckl, Caple and Pflaeging 2020) to account for
the ways in which multimodal resources combine and interact intersemiotically in
the communication of meaning. In each case, the multimodal texts and artefacts are
interpreted in relation to the situational and cultural context, following Halliday’s
(1978) social semiotic approach.
Expanding on this work, O’Halloran, Tan and Wignell (2019) propose an integrated
framework and approach for the analysis and interpretation of multimodal texts and
artefacts, including online media. An overview of these semiotic concepts, as interpreted
from a systemic functional multimodal discourse analysis perspective (SF-MDA), is
provided in Table 11.1. The analytical approach is demonstrated in the following section.

TABLE 11.1 Semiotic concepts for MDA (adapted from O’Halloran et al. 2016)
Metafunction Rank System Description
text
Experiential Discourse Configurations of activities Content in relation to
knowledge construction
Clause Processes; Participant Roles; Happenings, actions and
Circumstance relations
Interpersonal Discourse Appraisal Evaluation in terms of
attitude, emotion and
judgement
Clause Speech Function Exchange of information
(e.g. statements and
questions) and goods &
services (e.g. commands and
offers)
DISCOURSE ANALYSIS AND SEMIOTICS 235

Metafunction Rank System Description

text
Textual Clause & Information Focus Organization of
Discourse information, with points of
departure for what follows
Logical Clause Logico-Semantic Relations Logical relations between
Complex happenings, actions and
relations
Images
Experiential Work Narrative Theme; Nature of the scene
Representation; Setting
Episode Processes; Participant Roles; Visual happenings, actions
and Circumstance and relations
Figure Posture; Dress Characteristics of the
participants
Interpersonal Work Angle; Camera Distance; Visual effects
Lighting
Episode Proportion in Relation to Happenings, actions and
the Whole Image: Focus; relations with respect to the
Perspective whole image
Figure Gaze-Visual Address Direction of participant’s
gaze as internal to image or
external to viewer
Textual Work Compositional Vectors; The organization of the
Framing parts as a whole, with the
visual marking (e.g. framing)
of certain parts
Episode Relative Placement of Position of the happenings,
Episode; Framing actions and relations in
relation to the whole image,
and the visual marking of
certain aspects
Figure Relative Placement of the Position of figures in
Figure within the Episode; relation to happenings,
Arrangement; Framing actions or relations and the
visual marking of certain
aspects of those figures

KEY CONCEPTS WITH ILLUSTRATIONS

The multimodal social semiotic approach can be applied to the analysis of multimodal texts
by drawing on semiotic concepts for text and image analysis displayed in Table 11.1. The
approach is demonstrated through the analysis of terrorist propaganda material drawn
from the English-language magazines Dabiq and Rumiyah, the official online magazines
236 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

produced by the violent extremist terrorist organization known as Islamic State, or ISIS.
Figure 11.1 is an advertisement for the ISIS video Just Terror, which first appeared in the
ISIS online magazine Dabiq, Issue 13, page 55, on 19 January 2016. The analysis of the
Just Terror movie poster in Figure 11.1 was first undertaken with the interactive software
Multimodal Analysis Image.1 The software application has facilities for annotating images
on the basis of purpose-built catalogues of semiotic resources and systems, using overlays
in the form of geometrical shapes, arrows and pins, which correspond to the various
semiotic choices displayed in Table 11.1. The analysis of selected semiotic choices in the
Just Terror movie poster is discussed below.
In terms of its composition and arrangement in space, the movie poster consists of
three distinct sections or panels (marked A, B and C in Figure 11.1), following the genre
convention of regular movie posters. For example, in movie posters elements placed in

FIGURE 11.1 Multimodal analysis of Just Terror movie poster, Dabiq, 13: 55.
DISCOURSE ANALYSIS AND SEMIOTICS 237

the top half of the image are sometimes used to express the ‘promise of the product’ by
depicting a scene from the movie (Tan, E and O’Halloran 2012). Elements placed at the
centre of the image, in turn, are commonly used to attract the reader’s attention, while
elements placed at the bottom half may provide additional information about the movie.
Kress and van Leeuwen (2006) interpreted such compositional arrangements in terms of
representational (i.e. experiential) and interactive (i.e. interpersonal) meanings created
through the three interrelated systems of:
1. information value, realized through the placement of elements (participants and
syntagms that relate them to each other and to the viewer);
2. salience, realized through elements (participants as well as representational and
interactive syntagms) will attract the viewer’s attention to different degrees; and
3. framing, realized through the presence or absence of framing devices which connect
or disconnect elements of the image, signifying that they belong or do not belong
together in some sense (Kress and van Leeuwen 2006: 177).
Read from top to bottom, the poster analysed in Figure 11.1 displays the title or name
of the advertised movie (Just Terror) in large capital letters at the top of the page,
superimposed over a stylized monochrome representation of the Paris landscape, then
features the names of the principal actors underneath their images, and ends with
informing the reader of the movie’s overall motto or leitmotif: ‘Let Paris be a lesson for
those nations that wish to take heed . . . ’. Linguistically, Just Terror can be interpreted in
two ways, as displayed in Table 11.2. That is, it can be interpreted as ‘justifable’ terror,
based on an interpretation of the experiential modifier ‘just’ as what is morally right or
fair. Alternatively, the movie title Just Terror can be read as ‘simply’ terror, based on an
interpersonal interpretation of ‘just’ as related to intensity, arising (in part) intertextually
from Nike’s trademark, ‘JUST DO IT’. In this way, the movie title is designed to attract the
attention of readers in the West. Indeed, Dabiq and other propaganda materials produced
by ISIS exploit Western media traditions to attract a Western audience (e.g. Speckhard
2015). The movie title Just Terror is followed by the marked linguistic command to
pay attention and learn from these latest terrorist attacks in Paris which took place on
13 November 2015, killing 130 people and injuring 416 others.2 As we shall see, the
linguistic text functions intersemiotically with the images to realize the main narrative
themes of the movie poster.
According to Kress and van Leeuwen (2020), the information value in visual
compositions which are arranged along the vertical axis can be interpreted as representing

TABLE 11.2 Interpretations of Just Terror

Experiential interpretation:
Just Terror
based on what is morally right and fair

Interpersonal interpretation:
Just Terror
Intensity
238 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

the ‘Ideal and the Real’, whereby the ‘ideal’ is presented as ‘the idealised or generalised
essence of the information’ (2020: 191). The ‘real’ in contrast then presents ‘more
“down-to-earth” information (e.g., photographs as documentary evidence, [. . .]), or more
practical information (e.g., practical consequences, directions for action)’ (2020: 191).
In terms of the experiential meaning potential conveyed at the level of Work, the scene
depicted in the top panel of the movie poster is a stylized monochrome representation of
the Paris streetscape, and can thus be interpreted as expressing the ‘promise of the product’
(in this case, alluding to the Paris attacks on 13 November 2015), while the bottom half
of the poster displays an actual, ‘down to earth’ photograph of a realistic scene from
aftermath of the Paris attacks. Together, they present the poster’s overall narrative theme.
The title and motto, placed in bright big uppercase letters, which contrast and stand out
against the muted background, suggest importance, even grandeur. Experientially, they
represent ISIS’s worldview and ‘jurisprudential rationale’ (Ingram 2018: 22) for carrying
out terrorist attacks against nations ISIS perceive to be their enemies (Wignell, Tan and
O’Halloran 2017; Wignell, O’Halloran, Tan, Lange and Chai 2018).
The centre panel of the movie poster, which in Kress and van Leeuwen’s (2006, 2020)
terms can be interpreted as functioning as a mediator between the Ideal and the Real,
showcases the main participants in the advertised video and – in a wider sense, in the
military theatre of jihad, a term used by ISIS to refer to the ‘just’ warfare between the
Muslim and non-Muslim world (Wignell, Tan and O’Halloran 2017), drawing attention to
participant roles and processes. In terms of visual prominence or salience, certain elements
in images can attract more attention than others due to their size, or because they are
placed in the foreground. A common persuasive technique of movie posters is to highlight
the qualities or characteristics of the principal actors by means of visual prominence. In
this case, it is not surprising to find that the participant depicted at the top centre of the
whole Episode (marked 1 in Figure 11.1), who in reality was the mastermind of the Paris
attacks, is shown to tower over the other participants. Visually, he is an imposing figure,
due to his relative size and the AK47 rifle he holds in his hand. His posture is carefully
staged so that the way he extends his rifle upwards creates a compositional vector that
is intensified by the bright blue spotlight placed upon it. Together, these elements draw
the eye upward to the headline or movie title Just Terror, highlighting one of the central
themes in ISIS’s propaganda campaign (Ingram 2018), as elaborated below.
Vectors not only realize textual or compositional meaning, but they also contribute to
the text’s experiential meaning potential. According to Kress and van Leeuwen (2020:
55), the hallmark of a visual narrative proposition is the presence of a vector, which
emanates from a participant and connects to another participant or element in the
image. In the movie poster, these action vectors are formed mostly by the participants’
hands holding up rifles and knives or performing the ‘Tawheed’ gesture (right forefinger
extended skywards). In MDA terms, these participants are portrayed as being engaged in
material processes of action. Other participants in the image, ostensibly supporting actors
in the advertised video, are arranged in space beneath the main actor, disproportionally
smaller in size. The majority of these participants are portrayed as being engaged in
processes of reaction, that is, they are just looking, but they are not looking at each other,
nor are they looking at the viewer. In terms of the interpersonal relations created, apart
from the main figure (the mastermind of the Paris attack) who stares menacingly at the
viewer, other figures arranged in the foreground at the bottom tier of this Episode, and
who are shown holding up knives or performing the ‘Tawheed’ gesture, also interact
directly with the viewer via direct visual address, as they are shown gazing directly at the
DISCOURSE ANALYSIS AND SEMIOTICS 239

viewer. The lead actor and main perpetrator of the Paris attacks also dominates over the
others (including the viewer) in view of the power relations that are established through
camera angle. For instance, he is the only figure that is captured from a low angle, so that
the viewer is placed in the position of looking up at him. As such, the central figure is
visually prominent and dominating.
However, from a social semiotic viewpoint, the portrayed participants, their actions
and the objects they are interacting with fulfil yet another purpose. According to van
Leeuwen (2001), the key to interpreting visual meaning is to develop an understanding of
the different layers of meaning. Drawing on Barthes’s ([1957] 1972, 1977) adaptations
of Hjelmslev’s notions of denotation and connotation, van Leeuwen (2005) perceives
of denotation representing the first layer of meaning, expressing the literal message of
‘what, or who, is being depicted here’ (van Leeuwen 2005: 37). Borrowing from SFT,
Kress and van Leeuwen (2006, 2020) elaborate that this type of meaning is commonly
realized through conceptual representations which convey information about the
represented participants’ qualities and characteristics through processes of categorization
or typification (e.g. clothing). Connotation, in contrast, is a more abstract concept, which
can convey much broader socio-cultural ideas and values. These connotative values and
ideas can be realized through symbolic processes which have to do with ‘what a participant
means or is’ (Kress and van Leeuwen 2020: 102), or through processes of iconographic
symbolism, which convey ideas and values attached to particular ‘object-signs’ or shapes
endowed with symbolic value (van Leeuwen 2001: 107).
A related concept that has been adapted from SFT and incorporated in the MDA
approach is the notion of ‘iconization’. Different from the term used in Peircian semiotics,
where an icon does not refer to or ‘stand for’ anything other than itself’ (Nöth 2007: 104),
from an MDA perspective, the concept of iconization is viewed as a process whereby the
ideational meaning of an object or an activity is backgrounded and the interpersonal
connotations or shared values around which fellowships are formed are foregrounded
instead (Martin 2017: 129). In this way, multimodal texts and artefacts turn into social
emblems of belonging or ‘bonding icons’ (Stenglin 2008). Building on this notion, Wignell,
Tan and O’Halloran (2016, 2017; see also Wignell, Tan, O’Halloran, Lange, Chai and
Wiebrands 2020) propose a set of options for identifying different types of bonding icons
in the context of violent extremist terrorist communication (see Table 11.3).

TABLE 11.3 Bonding icons in violent extremist terrorist communication (adapted

from Wignell et al., 2016, 2017, 2020)
Types of bonding icons Description
ICON
Hero People (or personifications), e.g., revered members of a
community; role models
Relic Physical objects and images of physical objects, e.g. flags;
emblems; logos
CREED
Scripture Written or spoken representations of scripture, e.g. the Qur’an
Enacted creed Participation in rituals, e.g. religious services or communal prayer
240 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Interpreted in this way, the represented participants in Figure 11.1, i.e. the so-called
‘Soldiers of the Caliphate’, are the ‘heroes’, role models worthy of emulation and designed
to inspire likeminded followers. In the same manner, the ideational (or denotative)
meaning of the objects these ‘heroes’ are shown to be interacting with (i.e. knives) and
their actions (i.e. performing the Tawheed gesture) is eclipsed by the interpersonal
connotations these objects and actions have acquired in the context of violent extremist
terrorist communication. Consequently, they become bonding icons for followers with
similar shared values and dispositions (Wignell et al. 2016, 2017, 2020).
Bonding icons, however, do not acquire their connotative layers of meanings
spontaneously or in isolation, but through processes of resemiotization and intertextual
references to other multimodal text types and genres. As Ingram (2018: 37) comments,
the emergence of ISIS’s Just Terror campaign was the ‘product of several interconnected
dynamics’. For example, the various themes and motifs that are present in the movie
poster in Figure 11.1 were announced on the cover of Dabiq, Issue 12 (Figure 11.2,
left), which features the slogan Just Terror in big bold white and red capital letters
and depicts a realistic photograph of a scene from the 2015 Paris attacks. Likewise,
the ideological value of ISIS relics such as knives which are heralded as the weapon of
choice for lone-wolve terrorists in relation to ISIS’s Just Terror tactics campaign was
reinforced in the online magazine Rumiyah which superseded Dabiq. For example,
as shown in Figure 11.2, the theme of knives as the weapon of choice featured
prominently on the cover as well as in a procedural text on page 12 of Rumiyah, Issue
2 (Figure 11.2, centre) and was reiterated in a full-page infographic in Rumiyah, Issue
4 (Figure 11.2, right).
As Wignell et al. (2016, 2017, 2020) point out, the connotative meanings of bonding
icons arise out of a community’s view or vision of the world, by highlighting aspects
of this vision of an ideal world which a community treasures, which in this case had
‘an almost guaranteed “force multiplying” impact’ on ISIS’s propaganda aims (Ingram
2018: 37). When these and similar messages are redistributed in Western online media,
the values encoded in these texts and images are then further reinforced or negated
through resemiotization and recontextualization processes (Wignell, O’Halloran, Tan,

FIGURE 11.2 Resemiotizations and intertextuality of Just Terror tactics and knife attacks in
Dabiq and Rumiyah magazines.
DISCOURSE ANALYSIS AND SEMIOTICS 241

Lange and Chai 2018), which according to Iedema (2003) ‘provide the analytical
means for:
1. tracing how semiotics are translated from one into the other as social processes
unfold, as well as for
2. asking why these semiotics (rather than others) are mobilised to do certain things at
certain times’ (Iedema 2003: 29).
In order to map such patterns, it is necessary to develop new techniques for large-scale
analyses of multimodal texts, as discussed below.

DIGITAL APPROACHES TO MULTIMODAL

DISCOURSE ANALYSIS
Research in multimodal analysis has traditionally involved the development of theoretical
frameworks which are demonstrated with examples, as illustrated above. However,
this approach has limitations arising from the fact that the insights are derived and
generalized from a few sample texts, and the theoretical frameworks are not rigorously
tested under different conditions as a result. For this reason, interdisciplinary mixed
methods involving computational approaches for large-scale multimodal analysis are
being explored to overcome these issues (e.g. O’Halloran et al. 2018; O’Halloran, Pal
and Jin 2021). This is a challenging undertaking, however, given the large amounts
of training data which are required to develop machine learning and data mining
approaches based on multimodal semiotic theory. Moreover, existing computational
tools are largely monomodal, having been developed for language analysis, image
analysis and video analysis alone. With this in mind, approaches that integrate existing
computational tools with multimodal frameworks for large-scale semiotic analysis are
being investigated to test the effectiveness of this approach and to identify the gaps
in existing computational approaches. Initial explorations in this direction, involving
the use of natural language understanding algorithms for MDA, appear promising
(Wignell, Chai, Tan, O’Halloran and Lange, 2018). In what follows, a mixed methods
approach for large-scale multimodal analysis is illustrated using online terrorist
materials discussed above.
Many images which appear in violent extremist propaganda are re-used and
recontextualized across different websites (e.g. mainstream news media, magazines, blogs,
photo sharing sites, etc.) and social media. In order to investigate the recontextualization
patterns, reverse image searches of images from Dabiq and Rumiyah were undertaken
using TinEye3 which crawls the web to identify the websites where these images appear.
A sample set of 20 images from Dabiq and Rumiyah and 254 English language webpages
with these images were identified to explore how the multimodal analysis framework
(Table 11.1) can be integrated with computational tools (i.e. natural language processing
and image processing) in order to analyse the text and images in the articles from these
websites.
The sample data set of images from Dabiq includes the Just Terror movie poster
(Figure 11.1) discussed above. An example of an article4 with the image from Dabiq
is displayed in Figure 11.3a. The article appears in FDD’s Long War Journal (LWJ),5
an American news blog which reports on the US war on terror. The linguistic text was
242 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

analysed using natural language understanding algorithms provided by IBM Watson6

and the image was analysed using image recognition models provided by Clarifai.7 The
linguistic analysis involves IBM Watson’s algorithms for key concepts, entities, keywords,
categories, relations and semantic roles and sentiment and emotion and the visual analyses
involve Clarifai’s algorithms for concepts, focus and demographics in the image. The
linguistic and visual features derived from these computational algorithms are organized
according to the metafunctional systems and ranks from the multimodal theoretical
framework (Table 11.1), as displayed below. The results for the linguistic and visual
analyses derived from IBM Watson and Clarifai models for the article in Figure 11.3a are
displayed in Figures 11.3b and 11.3c respectively.

Text Analysis: IBM Watson

1. Discourse/Register – Experiential: Watson’s categorical values (twelve categories/

features)
2. Discourse Semantics – Interpersonal: Watson sentiment and emotion analysis for
each paragraph (Sentiment: three features, Emotion: five features)
3. Lexicogrammar – Logical: conjunctive adverb analysis (two features)

Image Analysis: Clarifai

1. Work – Ideational/field: Clarifai Concepts (195 features)

2. Episode – Interpersonal/tenor: Clarifai Focus (two features)
3. Figure – Ideational/field: Clarifai Demographics (seven features)
The results of the language and image analyses were clustered using K-modes
algorithm (Huang 1997, 1998), which is a form of unsupervised learning for data which
does not have predefined categories. The clusters (or sub-groupings) are derived based
on the features provided, which in this case are categorical features derived from the
linguistic and visual analysis and the multimodal framework (i.e. metafunctions and
rank). The K-modes algorithm defines the clusters based on the number of matching
categories between data points – in this case, the vectors arising from the text analysis,
image analysis and the metafunctional framework. In other words, the algorithm
produces clusters depending on the dis/similarities between the features of the articles,
which in this case is derived from the linguistic and visual analyses. As the process
involves unsupervised learning with no prior knowledge of the texts, the Elbow Method
(He, Deng and Xu 2006) is used to calculate the optimal number of clusters, beyond
which no significant gain (in terms of average dispersion) is obtained by increasing k, the
number of clusters.
The results of the cluster analysis (for k = 9) are displayed using an interactive web-
based visualization application in Figure 11.4. Each cluster (represented as a spiral)
has a centroid with a mean value for that cluster, and the clusters are positioned
according to their relations to the cluster at the centre of the display (in this case
Cluster 8). The articles in the cluster (i.e. represented as points on the spiral) are
positioned according to their relations to the centroid. Individual results for each
article are displayed as separate webpages which can be accessed by clicking on the
point. The webpages contain the article image and text (Figure 11.3a), the URL of the
DISCOURSE ANALYSIS AND SEMIOTICS 243

FIGURE 11.3a Just Terror article: Image and text.

source article and a set of tables with the results of the language and image analyses
(Figures 11.3b and 11.3c).
The image from the Just Terror movie poster (Figure 11.1) appears in Cluster
6, positioned on the right of Figure 11.4. This is the largest and most diverse cluster
244 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 11.3b Language analysis for Just Terror article.

containing 61 webpages with 11 different images. To interpret and evaluate the cluster
analysis results, a Sankey diagram consisting of the:
1. type of visual bonding icon;
2. Watson category; and
3. website classifications derived from UClassify8 Taxonomy classifier and Tonality
classifier (corporate or personal) was constructed, displayed in Figure 11.5.
DISCOURSE ANALYSIS AND SEMIOTICS 245

FIGURE 11.3c Image analysis for Just Terror article.

FIGURE 11.4 K-Modes cluster visualization.

As can be seen from the Sankey diagram, the images in Cluster 6 are predominantly
ICONS (i.e., ISIS HEROES and ISIS RELICS), together with ISIS advertisements,
and images of enemy soldiers and actions by ISIS against local enemies. The Watson
categories are predominantly Society, with others being Law, Government and Politics
and, to a lesser extent, Religion and Spirituality. These categories form part of the
246 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 11.5 Clusters with image categories, Watson categories and website taxonomy and
tonality.

taxonomy of News and Politics, with a corporate tone, following uClassify classifications
of the websites. Closer inspection of the articles and websites reveals that the majority
of the articles with the images appear on Western news media sites, as suggested by
Figure 11.5.
The results are promising, showing how multimodal texts can be clustered and
analysed in order to display similar features in terms of linguistic and visual choices.
The results are quite different when the linguistic and visual analyses are conducted
separately. That is, the clusters for the text analysis (k = 6) and image analysis (k =
2, 3 and 4) are tightly grouped, showing similarities between the language and images
in the articles. This suggests that a multimodal analysis approach is required in order
to explore the features of multimodal texts. Most importantly, however, when the
computational tools are mapped to the multimodal framework (see Table 11.4), it
becomes evident how few multimodal semiotic systems are actually accounted for in
the computational algorithms provided by major technology companies. In particular,
gaps exist at the level of the clause for language and interpersonal meaning for
images, and the textual metafunction for text and images. These findings suggest that
multimodal frameworks have the potential to improve computational approaches to
big data analytics by:
1. providing an overall coherent approach to multimodal analysis and
2. identifying areas in which computational tools need to be developed.
DISCOURSE ANALYSIS AND SEMIOTICS 247

TABLE 11.4 MDA framework and IBM Watson and Clarifai algorithms
Metafunction Rank System Description
Text
Experiential Discourse Configurations of activities Content in relation to
knowledge construction
IBM Watson – categorical
values
Clause Processes; Participant Happenings, actions and
Roles; Circumstance relations
Interpersonal Discourse Appraisal Evaluation in terms of attitude,
emotion and judgement
IBM Watson – sentiment and
emotion analysis for each
paragraph
Clause Speech function Exchange of information (e.g.
statements and questions)
and goods & services (e.g.
commands and offers)
Textual Clause & Information Focus Organization of information,
Discourse with points of departure for
what follows
Logical Clause Logico-Semantic Relations Logical relations between
Complex happenings, actions and
relations
Conjunctive adverb analysis
Images
Experiential Work Narrative Theme; Nature of the scene
Representation; Setting Clarifai – concepts
Episode Processes; Participant Visual happenings, actions and
Roles; and Circumstance relations
Figure Posture; Dress Characteristics of the
participants
Clarifai – demographics
Interpersonal Work Angle; Camera Distance; Visual effects
Lighting
Episode Proportion in Relation to Happenings, actions and
the Whole Image: Focus; relations with respect to the
Perspective whole image
Clarifai – focus
Figure Gaze-Visual Address Direction of participant’s gaze
as internal to image or external
to viewer
248 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Metafunction Rank System Description

Text
Textual Work Compositional Vectors; The organization of the parts
Framing as a whole, with the visual
marking (e.g. framing) of
certain parts
Episode Relative Placement of Position of the happenings,
Episode; Framing actions and relations in
relation to the whole image,
and the visual marking of
certain aspects
Figure Relative Placement of the Position of figures in relation
Figure within the Episode; to happenings, actions or
Arrangement; Framing relations, and the visual
marking of certain aspects of
those figures
Figure Gaze-Visual Address Direction of participant’s gaze
as internal to image or external
to viewer

FUTURE DIRECTIONS: MULTIMODAL

SEMIOTIC SCIENCES
Multimodal discourse analysis holds much promise for analysing and interpreting
communication in the digital era. In particular, the social semiotic approach offers a way of
modelling semiotic resources as interacting systems of meaning and analysing multimodal
texts as clusters of multimodal choices, following Halliday’s (2008) foundational principle
of ‘language as system’ and ‘language as text’. As Lemke (2000) explains, semiosis involves
different scales from the materiality of signs to acts of interpretation which are enabled
and at the same time constrained by semiotic resources and socio-cultural practices (e.g.
see van Leeuwen 2008). If semiotic behaviour is to be captured, mapped and interpreted
across space and time, then multimodal contextual-based approaches which are capable
of managing the complexity of large scale semiotic analysis are required. Inevitably, this
will involve the development of interactive digital approaches to MDA.
For example, it is possible to set up cloud-based platforms with facilities for collecting,
indexing, storing and analysing online communications using multimodal frameworks
and computational models for big data, cloud computing, natural language processing,
image processing, video processing and contextual metadata, together with interactive
visualizations and dashboard for displaying the results. The mixed methods big data
approach appears promising, based on preliminary studies of online news and social media
which reveal differences about how official news reports differ from public reactions on
social media. For example, George Floyd’s death on 25 May 2020 gave rise to a higher
collective level of negative reactions in online news media and social media, compared to
the polarizing and highly differentiated response to the Covid-19 pandemic during the
mid-2020s (O’Halloran, Pal and Jin 2021). Furthermore, it was possible to demonstrate
DISCOURSE ANALYSIS AND SEMIOTICS 249

that concerns about the coronavirus pandemic at this time are related predominantly to
politics in the online newspapers, compared to health and well-being in social media.
Semiotics as the foundational science of signs holds much promise for understanding
society and culture in the age of digital media which has revolutionized the world socially,
culturally, politically and economically. In particular, mixed methods approaches to
multimodal discourse analysis provide the means for mapping and interpreting semiosis in
ways which have not been possible before, given advances in technology and the contextual
information which is now available. This includes metadata about communications
(e.g. time, place, source, author, etc.) and semantic classifications which span the entire
breadth of human activity (e.g. Wikipedia). From this perspective, the development of
the multimodal semiotic sciences must necessarily be a collective multi-disciplinary effort
involving discourse analysts, cultural semioticians, computer scientists and scientists. The
need to bring together social semiotic theory and technology to develop the multimodal
sciences for mapping and understanding human activity is evident. Furthermore, it is
clearly an urgent research goal if we are to address major issues in the world today.

NOTES
1 http://multimodal-analysis.com/products/multimodal-analysis-image/software/index.html
2 https://en.wikipedia.org/wiki/November_2015_Paris_attacks
3 https://tineye.com/
4 https://www.longwarjournal.org/archives/2016/01/before-paris-attack-islamic-state-
terrorists-committed-grisly-executions.php
5 https://www.longwarjournal.org/
6 https://www.ibm.com/uk-en/cloud/watson-natural-language-understanding
7 https://www.clarifai.com/
8 https://www.uclassify.com/

REFERENCES
Barthes, R. ([1957] 1972), Mythologies, trans. A. Lavers, London: J. Cape.
Barthes, R. (1977), Image-Music-Text, London: Fontana.
Bateman, J., J. Wildfeuer and T. Hiippala (2017), Multimodality Foundations, Research and
Analysis – A Problem-oriented Introduction, Berlin: de Gruyter Mouton.
Bednarek, M. and H. Caple (2012), News Discourse, London: Continuum.
Djonov, E. (2007), ‘Website Hierarchy and the Interaction between Content Organization,
Webpage and Navigation Design: A Systemic Functional Hypermedia Discourse Analysis
Perspective’, Information Design Journal, 15 (2): 144–62.
Djonov, E. and S. Zhao (2014), Critical Multimodal Studies of Popular Discourse, London and
New York: Routledge.
Forceville, C. (1996), Pictorial Metaphor in Advertising, London: Routledge.
Forceville, C. and E. Urios-Aparisi (2009), Multimodal Metaphor, vol. 11, Berlin: Walter de
Gruyter.
­Halliday, M. A. K. (1978), Language as Social Semiotic, London: Edward Arnold.
Halliday, M. A. K. (2008), Complementarities in Language, Beijing: Commercial Press.
Halliday, M. A. K. (2009), Collected Works of M. A. K. Halliday (10 Volume Set), London and
New York: Continuum.
250 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Halliday, M. A. K. and R. Hasan (1985), Language, Context and Text: Aspects of Language in a
Social-Semiotic Perspective, Waurn Ponds, VIC: Deakin University.
Halliday, M. A. K. and C. M. I. M. Matthiessen (2014), Halliday’s Introduction to Functional
Grammar, London: Routledge.
He, Z., S. Deng and X. Xu (2006), ‘Approximation Algorithms for K-Modes Clustering’, in
D. S. Huang, K. Li and G. W. Irwin (eds), Computational Intelligence, ICIC 2006, Lecture
Notes in Computer Science, 4114, 296–302, Berlin: Springer-Verlag.
Huang, Z. (1997), ‘Clustering Large Data sets with Mixed Numeric and Categorical Values’, in
H. Motoda and H. Lu (eds), Proceedings of the First Pacific Asia Knowledge Discovery and
Data Mining Conference, Singapore, 21–34, Singapore: World Scientific.
Huang, Z. (1998), ‘Extensions to the K-Modes Algorithm for Clustering Large Data Sets with
Categorical Values’, Data Mining and Knowledge Discovery, 2 (3): 283–304.
Iedema, R. (2003), ‘Multimodality, Resemiotization: Extending the Analysis of Discourse as a
Multisemiotic Practice’, Visual Communication, 2: 29–57.
Ingram, H. J. (2018), ‘Islamic State’s English-Language Magazines, 2014–2017: Trends &
Implications for CT-CVE Strategic Communications’, International Centre for Counter
Terrorism, 1–46.
Jewitt, C., J. Bezemer and K. L. O’Halloran (2016), Introducing Multimodality, London:
Routledge.
Knox, J. (2007), ‘Visual-Verbal Communication on Newspaper Home Pages’, Visual
Communication, 6 (1): 19–55.
Kress, G. (2011), ‘Multimodal Discourse Analysis’, in J. P. Gee and M. Handford (eds), The
Routledge Handbook of Discourse Analysis, Abingdon: Routledge.
Kress, G. and T. van Leeuwen (2001), Multimodal Discourse: The Modes and Media of
Contemporary Communication, London: Arnold.
Kress, G. and T. van Leeuwen (2006), Reading Images: The Grammar of Visual Design, 2nd
edn, London: Routledge.
Kress, G. and T. van Leeuwen (2020), Reading Images: The Grammar of Visual Design, 3rd
edn, London: Routledge. https://doi.org/10.4324/9781003099857.
Lemke, J. L. (2000), ‘Opening up Closure: Semiotics Across Scales’, in J. Chandler and G. van
de Vijver (eds), Closure: Emergent Organizations and their Dynamics, vol. 901, Annals of
the NYAS, 100–11, New York: New York Academy of Science Press.
Lotman, Y. ([1984] 2005), ‘On the Semiosphere’, Sign System Studies, 33 (1): 201–29.
Lotman, Y. and B. Uspensky (1978), ‘On the Semiotic Mechanism of Culture’, New Literary
History, 9 (2): 211–32.
Machin, D. (2013), ‘What Is Multimodal Critical Discourse Studies?’ Critical Discourse Studies,
10 (4): 347–55.
Machin, D. (2016), ‘The Need for a Social and Affordance-driven Multimodal Critical
Discourse Studies’, Discourse & Society, 27 (3)A: 322–34.
Machin, D. and A. Mayr (2012), How to Do Critical Discourse Analysis: A Multimodal
Introduction, London: Sage.
Martin, J. R. (1992), English Text: System and Structure, Amsterdam: John Benjamins.
Martin, J. R. (2017), ‘Revisiting Field: Specialized Knowledge in Secondary School Science and
Humanities Discourse’.
Martin, J. R. and D. Rose (2007), Working with Discourse: Meaning beyond the Clause, 2nd
edn, London: Continuum.
­Martin, J. R. and P. R. R. White (2005), The Language of Evaluation: Appraisal in English,
London: Palgrave Macmillan.
DISCOURSE ANALYSIS AND SEMIOTICS 251

Norris, S. (2004), Analyzing Multimodal Interaction: A Methodological Framework, London:

Routledge.
Norris, S. and R. H. Jones (2005), Discourse in Action: Introducing Mediated Discourse
Analysis, London: Routledge.
Nöth, W. (2007), ‘The Death of Photography in Self-reference’, in W. Nöth and N. Bishara
(eds), Self-reference in the Media, 95–106, Berlin: Walter de Gruyter.
O’Halloran, K. L. (2008), ‘Systemic Functional-Multimodal Discourse Analysis SF-
MDA: Constructing Ideational Meaning Using Language and Visual Imagery’, Visual
Communication, 7: 443–75.
O’Halloran, K. L. (2015), ‘The Language of Learning Mathematics: A Multimodal Perspective’,
The Journal of Mathematical Behaviour, 40 (Part A): 63–74.
O’Halloran, K. L. (2021), ‘Multimodal Discourse Analysis’, in K. Hyland, B. Paltridge and
L. Wong (eds), Bloomsbury Companion to Discourse Analysis, 2nd edn, 249–266, London:
Bloomsbury.
O’Halloran, K. L., S. Tan, P. Wignell, J. Bateman, D. S. Pham, M. Grossman and A. Vande
Moere (2016), ‘Interpreting Text and Image Relations in Violent Extremist Discourse: A
Mixed Methods Approach for Big Data Analytics’, Terrorism and Political Violence, 31 (3):
454–74.
O’Halloran, K. L., S. Tan, D. S. Pham, J. Bateman and A. Vande Moere (2018), ‘A Digital
Mixed Methods Research Design: Integrating Multimodal Analysis with Data Mining and
Information Visualization for Big Data Analytics’, Journal of Mixed Methods Research, 12
(1): 11–30.
O’Halloran, K. L., S. Tan and P. Wignell (2019), ‘SFL and Multimodal Discourse Analysis’, in
G. Thompson, W. L. Bowcher, L. Fontaine and D. Schönthal (eds), The Cambridge Handbook
of Systemic Functional Linguistics, 433–61, Cambridge: Cambridge University Press.
O’Halloran, K. L., S. Tan, P. Wignell, R. Wang, K. Chai and R. Lange (2020), ‘Towards a
Discipline of Multimodality: Parallels to Mathematics and Linguistics and New Ways
Forward’, in J. Wildfeuer, J. Pflaeging, C. I. Tseng, O. Seizov and J. Bateman (eds),
Multimodality: Towards a New Discipline, 35–54, Berlin: de Gruyter.
O’Halloran, K. L., G. Pal and M. Jin (2021), ‘Multimodal Approach to Analysing Big Social
and News Media Data’, Discourse, Context and Media, 40. https://doi.org/10.1016/j.
dcm.2021.100467
O’Toole, M. (2011), The Language of Displayed Art, 2nd edn, London and New York:
Routledge.
Scollon, R. (2001), Mediated Discourse: The Nexus of Practice, London: Routledge.
Scollon, R. and S. W. Scollon (2004), Nexus Analysis, London: Routledge.
Speckhard, A. (2015), ‘The Hypnotic Power of ISIS Imagery in recruiting Western Youth’, in
ISCVE Report, International Center for the Study of Violent Extremism.
Stenglin, M. (2008), ‘Interpersonal Meaning In 3D Space: How a Bonding Icon Gets Its
“Charge”’, in L. Unsworth (ed.), Multimodal Semiotics: Functional Analysis in Contexts of
Education, 50–66, New York: Bloomsbury Publishing.
Stöckl, H., H. Caple and J. Pflaeging (2020), Shifts towards Image-centric Practices in the
Contemporary Multimodal Practices, London: Routledge.
Tan, S. and K. L. O’Halloran (2012), Multimodal Analysis Image (Teacher edn and Student
edn), Singapore: Multimodal Analysis Company.
­Tan, S., K. L. O’Halloran and P. Wignell (2020), ‘Multimodality’, in A. De Fina and A.
Georgakopoulou (eds), The Cambridge Handbook of Discourse Studies, 263–81, Cambridge:
Cambridge University Press.
252 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Tannen, D., H. E. Hamilton and D. Schiffrin (2015), The Handbook of Discourse Analysis, 2nd
edn, New Jersey: John Wiley & Sons.
van Leeuwen, T. (1999), Speech, Music, Sound, London: Macmillan.
van Leeuwen, T. (2001), ‘Semiotics and Iconography’, in T. van Leeuwen and C. Jewitt (eds),
Handbook of Visual Analysis, 92–118, London: Sage.
van Leeuwen, T. (2005), Introducing Social Semiotics, London: Routledge.
van Leeuwen, T. (2008), Discourse and Practice: New Tools for Critical Discourse Analysis,
Oxford: Oxford University Press.
van Leeuwen, T. (2012), ‘Critical Analysis of Multimodal Discourse’, in C. A. Chapelle (ed.),
The Encyclopedia of Applied Linguistics, 4002–6, London: Blackwell.
van Leeuwen, T. (2015), ‘Multimodality’, in D. Tannen, in H. E. Hamilton and D. Schiffrin
(eds), The Handbook of Discourse Analysis, 2nd edn, 447–65, New Jersey: John Wiley &
Sons.
Wignell, P., S. Tan and K. L. O’Halloran (2016), ‘Violent Extremism and Iconisation:
Commanding Good and Forbidding Evil?’ Critical Discourse Studies, 14 (1): 1–22.
Wignell, P., S. Tan and K. L. O’Halloran (2017), ‘Under the Shade of AK47s: A Multimodal
Approach to Violent Extremist Recruitment Strategies for Foreign Fighters’, Critical Studies
on Terrorism, 10 (3): 429–52.
Wignell, P., K. L. O’Halloran, S. Tan, R. Lange and K. Chai (2018), ‘Image and Text Relations
in ISIS Materials and the New Relations Established through Recontextualisation in Online
Media’, Discourse & Communication, 12 (5): 535–59.
Wignell, P., S. Tan, K. L. O’Halloran, R. Lange, K. Chai and M. Wiebrands (2020), ‘Images
as Ideology in Terrorist-related Communications’, in H. Stöckl, H. Caple and J. Pflaeging
(eds), Shifts towards Image-centricity in Contemporary Multimodal Practices, 253–74, New
York: Routledge.
Wignell, P., K. Chai, S. Tan, K. L. O’Halloran and R. Lange (2021), ‘Natural Language
Understanding and Multimodal Discourse Analysis for Interpreting Extremist
Communications and the Re-use of these Materials Online’, Terrorism and Political
Violence, 33 (1): 71–95.
 CHAPTER TWELVE

Integrational Linguistics
and Semiotics
ADRIAN PABLÉ

SEMIOTIC IDEOLOGY: THE THREE

ONTOLOGIES OF THE SIGN
What discipline do you turn to when you wish to know about the nature of the sign?
Semiotics, as the general study of signs, would seem the most obvious place to look for
enlightenment. The two related questions ‘What is a sign?’ and ‘How does a sign acquire
its meaning?’ have been approached from all kinds of angles in semiotics, depending on
the various schools and theorists. As such, one would expect that these two questions
have been settled definitely or at least discussed exhaustively, with little more to say about
the ontology of signs. This was also the opinion of German linguist Rudi Keller, author
of a work on theories of the sign entitled Zeichentheorie. Zu einer Theorie semiotischen
Wissens (Keller 1995), translated into English three years later as A Theory of Linguistic
Signs (Keller 1998). Keller (1995: 9) draws the following conclusion about the history of
semiotic theory: ‘Alles was über sprachliche Zeichen gesagt werden kann, ist vermutlch
irgendwann zwischen Platon und heute gesagt worden. In einem sprachphilosophischen
Gebiet mit mehr als zweitausendjähriger Tradition lässt sich wirklich Neues wohl kaum
noch entdecken.’ [Everything that can be said about linguistic signs has probably been said
sometime between Plato and today. In a language-philosophical field that is more than
two thousand years old, it seems unlikely that anything new will ever be discovered.] The
present contribution would beg to differ on this point. In fact, Keller’s verdict corresponds
pretty much to the official position in semiotics, with linguistic signs being only one
type of sign among many others. When it comes to questions about the ontology of the
sign, one endorses either a semiological position (i.e. Saussurean and Post-Saussurean
dyadic sign conceptions) or the much older ‘semiotic’ position, which construes signs
in a broader sense (i.e. the various ‘surrogational’ traditions, most notably the Peircean
triadic sign model). It would thus seem that everything has been said about what a sign
is and how it acquires meaning. What else could it be than a substitute for – or surrogate
of – something other than itself, and its meaning being what the sign replaces (or ‘stands
for’)? This conception broadly aligns with lay thinking about signs. Even Saussure had to
concede that at the level of parole words are taken by the language users to be substitutes
for material things or for private ideas in the mind. Saussure’s definition of the linguistic
sign as having a purely psychological ontology – a ‘psychological reality’, as he insisted –
and leading a separate real existence from its material manifestation as a word was the
254 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

idealist answer to lay materialist semiotics. What in ordinary English, French, Italian,
etc., is called a ‘sign’ is not a sign in Saussurean terms, but the word sign, which identifies
material ‘signs’, is one. According to Saussure, the human capacity to mentally construct
collective systems of signs was to be at the centre of semiology as a ‘science’, while
material ‘signs’ were relegated to objects like any others.
Other forms of idealism, e.g. Benedetto Croce’s aesthetic science, equally rejected
the materiality of art (including language) and privileged the individual’s mental creative
powers. Unlike Saussure, Croce had no interest in a theory of signs that dealt with either
abstractions or physical stimuli (Dessì Schmid 2004: 699), neither of which concerned
language as ‘perpetual creation’. The Italian idealist linguistic schools of a Crocean
stamp, among them Giulio Bertoni’s idealist ‘science’, exerted very little influence on
general linguistics as a contemporary academic field, remaining limited to Romance
linguistics (in spite of their universalist outlook). One reason for this was undoubtedly
the idealist rejection of positivism as embodied in the Neogrammarian school (Bonfante
1947). General linguistics took a Saussurean turn instead and made Saussure’s idealism
compatible with an empirical-historical study of language inasmuch as synchrony and
diachrony came to be regarded as different perspectives of the same reality. Saussure’s
dictum that it is ‘the viewpoint that creates the object’ (Saussure 1983: 8) was thus
ignored in structuralist sociolinguistics and replaced by a ‘multi-fixed-code’ theory able
to account for the alleged continuity of language variation and change (Weinreich, Labov
and Herzog 1968). In other words, the ‘history of English’ (or of any other language for
that matter) was to be tackled from within parole, however without having to give up
the notion of langue. This also meant that linguists no longer had to bother about a strict
separation between the word and the linguistic sign, and as a consequence they did not
hesitate to treat languages as fixed-codes with continuously changing forms but relatively
stable meanings.
From the twentieth century onwards, the questions ‘What is a sign?’ and ‘How does
a sign acquire its meaning?’ have thus had two possible answers within mainstream
semiotics/semiology, roughly aligning with a Peircean and a Saussurean stance. However,
this view is not representative of the whole range of ontologies of the sign proffered by
theorists. This chapter will introduce a third approach to sign ontologies. Like Croce’s
view of language as pure creation, it emphasizes human creativity, however without
subscribing to any form of idealism (Pablé 2018). To the best of the author’s knowledge,
no fourth ontology has been proposed and it is not likely that one ever will be. This
third ontology has largely gone unnoticed in semiotics (with few exceptions, see Cobley
2011, 2014) and although it originated as part of semiology it has not received much
attention in linguistics, a discipline (as mentioned above) with an ever-declining interest
in theoretical questions about the linguistic sign and a predilection for empirical questions
about phonetic variations in words. This third approach has been described as too ‘radical’
by empirically minded linguists (Linell 2018) or has been criticized for making nonsense
of the abstract notion of signhood as established by Saussure (Weigand 2018). Being
founded on a semiological approach, this third ontology has much to say about human
signs but very little about non-human signs. Being semiological, moreover, it also rejects
the ‘standing for’ formula common to semiotics (Daylight 2012). The sign conception at
issue was developed by Oxford linguist Roy Harris (1931–2015) in the late 1970s, as part
of an approach to language and communication named ‘integrationism’ and ‘integrational
linguistics’, respectively. The latter approach is said to apply the insights of integrationism
to the particular case of language (Harris 1998: 4). Throughout his work, Harris referred
INTEGRATIONAL LINGUISTICS AND SEMIOTICS 255

to integrational linguistics as ‘lay-oriented’ (e.g. Harris 1981). The integrational sign

conception, however, is not a lay conception any more than Saussure’s was. Throughout
the history of Western lay linguistics, signs were construed as having a stable relation to
something (other than itself) in the real world: the dark clouds in the sky are a natural
sign of bad weather, while the words used to make that statement are conventional signs
mirroring a state of affairs in the non-linguistic world. Both Saussure and Harris regarded
the lay view of signs as inadequate for semiology, though they proposed very different
answers to the question what a sign really was. The difference between their two sign
conceptions is best explained as the difference between the abstract collective sign versus
the contextualized personal sign.
The belief that signs have a certain ontology is always embedded in a certain theory
of the sign and presupposes a certain worldview. Keane (2018) speaks of ‘semiotic
ideologies’ and uses this concept to denote people’s underlying assumptions of what signs
are and how they function. Keane (2018: 66) explains that it is a matter of semiotic
ideology ‘whether signs are taken to be interpretable because their relation to the world
is arbitrary, or logical, or natural, or divinely ordained’. He is thus interested in semiotic
ideologies insofar as they tell us what counts as a sign (and a sign of what) in a certain
community among different sign-users. For the integrationist, ‘semiotic ideology’ is
equally of interest as a term if extended to cover semioticians’ and linguists’ assumptions
about the ontology of the sign and what conditions must be given for signs to exist.
Integrationists thus take an interest in philosophy of linguistics, as Harris (1993) made
clear, but theirs is also a commitment to philosophy of semiotics. Keane talks about icons
and indexes as if the sign classification itself was not ‘ideological’, i.e. the result of what
he (2018: 67) calls ‘semiotic reflexivity’ – and what would be subsumed under the label
‘decontextualization’ in Harrisian terms. For Keane, semiotic ideology guides people ‘as
they sort out which aspects of their experience are or are not candidates for even being
signs at all’ (2018: 69). But Keane seems oblivious to the possibility that this very view
of semiotic ideology already presupposes a certain view of the sign. In Keane’s case it is a
view linked to a human world in which only certain things become sign-mediated, in turn
becoming different ‘objects’ in different cultures, societies and times. Thus Keane (2018:
69) asks the ontological question: what are possible objects for a sign? Keane discusses
semiotic ideology based on the Peircean triadic sign model, presenting it as promoting a
‘quasi-scientific’ understanding of what a sign is. The tripartition into iconic, indexical
and symbolic signs is thus treated as a universal semiotic reality. However, Keane (2018:
80) admits that analysts and laypeople do not always agree on the reality of an object or
what the object is (i.e. what the sign ‘stands for’). And therefore there is also disagreement
about the sign modality. Keane (2018: 82) concludes that in ‘interpretive communities’
sign vehicles remain the same while objects change. Keane (2018: 69) tells the reader that
his semiotic approach is done ‘in the spirit of pragmatism’: i.e. his aim is to find ‘usable
concepts for empirical purposes’. Against this background the legitimate question then
arises whether the integrational conception of the sign may itself be ‘ideological’, e.g.
motivated by the purpose of not being empirical. In other words, can there be a theory of
the sign free of any disciplinary impositions?
Like Saussure’s semiology, integrational linguistics is founded on a self-reflexive
theory of the sign, but presupposes an altogether different view concerning the centrality
of signs in human life. Just as Saussure declared that all questions about the sign were
inseparable from raising these questions by means of linguistic signs, Harris introduced the
revolutionary idea that signs were what their sign-makers ‘made’ them: no sign without a
256 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

sign-maker, including the word sign. Neither Saussure nor Harris thus believes that there
is a meaningful world detached from the collective linguistic system or the individual
sign-maker, respectively. Semiologists maintain that signs are the human reality and that
the separation between signs and objects is the result of the human imagination of a
detached but still graspable reality. Harris devoted much of his career to the critique of
an all-encompassing ‘semiotic ideology’ which he termed the ‘Language Myth’ (Harris
1981). One facet of this myth is precisely the belief that there are things in the real world
identified by their true names: e.g. the word universe ‘stands for’ the universe, whereby
the universe exists independently of humans attaching a linguistic label (‘universe’) to it.
What makes the Harrisian approach to the sign different from both the Saussurean and
the Peircean counterparts is the insight that nobody (including the analyst or theorist)
can tell what a sign means for somebody else, let alone for a whole community (e.g. the
word universe). The objection that certain communities agree on what a sign means, and
therefore that it is possible to tell what the sign means for at least some people (e.g. the
word universe for the scientific community) is not an argument that an integrationist
would endorse. On the contrary, the integrationist discards abstract and reductionist
terms like Keane’s ‘interpretive community’ because they do not explain individuals’
meaning-making in given circumstances. Ultimately, the integrationist argues, resorting
to entirely macrosocial explanations of sign-making (sign-makers A, B, C, D belong to the
same ‘interpretive community’) does not take into consideration the individual’s personal
communicational biography, replacing it with an abstract communal biography on which
individual communication allegedly rests. In the latter framework individual variations
are still entirely possible but are based on a shared underlying core. In other words,
there is a communal psychological ‘fixed-code’ already in place. The question that Harris
(1996: 7) invites his readers to ponder is precisely whether ‘communication presupposes
signs’ or whether ‘signs presuppose communication’.

SIGNS AND ACTIVITIES

Harrisian integrationism, discussed most comprehensively in the books Signs, Language
and Communication (Harris 1996) and Introduction to Integrational Linguistics (Harris
1998), takes an altogether humanist and anthropocentric view on signs and sign-making.
Unlike more traditional forms of humanism, it does arguably not operate with ethnocentric
linguistic categories, even though Harris’s intellectual formation occurred in a Eurocentric
context. At the same time integrationism claims to be ‘lay-oriented’ in a way that renders
its understanding of human communication immune to the charge of ethnocentrism (Pablé
2019). In fact, Harris rejected the traditional humanist – or what Sebeok (1975) called the
‘glottocentric’ – views about communication and the role that ‘language’ and ‘languages’
play within these paradigms. Harris (2010) argued that linguistic theory has to start with
‘understanding linguistic experience from the inside’ (and hence the ‘lay-orientation’),
instead of promoting itself as an empirical science tasked with interpreting other people’s
sign-making. In fact, mainstream linguistics, despite its claims to being ‘scientific’, could
actually be accused of harbouring a cultural bias in the way it investigates and describes
human languages: thus while defining ‘languages’ as abstract systems of signs shared by
the members of a collectivity (Saussure’s langue), it also proclaims the professional linguist
to be an expert, whose knowledge transcends the limitations of the lay linguistic user.
The view of ‘languages’ as consisting each of identifiable – though only interdependently
INTEGRATIONAL LINGUISTICS AND SEMIOTICS 257

existing – abstract signs amenable to systematic description was indeed the result of a
Western idea, originally developed in nineteenth-century Europe, and modelled on the
natural sciences, which, as Harris argued in his challenging book The Semantics of Science
(Harris 2005), equally subscribe to traditional humanist assumptions about how speaker
A can rationally and successfully communicate with hearer B under ideal circumstances
(e.g. both need to be native speakers of the same language, they need to have been brought
up in a Western cultural context, etc.). On this view, science is grounded in a Eurocentric
conception of how language relates to the world (the ‘semiotic ideology’ of science),
i.e. the belief in true names, which correctly identify, irrespective of contextual factors,
the things that they are names of (i.e. what they ‘stand for’). For the integrationist, in
turn, there are no linguistic experts because there are no scientifically describable objects
called ‘languages’ nor independently existing words of ‘a language’. Integrationists insist,
however, that a theory of communication has to be semiologically grounded (Harris
1996: 12). Unlike Saussure’s semiology, Harrisian semiology does not take the linguistic
sign qua psychological unit as its point of departure. In the integrational conception,
the sign serves the activities it integrates for someone under the present circumstances.
Anything (articulated sounds, material objects, etc.) can be a sign if someone bestows on
it a semiological function as part of a programme of activities he or she is engaged in, but
more often than not it would not be possible to identify ‘the sign’ or ‘the signs’ involved
in the communicative situations typically analysed in linguistics (and how many there are
exactly). The reason why this is not possible is the temporality that characterizes the sign:
in fact, in order to analyse communication in terms of signs you need to decontextualize
them (i.e. treat them as atemporal). The integrationist denies precisely this, namely that
there is an ‘objective’ way of representing the (linguistic) signs which, mirabile dictu,
corresponds exactly to how the communicants themselves experienced the signs when they
made them. Harris’s own illustrative examples of what a sign is focus mostly on everyday
situations in which something familiar is imagined to happen (sometimes involving Harris
himself): someone knocks on the door, a landmark acquires a signalling function on one’s
ride home, someone asks a question and someone else replies or does something non-
verbally, someone sees a cat at the end of one’s garden and tells somebody else about it.
Harris does not seem interested in engaging in ‘conversation analysis’, which calls for a
third-person perspective: the complexity of such exchanges involving an open range of
cotemporal activities can only be analysed at the cost of ‘segregating’ the linguistic from
the non-linguistic activities, and artificially putting them together again a posteriori (as,
e.g., in multimodal approaches to language). Other integrationists have provided more
detailed introspective analyses of sign-making activities (e.g. Jones 2011; Conrad 2020)
or have analysed the unfolding of longer stretches of other people’s written discourse,
though without any accompanying ‘mind-reading’ claims (e.g. Duncker 2018). At any
rate, all of these analyses by integrationists serve first of all to explicate the ontological
nature of the sign and its embeddedness in human activities.
Integrationists focus on the notion of activity as a fundamental lay concept when it
comes to understanding human communication, for which they need no specific ‘theory’
of activities. Instead they start from the insight that communication is a process which
consists of activities that are integrated by means of contextualized signs of all kinds. At the
centre of an integrational theory are thus the concrete first-order sign-making processes
that individuals (not members of ‘interpretive communities’) engage in, for whom the
‘linguistic’ and the ‘non-linguistic’ are precisely not separate but always integrated
domains of knowledge, as stated by the Harrisian ‘principle of cotemporality’ (Harris
258 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

1998). In other words, what matters in human life is that activities of whatever kind are
integrated by whatever semiological means are available to individuals engaged in specific
programmes in order to further their specific communicational objectives. There is no
detachable ‘linguistic message’ that is experienced identically by speaker A and hearer B
(or analyst C). Communication, as far as human beings are concerned, cannot be analysed
‘scientifically’ in terms of ‘verbal’ and ‘non-verbal’. Not because lay speakers would not
analyse communication in this way, but precisely because communication is not a fact
but a lay perspective (Harris 2006a: 13). In other words, communication is radically
open-ended (both ontologically and in its interpretations). This, in turn, means that
communication needs to be treated as a much broader concept than any ‘glottocentric’
(Saussurean) model allows for. Communication, for the integrationist, is bound to one’s
sign-making. Realizing the one activity is linked to another activity (in whatever way this
link is construed) presupposes the making of signs. This communication with oneself –
self-communication, as Harris (1996) calls it – may in turn lead to further activities
(thinking, speaking, gesturing, etc.). What is radically open-ended about communication
is not only what is part of it and for whom, but also when communication begins and
ends, between whom, and which personal past and future experiences combine in order
to make the present communication episode meaningful.
Regarding the ontology of the sign, the integrationist holds that the signs exist only
insofar as they serve an integrational purpose: once they have fulfilled their integrational
function, the signs no longer exist. As Harris puts it: ‘the sign exists as a construct by
reference to which certain communicational activities [. . .] are integrated’ (2006b: 715).
It is important to keep in mind that integrationists deny that signs ‘pre-exist’ in any way,
which is tantamount to rejecting the abstract sign altogether. In other words, there is no
type that is repeatable innumerable times as a token, i.e. as the ‘same’ sign in different
situations. Rather it is the other way round: it is the recurring of tokens that allows for
the creation of the type as a product of the here-and-now (Ellis 1994; Harris 2013).
‘Sameness’ is surely a semiological process, constrained by cultural factors, that people
recognize as playing an important communicational role, and as such integrationists do
not necessarily regard it as problematic. However, its rejection at the level of the sign
ontology is of paramount importance: arguably it is that very rejection which makes the
Harrisian sign incompatible with both the Saussurean and the Peircean sign. What lay
people might (retrospectively) argue is the ‘same sign’ produced several times (within
a particular reconstructed communication episode) must be, from an integrational
point of view, unique signs made on the various occasions along the sign-maker’s
biographical continuum. In this respect, the integrationist does not accept that there
are ‘facts’ (linguistic and otherwise) that can be established independently of the sign-
makers themselves (Harris 1998). Crucially, however, even a sign repeated is, from an
integrational semiological point of view, a creation ex nihilo: its creation occurs in a
new situation, but its integrational function relies on its being construed by the sign-
maker(s) as having a predecessor sign. The first sign (e.g. ringing the doorbell for the
first time) and the second sign (ringing the doorbell for the second time) cannot possibly
mean ‘the same’ for either the person ringing or the person hearing it. However, the two
signs might be construed as being ‘the same’ under specific circumstances and for certain
communicational purposes. Sameness is never ‘already there’.
On Harris’s view, signs are thus the products of contextualization: they both result from
and facilitate the integration of particular activities in particular circumstances (Harris
2006a). The signs do not exist prior to or independently of communication because
INTEGRATIONAL LINGUISTICS AND SEMIOTICS 259

the signs are personal creations here-and-now: that is, there are no signs belonging to
a metaphysical realm, but only signs that are made (by particular sign-makers) in order
for various activities to be integrated. If signs only served one single activity rather than
the integration of different activities, human communication would be radically different
from how we know it based on personal experience. Activities are recognizable to us as
general activities (e.g. thinking, speaking, listening, writing, reading and gesturing), but
an activity only exists in a particular moment as engaged in by a particular person. There
is only my view (and yours, etc.) on whether what I am doing right now is engaging in
the activity of ‘reading’. Nobody else can determine this in my stead. What counts as
‘reading’, in fact, may arguably be a matter on which people might disagree. Moreover,
does reading silently and reading aloud (respectively to oneself, to somebody else present
or recording it for somebody else not present) count as the same basic activity? For one,
reading silently and reading aloud involve very different activities. The same goes for
writing a note to oneself as opposed to writing a note for somebody else. Integrationists
recognize that humans are always circumstantially determined, aware of themselves, which
is why the here-and-now is communicationally constrained for any individual by factors
of three kinds, namely (i) biomechanical factors, which relate to the physical and mental
capacities of the human being, (ii) circumstantial factors, which relate to the specifics of
particular situations, and (iii) macrosocial factors, which relate to practices established in
the community (Harris 1998: 29). Integration in any semiologically relevant sense is the
individual’s continuous integration of (remembered) past experience and (anticipated)
future experience with the here-and-now, which, the integrationist argues, cannot rely
on a set of shared abstract signs but necessitates the making of signs. To manage the daily
integrational tasks is to know how activities are temporally integrated in the macrosocial
patterns of the community and to act accordingly (not doing anything may also be a
form of action). Harris (2009a) calls this knowledge ‘integrational proficiency’; however,
it is not a ‘proficiency’ that any traditional intelligence tests would be able to disclose
(Harris 2013). It is not knowledge atemporally stored in the brain (‘knowing that’) but
rather knowledge in the sense of ‘knowing what/how to do’, i.e. knowledge embedded
in activities (Harris 2009b). Thus, if I do not know a word I have encountered as part of
a particular activity (e.g. reading a newspaper article) and I look it up in the dictionary,
and if as a result I can make sense of that word as it occurs in that particular article I am
reading, I know its meaning sufficiently well for the purposes at hand, i.e. I know ‘what to
do with it’ (Harris 1998: 63). On this view, the idea of knowledge for the sake of knowledge
is a product of the Language Myth, i.e. our ability to decontextualize communication and
extract from it something called ‘knowledge’ (‘linguistic knowledge’ in the case of the
unfamiliar word): it is the belief that knowledge exists independently of the knower and
thus independently of individuals integrating their own activities (with other people’s
activities) in specific circumstances in order to achieve specific purposes, constrained by
the three aforementioned parameters (biomechanical, circumstantial, macrosocial). In
other words, for the integrationist, there is no contextless knowledge (Harris 2009b).
Harris (1981) specifically rejects two fallacies that characterize the Language Myth as
a Western cultural enterprise as well as academic linguistics as a ‘segregationist’ project:
they are the fallacies of ‘telementation’ and of the ‘fixed-code’ (or ‘determinacy’), which,
taken together, explain the role of words as enabling speakers of the ‘same language’ to
transfer mental concepts from one mind to another, resulting in mutual understanding,
which Pennycook (2018) identifies as a Eurocentric humanist ideal. The integrationist,
however, goes on step farther than Pennycook, who equally rejects the notion of
260 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

‘languages’ as scientifically describable objects, and adds ‘words’ to the list of second-
order abstractions, hence viewing them as products of human communication – not as its
prerequisites (e.g. Love 1990, 2017). Since integrationists acknowledge communicational
activities of the first order as the only human reality, they view signs as spontaneous
creations of the individual’s mind, i.e. they are, as Harris and Hutton (2007) put it,
radically indeterminate in both form and meaning. ‘Radical indeterminacy’ refers to the
impossibility to understand signs outside of their contexts: any understanding is done in
‘its’ context. In other words, there is no third-person perspective on signs and hence there
can be no ‘science’ of semiology (pace Saussure). The signs are what they are and mean
what they do to the various sign-makers when they make them. The signs are not radically
indeterminate to their makers, though, as the signs they make integrate activities. In order
to have any integrational function, signs need to be ‘determinate’ for their makers: failing
to understand what the other is saying does not render my signs ‘radically indeterminate’.
The signs – i.e. my interpretations – may simply be inadequate, in my perception, to
integrate my activities with another person’s activities here-and-now. In order for
them to be adequate I do not need to know the signs ‘objectively’. In other words,
semiological determinacy, for the integrationist, is not of the (fixed-code) lexicographical
formula ‘bachelor:unmarried man’. Signs do not acquire an integrational function the
way determinate words are bestowed determinate meaning through the authority of
the dictionary. Using a dictionary has certainly become one viable ‘glossing practice’
(Harris 1998) in order to reduce semantic indeterminacy and facilitate certain kinds of
communication, but crucially using a dictionary is culturally conditioned: it is a particular
(historically grown) activity embedded in other activities. On that score the integrationist
agrees with Saussure: the linguistic signs we make are not the same as the words we find
in the dictionary, though there must be some perceived connection between them for
the individual sign-maker. The contextualized signs I am making when interacting with
others, for example, have no first-order connection with the (second-order) words in the
dictionary that my signs allegedly correspond to. To ask ‘what are these signs “signs of”?’
is a culturally conditioned question, with many communicationally relevant ramifications,
but arguably it hinders a non-ethnocentric understanding of the ontology of the sign. A
sign is never a sign of something independently of the activities for which it was created
in the first place. Harris also does not treat the sign as having a form and a meaning (as in
Saussure’s signifier and signified). When Harris talks about radical indeterminacy in both
form and meaning, he does so through the lens of mainstream linguistics. Strictly speaking,
a sign ‘made’ (in the Harrisian sense) cannot have ‘a form’ and ‘a meaning’ (Davis 1997).
Analysing signs in this way presupposes their decontextualization: i.e. treating signs as
detachable from communication. From this it follows that the integrationist rejects the
substitute (or ‘surrogational’) sign (Harris 1996), i.e. the very notion that a sign is one
thing ‘standing for’ (or representing) another thing. The surrogational sign is not a sign
embedded in activities, created for and as part of the activities that particular individuals
engage in. In everyday lay discourse it is common to explain meaning surrogationally, but
in order to see such a relation between a sign and its ‘object’ (in the Peircean sense), I have
to assign some sort of semiological value to something, i.e. make it a sign. The signs never
antecede the activities that make them signs in the first place, but it is tempting to think
otherwise, especially when it comes to material objects whose signification we assume to
be permanent, such as a wedding ring or a traffic sign (Harris 1996). The integrationist
equally has to reject the Saussurean sign, i.e. the very notion that signs are defined purely
internally by an abstract (linguistic) system they belong to, and this allegedly is how a
INTEGRATIONAL LINGUISTICS AND SEMIOTICS 261

sign is known by its user. Given the temporality of the sign, however, neither form nor
meaning can exist as abstractions. In turn, Harris proposes a semiology grounded in first-
person experience: there is no sign without its personal maker.

­INTEGRATIONISM AND BIOSEMIOTICS

The only branch of semiotics that shares several points of convergence with Harrisian
integrationism is Thomas Sebeok’s global semiotics (Sebeok 1979), which is founded
on a Peircean understanding of the sign. Peirce is a ‘surrogationalist’ (Harris 1996),
construing a sign, as philosopher and semiotician John Deely (2003: 185) put it, as ‘one
thing representing another than itself to yet another’. There is no indication that Sebeok
was aware of Harris’s work, while Harris mentioned Sebeok only once in a footnote
(1996). In terms of their sign conceptions, global semiotics and integrationism differ
from each other in relevant ways, at least from an integrational point of view. In fact,
from a semiotic standpoint it could be argued that Deely’s explanation of how organisms
with cognitive abilities make meaning comes close to an integrationist position of a more
‘global’ stamp:

The organism, according to its own nature and past experiences, attaches a value to
the stimulus and relates that stimulus to its own needs and desires. In other words, the
mere stimulus of sensation becomes incorporated objectively [i.e., species-specifically]
into a whole network of experience wherein it acquires a meaning.
(2003: 178–9)

However, the concept of ‘activity’ does not take centre stage in Deely’s analysis –
in fact, it is not on stage at all. The reason for this is arguably that the Peircean sign
only presupposes activities against which the sign-object relation is established by the
interpretant. Thus while the activities are clearly there in Peirce, the focus remains on the
signs, without which there would be no activities in the first place for any sign relations
to be realized. Activities presuppose signs from a Peircean semiotic perspective – which
is why the signs and the activities can be considered separately – while signs presuppose
activities in Harrisian semiology: the signs, for Harris, are only acknowledged as existing
insofar as they integrate (human) activities. Those signs are unique and therefore they
only make sense the way they do (to the sign-makers) because of the very activities here-
and-now. This is not to deny that sign-makers make connections between the activities
here-and-now and the activities of the past (as integrated by them). However, there is
no neutral perspective on why someone makes the signs he/she makes, i.e. how exactly
they connect to his/her communicational biography. So to say that a sign ‘stands for’
something other than itself to someone cannot capture the dynamic nature of human
activities, in which signs facilitate their integration and at the same time result from
those integrations. Peirce’s notion of ‘semiosis’ (signs generate further signs) comes
close to capturing this dynamism, but, as the integrationist would maintain, the signs
by themselves do not exist. Our reluctance to let go altogether of the sign in and by
itself (if only as a potential), the integrationist argues, stems precisely from our ability
to analyse signs as detached from the activities in which they were created (hence we
cling to the idea that a sign is something that ‘stands for’ something else), while we easily
forget that this very analysis occurs itself as part of a macrosocially, circumstantially and
biomechanically conditioned activity.
262 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

In spite of significant differences in their respective conceptions of the sign, Harrisian

integrationism and global semiotics, and in particular biosemiotics (the semiotic
branch developed from biologist Jakob von Uexküll’s work [1982]), are united in their
philosophical views on the issues of reality and objectivity. Building on Uexküll, John
Deely (e.g. 2003, 2009) has developed the most comprehensive biosemiotic account
of what constitutes ‘objective reality’ (as opposed to an orthodox scientific account of
‘reality’). In his work, Deely resurrected the intriguing distinction between thing and
object, together with the distinction between physical reality and objective reality. His is
a rejection of both naïve realism and nominalism, but it is not per se a critique of academic
linguistics or its underlying theory of the sign. While not using Deely’s terminology, Harris
deals with the same issues in his work on science (Harris 2003a, 2005), however with
particular reference to the Language Myth as it pertains to the so-called ‘supercategories’
(e.g. Harris 2003b, 2004, 2005). How are we to understand Deely’s critique of objectivity,
and how does it differ from an integrational critique? Building on the notion that the sign
is constituted as a triadic relation, Deely affirms that reality is always ‘objective’ because it
is species-specific and thus constrained by the sign-making capabilities of a species and its
sensoria. Deely’s account makes clear how orthodox science has ignored ‘objective reality’
in the biosemiotic sense, confusing objectivity with the physical environment because of
its Baconian goal to investigate the ‘things as they are’, i.e. as God knows them. As Deely
rightly argues, only human beings are capable of contemplating objects in relation to
the objects themselves, while non-human animals make sense of objects exclusively in
relation to their own needs and perception. Contemplating objects for their own sake
(i.e. as ‘things’) and to treat them as ‘objectively’ given (not in the biosemiotic sense)
is an exclusively human attribute (among the terrestrial species). The cardinal mistake
committed by orthodox science, for Deely (2009: 167), thus lies in assuming that the
physical environment is the way it is regardless of the creatures that inhabit it and make
sense of it. Deely does not go as far as to deny the reality of the physical universe (as a
thing): he states, in fact, that there is a physical universe which is the same for all types
of organisms, but adds that this physical environment is different from the points of view
of the various ‘Umwelten’ (Uexküll [1940] 1982). This comes as close to admitting that
‘reality’ is unknowable as it gets in biosemiotics. On the other hand, Deely (2009: 160)
talks extensively about so-called ‘hardcore reality’ when it comes to the relations between
subjectivities (substances): thus the relation between a father and his biological son is
intersubjectively real: in other words, it is not a sign-mediated relationship. On the other
hand, the relation between a woman and her being a witch is sign-mediated – if witches
are not real – i.e. it involves a triadic, and not a dyadic, relationship. As Deely argues at
length in his work, the distinction between reality and unreality is not a consequential one
from the point of view of semiotics. As Deely (2009: 167) puts it, ‘signs are indifferent
to reality in the physical sense’. Biosemiotics cannot afford to take a ‘scientific’ point of
view, as the latter pretends to be a perspective unmediated by signs. For the semiotician,
in fact, science could be seen as a branch of anthroposemiotics, i.e. the institutionalized
human activity of investigating a sign-less reality through signs (science communication in
its various forms), a reality made of ‘things’, not ‘objects’. Crucially this insight is shared
by Harris and elaborated on in his book The Semantics of Science (Harris 2005), i.e. the
insight that the study of ‘things as they are’ is intrinsically part of the human umwelt
and therefore species-specific. Science cannot exist detached from the integrated human
activities that actually make it, including the activities lay people commonly identify
as involving ‘language’. In this sense we may conclude that Science, Art, Religion and
INTEGRATIONAL LINGUISTICS AND SEMIOTICS 263

History are human enterprises (among other so-called ‘supercategories’): they make sense
the way they do because they are the product of the human ‘communicational universe’
(Harris 1996) – they do not have to be graspable by any languageless creature. Hence it
would be absurd to imagine highly intelligent alien races that have ‘science’ as we know
it. There simply is no languageless science. On that point, Sebeok and Harris would agree,
‘language’ is an entirely human experience, and only humans can attribute ‘language’ to
other creatures (higher-order animals, extraterrestrials, even plants), which, however,
does not mean that those creatures possess it (whatever their cognitive abilities). Recent
developments within posthumanist approaches to language, such as Leonie Cornips’s
suggestion to ‘put the animal on the linguistic research agenda’ (Cornips 2019), would
have been rejected out of hand by both Sebeok and Harris.
Deely’s critique of science’s failure to distinguish between objects and things is not
at the same time a critique of the language of science. The latter is part of the human
objective world, while the ‘things’ themselves are not. Sebeok’s semiotics thus views
language as the ‘primary modelling device’ that humans make use of in the way they
constitute their objective world (Petrilli and Ponzio 2011: 138). It is something additional
that non-human animals lack. Harris (2005), in turn, criticized the worldview sustaining
the scientific notion of ‘objective reality’ (in the non-biosemiotic sense) as part of his
critique of the Language Myth. The very notion of such a reality is only possible because
we construe languages to be labelling devices, as Saussure (1983: 16, 65) famously noted
in his critique of nomenclaturism. Biosemiotics, for all its material grounding, cannot
dismiss altogether Saussure’s idealist explanation of what ‘a language’ is, i.e. a system of
abstract signs. Within segregationism, Saussure’s model of communication is the most
convincing one, precisely because it constitutes an idealization. Harris, on the other hand,
rejects the idea of ‘a language’ altogether (other than as a lay concept), thus dismissing
both nomenclaturist materialism and Saussurean idealism. From an integrational point
of view, there can be no ‘language of science’ understood as a mini fixed-code shared
by the members of the scientific community. The idea that there is such a language is
the intellectual product of segregating language from ‘communication’ (in the broad
integrationist sense) and treating ‘linguistic sign-making’ as something sui generis. To
see this in all clarity, however, one needs to theorize the sign in relation to the activities
that it uniquely serves to integrate. The Harrisian argument is thus that science (as the
‘supercategory’ that we know it to be) owes its existence to (what is perceived as) the
availability of a specific kind of language and a macrosocially institutionalized discourse,
upon which membership and excommunication ultimately rest. As Harris shows in The
Semantics of Science, science tacitly relies on a ‘surrogationalist semantics’, itself divided
into two distinct branches: reocentrism and psychocentrism. It is chiefly the reocentric
conception of how words get their meaning, namely by ‘standing for’ the things they are
names of, that prevails in orthodox science. It is a language-philosophy that ties in with
the view that the world exists mind-independently and that knowledge of the things that
populate the universe requires knowledge of the correct linguistic labels, by means of
which things can be identified as the same over and over again. Both types of knowledge
are assumed to be context-independent (i.e. the biomechanical and circumstantial
parameters of human communication are not taken into account).
It has been the scientist’s job to continuously map the universe, as scientific knowledge
progresses, whereby one name gets replaced by a new name (or the name remains but
is redefined). Name variation, within a ‘reocentric’ model of language, must be handled
with suspicion and possibly dissolved, i.e. cases in which one and the same thing is known
264 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

by different (potentially contradictory) names, as well as cases in which the same name
refers to different things. In either case a disruption of the reocentric ideal (one name-one
referent) has occurred. Some names must therefore be rejected – scraped off the linguistic
map – on the grounds that they are psychocentrically defined, i.e. they merely ‘stand for’
ideas in the mind. From a Harrisian point of view, either surrogationalist conception
of meaning is fallacious for its failure to treat words in relation to the activities they
integrate, which is why the alleged ‘determinacy’ of scientific terms is an illusion brought
about by decontextualization. For the integrationist, ‘hardcore reality’ is not necessarily a
notion worth preserving: especially not if that commits one to the idea that humans can at
any moment step outside their own human umwelt constituted by human communication
(which involves but is not determined by what we commonly refer to as ‘language’). We
may go as far as to say that there is something ‘there’, independently of how human
individuals integrate it, but even that insight is semiologically grounded. While Harris
views science as one of many terrestrial activities with a high degree of institutionalized
knowledge, he is not a denier of science nor of applied science and its achievements.
However, unlike biosemiotics, integrationism provides a powerful explanation of
‘languages’ (including the ‘language of science’) as second-order abstractions, i.e. products
of communication presupposing linguistic reflexivity (Love 1990). The distinction
between thing and object, as Deely argues, is an important one to understand ourselves
as a species. What is equally important, however, is to recognize that both things and
objects are integrational products of human thinking and human discourse. They are
signs that serve different kinds of human activity, and personal experience tells us that
things and objects are intertwined as part of the human umwelt in such ways that a clear
distinction between the two requires an artificial categorization which cannot capture
first-order human sign-making.

CONCLUSION: THE HUMANITY OF SIGNS

Global semiotics, as the name suggests, is interested in theorizing the sign as a universal
phenomenon. Human signs and non-human signs are explained through the same sign
model, which is triadic and which treats sign-making as an attribute of life itself (Sebeok
1979). The Peircean sign is free of the limitations of the Saussurean dyadic sign, and is
able to incorporate the symbolic sign into its triadic model and account for the non-
idealist (i.e. material) dimension of Saussurean linguistics: parole. Biosemiotics, being
committed to such a global view of signs, cannot afford to adopt the Harrisian sign
conception, which takes personal linguistic experience as its guide to understanding what
a sign is and how it means. In this respect, integrationism is arguably antithetical to a
global semiotics. Within the latter, in fact, human activities and non-human activities
must ultimately be treated as ‘activities’ tout court which trigger signs. Integrationism, in
turn, takes the position that we cannot know other people’s minds let alone non-human
minds. Observable behaviour per se, in the eyes of the integrationist, does not provide any
insights into sign-making. Integrationism has thus very little to say about how non-human
creatures integrate activities by means of signs of their own making, while insisting that
‘integration’ is not unique to human beings. It also adopts a sceptical position when
it comes to the question whether lower life forms (e.g. micro-organisms) can be sign-
makers: does a bacterium make ‘its’ own signs? In this respect, integrationism is deeply
humanist and anthropocentric. It is not, however, ‘glottocentric’ in Sebeok’s sense: first
INTEGRATIONAL LINGUISTICS AND SEMIOTICS 265

of all, it is reluctant to admit any clear-cut separation between linguistic activities and
non-linguistic activities. Strictly speaking, there is no linguistic sign (as theorized in
segregational linguistics) but only the integrated sign. Secondly, it takes a lay-oriented
approach to the phenomenon of ‘language’, i.e. there is no reocentric surrogational
definition of ‘language (mass)’. Sebeok’s critique of semiology as glottocentric thus needs
to be revised in light of the present contribution: the main characteristic of semiology
(from a Harrisian point of view) would thus not be glottocentrism, but rather the
more positive insight that ‘language’ is an integrated mode of communication, which
also means that human communication cannot be decomposed in any facile way: the
formula ‘human communication equals primate communication plus language’ is itself
glottocentric, as it fails to take into account the differences that exist between species-
specific ‘communicational infrastructures’ (Harris 1996). Those (glottocentric) linguistic
theorists who argue that semiology can only be Saussurean (e.g. Weigand 2018) obviously
have to reject Harris’s view of the sign. One way they typically dismiss Harris is to say
that integrational linguistics is not ‘linguistics’ after all, or that integrationism takes too
broad a conception of ‘communication’.
Both Harris and Sebeok underlined the importance of taking a species-specific view on
communication in their respective work, while they would have disagreed on questions
concerning the ontology of the sign. For Harris (2009b), the term species is itself radically
indeterminate: pace Darwin, it is not a determinate sign. At the same time, Harris does
not discard the notion of a ‘species’, finding its integrational function a helpful one for the
(humanist) project of self-knowledge. However, it cannot be denied that even our human
experiences differ vastly across time, and between cultures and individuals (Pennycook
and Makoni 2019). That is precisely why a lay-oriented approach to the human sign is
indispensable.

REFERENCES
Bonfante, G. (1947), ‘The Neolinguistic Position (A Reply to Hall’s Criticism of
Neolinguistics)’, Language, 23 (4): 344–75.
Cobley, P. (2011), ‘Mythbusting’, Language Sciences, 33 (4): 511–16.
Cobley, P. (2014), ‘Codes and Coding: Sebeok’s Zoosemiotics and the Dismantling of the
Fixed-code Fallacy’, Semiotica, 1998: 33–45.
Conrad, C. (2020), ‘Creating Reality as a Locally Tailored Interface – An Integrational,
Pragmatic Account of Semiosis’, Sign System Studies, 48 (1): 12–31.
Cornips, L. (2019), ‘The Final Frontier: Non-human Animals on the Linguistic Research
Agenda’, in Linguistics in the Netherlands, vol. 36, 13–19, Amsterdam: John Benjamins.
Davis, D. R. (1997), ‘The Three-dimensional Sign’, Language Sciences, 19 (1): 23–31.
Daylight, R. (2012), ‘The Difference between Semiotics and Semiology’, Gramma: Journal of
Theory and Criticism, 20: 37–50.
Deely, J. (2003), The Impact on Philosophy of Semiotics, South Bend: St. Augustine’s Press.
Deely, J. (2009), Purely Objective Reality, Berlin: Mouton de Gruyter.
Dessì Schmid, S. (2004), ‘Das Zeichen in der Sprachtheorie Benedetto Croces’, in G.
Hassler and G. Volkmann (eds), History of Linguistics in Texts and Concepts/Geschichte
der Sprachwissenschaft in Texten und Konzepten, vol. 2, 699–711, Münster: Nodus
Publikationen.
Duncker, D. (2018), The Reflexivity of Language and Linguistic Inquiry. Integrational
Linguistics in Practice, Oxon: Routledge.
266 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Ellis, J. M. (1994), Language, Thought and Logic, Noyes: Northwestern University Press.
­Harris, R. (1981), The Language Myth, London: Duckworth.
Harris, R. (1993), ‘What Is Philosophy of Linguistics?’, in R. Harré and R. Harris (eds),
Linguistics and Philosophy: The Controversial Interface, 3–19, Oxford: Pergamon.
Harris, R. (1996), Signs, Language and Communication, London: Routledge.
Harris, R. (1998), Introduction to Integrational Linguistics, Oxford: Pergamon.
Harris, R. (2003a), History, Science and the Limits of Language, Shimla: Indian Institute for
Advanced Study.
Harris, R. (2003b), The Necessity of Artspeak, London: Continuum.
Harris, R. (2004), The Linguistics of History, Edinburgh: Edinburgh University Press.
Harris, R. (2005), The Semantics of Science, London: Continuum.
Harris, R. (2006a), ‘Communication: or How Jill Got Her Apple’, in Integrationist Notes and
Papers 2003–2005, 11–14, Crediton: Tree Tongue.
Harris, R. (2006b), ‘Integrational Linguistics and Semiology’, in K. Brown (ed.), Encyclopedia
of Language and Linguistics, 714–18, Oxford: Elsevier.
Harris, R. (2009a), ‘The Integrational Conception of the Sign’, in Integrationist Notes and
Papers 2006–2008, 61–81, Gamlingay: Bright Pen.
Harris, R. (2009b), After Epistemology, Gamlingay: Bright Pen.
Harris, R. (2010), ‘Linguistic Inquiry: An Integrational Approach’, Unpublished Manuscript.
(revised version of Harris 1998).
Harris, R. (2013), Language and Intelligence, Gamlingay: Bright Pen.
Harris, R. and C. Hutton (2007), Definition in Theory and Practice. Language, Lexicography
and the Law, London: Continuum.
Jones, P. E. (2011), ‘Signs of Activity: Integrating Language and Practical Action’, Language
Sciences, 33 (1): 11–19.
Keane, W. (2018), ‘On Semiotic Ideology’, Signs and Society, 6 (1): 64–87.
Keller, R. (1995), Zeichentheorie: Zu einer Theorie semiotischen Wissens, Tübingen/Basel:
Francke.
Keller, R. (1998), A Theory of Linguistic Signs, trans. K. Duenwald, Oxford: Oxford University
Press.
Linell, P. (2018), ‘Dialogism Is an Integrationism: Reply to Peter Jones’, Language and
Dialogue, 8 (2): 306–23.
Love, N. (1990), ‘The Locus of Languages in a Redefined Linguistics’, in H. Davis and T.
Taylor (eds), Redefining Linguistics, 53–117, London: Routledge.
Love, N. (2017), ‘On Languaging and Languages’, Language Sciences, 61: 113–47.
Pablé, A. (2018), ‘Radical Indeterminacy, Idealism, Realism: Benedetto Croce vs. Roy Harris’,
Language & Communication, 61 (4): 46–57.
Pablé, A. (2019), ‘Is a General Non-ethnocentric Theory of Human Communication Possible?
An Integrationist Approach’, Lingua, 230: Article 102735.
Pennycook, A. (2018), Posthumanist Applied Linguistics, London & New York: Routledge.
Pennycook, A. and Makoni S. (2019), Innovations and Challenges in Applied Linguistics from
the Global South, London and New York: Routledge.
Petrilli, S. and A. Ponzio (2011), ‘A Tribute to Thomas A. Sebeok’, in P. Cobley, J. Deely,
K. Kull and S. Petrilli (eds), Semiotics Continues to Astonish, 307–30, Berlin: Mouton de
Gruyter.
Saussure, F. ([1916] 1983), Course in General Linguistics, trans. R. Harris, London:
Duckworth.
INTEGRATIONAL LINGUISTICS AND SEMIOTICS 267

Sebeok, T. (1975), ‘Zoosemiotics at the Intersection of Nature and Culture’, in T. Sebeok (ed.),
The Tell-tale Sign. A Survey of Semiotics, 85–95, Lisse: Peter de Ridder.
­Sebeok, T. (1979), The Sign & Its Masters, Texas: The University of Texas Press.
Uexküll, J. ([1940] 1982), ‘The Theory of Meaning’, Semiotica, 42 (1): 25–82.
Weigand, E. (2018), ‘The Theory Myth’, Language & Dialogue, 8 (2): 289–305.
Weinreich, U., W. Labov and M. Herzog (1968), ‘Empirical Foundations for a Theory of
Language Change’, in W. Lehman and Y. Malkil (eds), Directions for Historical Linguistics,
95–195, Austin: University of Texas Press.
268
 CHAPTER THIRTEEN

Cognitive Linguistics
and Semiotics
JORDAN ZLATEV AND MÖTTÖNEN TAPANI

INTRODUCTION
Generative linguistics in the Chomskyan tradition has on occasion been referred to
as ‘cognitive’, in the sense that it aspires to deal with ‘what the grammar purports to
describe: a system represented in the mind/brain’ (Chomsky 1990: 679): the latter a
phrase that is as common as it is misleading. However, Cognitive Linguistics (henceforth,
CL) currently refers to a very different approach to language, with its dedicated journals,
conference series and handbooks (e.g. Dancygier 2017; Geeraerts and Cuyckens 2007).
This tradition emerged in the late 1970s, and a well-known book on how we ‘live by’
metaphor is often heralded as its birthdate (Lakoff and Johnson 1980). CL posed itself
in opposition to generative linguistics, on the one hand, and to logical semantics on the
other (e.g. Croft and Cruse 2004; Fauconnier and Turner 2002; Johnson 1987; Lakoff
1987; Langacker 1987; Talmy 2000). While Chomskyan linguists emphasize (a) syntax,
(b) innateness and (c) modularity as fundamental characteristics of linguistic knowledge,
CL researchers do the reverse, privileging (a´) semantics, (b´) ‘usage-based’ linguistic
knowledge, which is grounded in (c´) general cognition (e.g. Saeed 2009). And while the
logical tradition of semantics is based on truth and denotation (Dowty, Wall and Peters
2012), cognitive semantics boldly adopts the motto of ‘meaning as conceptualization’
(Langacker 1987; Gärdenfors 1999).
On the face of it, while CL and semiotics share a pre-occupation with meaning, they
also seem miles apart. The predominantly subjectivist, individual-psychological take on
meaning typical for CL is clearly at odds with the structuralist tradition of semiotics
stemming from Saussure ([1916] 1960). Perhaps less obviously, it is also at odds with the
Peircian tradition, which has endeavoured to ‘dementalize’ semiosis (sign use) throughout
much the twentieth century (see Daddesio 1995), converging in its criticism of psychologism
with logic and Husserlian phenomenology (e.g. Stjernfelt 2014). Furthermore, while
semiotics, of any kind, has always aimed at characterizing meaning (making) in general,
and at the same time seeking to explain what makes different semiotic systems such as
language, gesture, depiction and music unique, CL is, by definition, focused on language.
Yet, at least three separate developments have contributed to rapprochement between
CL and semiotics over recent years. First, cognitive linguists have, to an increasing
degree, been engaged in analysing meaning-making beyond language, involving semiotic
systems like gestures, pictures and music, often under the banner of ‘multimodality’ (see
270 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

section Multimodality and Polysemiosis), and thus finding themselves in increasingly

semiotic territory. Second, in relation to this and other factors, some cognitive linguists
have turned to semiotics (and phenomenology) for help with concepts like iconicity and
intersubjectivity (e.g. Devylder 2018; Oakley 2009). Third, a number of semioticians have
found cognitive linguistic constructs like mental spaces and image schemas consonant
with structures and processes in meaning-making at large (e.g. Brandt 2004; Daddesio
1995).
This mutual interest has been so strong in some quarters that, with additional impetus
from cognitive science (Deacon 1997; Donald 2001; Konderak, 2018), a new discipline
has emerged over the past decade: cognitive semiotics (Zlatev 2012, 2015). Within
this, concepts and methods from semiotics, cognitive linguistics and cognitive science
have been successfully integrated in the service of the study of social interaction (Tylén,
Fusaroli, Bundgaard, and Østergaard 2013), literature (L. Brandt 2013), music (Giraldo
2019), choice making (Mouratidou 2019), street art (Stampoulidis and Bolognesi 2019)
and many other topics.
What we aim to show in this chapter is, above all, how through a dialectical synthesis
of sorts, cognitive semiotics has been able to address problems with CL by bringing in
correctives from semiotics and phenomenology, and thus avoiding two recurrent, and
interrelated, fallacies: (1) the subjectivism that comes from the excessive psychologizing
of notions that are essentially intersubjective and (2) confusing perceptual, and other
non-signitive processes like imagination, with sign use and linguistic conventions. In
the following sections we focus on four topics where CL research has displayed these
fallacies, and suggest how (cognitive) semiotics can redress them. These four topics are
construal, iconicity, metaphor and ‘multimodality’.
First we discuss the notion of construal, which implies that ‘[. . .] meaning can be defined
as non-objective [. . .] in that it always manifests some restricted and perspectivized take
on the matters at hand’ (Möttönen 2016a: 13). The problem here has been that scholars
like Langacker (1987) and Croft and Cruse (2004) do not distinguish between construal
as (a) a private (and pre-linguistic) phenomenon, as (b) an intersubjective, pragmatic
phenomenon of language/sign use, and (c) as an aspect of the conventional, socially
shared semantics of the expressions in question. We review a number of ‘dimensions
of construal’ from the CL literature, both distinguishing (a–c) and showing their inter-
connectedness.
Given how frequently the notion of ‘imagery’ appears in CL, it is surprising how
seldom the notion of iconicity is explicitly employed in analysis, though it is often
implicitly assumed (see Möttönen 2016a). As we show in the section Iconicity, one
problem is that iconicity is rarely defined, and when it is, this is commonly done in
terms of a ‘resemblance-based mapping of form and meaning’ (Dingemanse, Perlman
and Perniss 2020: 2). Unless language-independent criteria for such resemblance can be
given, such a definition is potentially vacuous, and this problem is compounded in CL,
where both ‘form’ and ‘meaning’ are typically understood as internal, mental phenomena.
We illustrate the problem with Wilcox’s notion of ‘cognitive iconicity’ defined as
distance between structures in conceptual space (Wilcox 2004), which both lacks such
intersubjective criteria and makes iconicity indistinguishable from a complementary
interpretative principle (or semiotic ground): indexicality. In contrast, we show how the
seminal work of Jakobson (1965) lends itself to a cognitive-semiotic perspective and can
be used to both avoid the typical CL pitfalls and rehabilitate the fundamental role of
iconicity in language (Itkonen 2005; Devylder 2018).
COGNITIVE LINGUISTICS AND SEMIOTICS 271

In the section Metaphor we turn to the third and most complex type of iconic signs,
or (hypo)icons in the Peircean classification: The topic is huge, but we focus on how
most cognitive linguists have not taken a semiotic approach, but rather attempted to
explain metaphor in terms of (unconscious) mental processes such as ‘cross-domain
mappings’ and ‘mental simulations’ (Lakoff and Johnson 1999; Gibbs 2006). We argue
that this once again commits the two fallacies of subjectivism and motivation/signification
conflation. Based on recent work from cognitive semiotics (Stampoulidis, Bolognesi and
Zlatev 2019; Devylder and Zlatev 2020) we show how these can be avoided in a model
that distinguishes three different levels of meaning-making, while also highlighting the
dynamic relations between them.
In the section Multimodality and Polysemiosis, we briefly consider the area where CL
and semiotics most obviously overlap: in the analysis of meaning-making when – as in
most forms of human communication – different kinds of signs and signals like words,
gestures, facial expressions, music and drawings combine. As mentioned, CL has used
the ‘buzzword’ (Devylder 2019: 148) multimodality, but this term is highly ambiguous
(i.e. sensorial, semiotic, communicative, etc., ‘modes’ and ‘modalities’) and confusing.
Cognitive semiotics has helped to distinguish the combination of sensory modalities
from polysemiosis, where more than one semiotic system (e.g. language, gesture,
depiction, music) is used, irrespective of whether one or more perceptual modalities
(e.g. vision, hearing and touch) are involved (Zlatev 2019), and to begin analysing the
interactions between systems and modalities. In the final section, we conclude that the
cognitive semiotic syntheses outlined in this chapter suggest that connections between
CL and semiotics are likely to be further strengthened.

CONSTRUAL
In order to emphasize the claim that language does not denote the world (directly), but
that, if at all, it does this through the prism of the mind, many CL researchers adopt the
notion of construal (Croft and Cruse 2004; Verhagen 2007). Most famous in this respect
is the work of the founder of Cognitive Grammar (Langacker 1987, 2006, 2017), for
whom the following definition of construal is typical:

I will say that the speaker (or hearer), by choosing appropriate focal ‘settings’ and
structuring the scene in a specific manner, establishes a construal relationship between
himself and the scene so structured. The construal relationship therefore holds between
the conceptualizer of a linguistic predication and the conceptualization that constitutes
this predication.
(Langacker 1987: 128)

This excludes any possibility of describing meaning exhaustively in terms of truth-

conditions that render an assertion true relative to the extra-linguistic scenario it aims
to depict. It is not a (representation of a) ‘state-of-affairs’, a situation or a scene per
se that constitutes linguistic meaning, but a structure such as that shown in Figure 3.1,
which includes a focused onstage region with the profile as the most focal element. The
‘conceptualizer’ is included within the maximal scope of conceptualization, even when
not focused on, but assumed implicitly as a ‘tacit conceptualizing presence’ (Langacker
2006: 19). For example, when a speaker utters (1), apart from the ‘profiled’ clouds,
presented in a dynamic spatial relationship with the horizon, there is an implicit
272 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

conceptualizer (the speaker and/or the addressee, see below) who ‘sees’ this scene from a
certain kind of vantage point.1
(1) The clouds are floating gently/boringly over the horizon.
The implicit presence of the conceptualizer in (1) shows itself in at least two aspects.
Most obviously, the choice of adverb (gently/boringly) specifies the relationship to the
described scene with entirely different emotional valance. More subtly, the mentioned
horizon necessarily presumes the presence of a viewer, with a specific spatially positioned
viewpoint.
Over the years, Langacker and colleagues have proposed many different kinds of
construal ‘operations’, and have endeavoured to group them under a set of ‘dimensions’
(Croft and Cruse 2004; Möttönen 2016a). No agreement on a general taxonomy can be
found, but the following dimensions of construal are common. Starting from the profile
and moving outwards (see Figure 13.1), the dimension of specificity concerns how much
information is presented about the profile. As illustrated in (2), there is increase in specificity
moving from left to right, and if the expressions are co-referential (denoting the former
president of the USA), they follow an implicational hierarchy of sorts. The examples under
(3) display the dimensions prominence and focusing. Such reversals between the trajector
(what is focal) and the landmark (what serves as reference point) describe the same scene,
but focus on the tree or the bench, respectively. The examples in (4) involve such a reversal
as well, but also the perspective of the conceptualizer, since unlike beside, the predicates
left/right are inherently relative to a viewpoint (Zlatev 1997; Levinson 2003).
(2) the man < the old man < the old man with the fake blond hair < the old man
with the fake hair and the red baseball hat
(3a) The tree is beside the bench.
(3b) The bench is beside the tree.
(4a) The tree is to the right of the bench.
(4b) The bench is to the left of the tree.
A more subtle case of involvement of the perspective of the conceptualizer is shown
in (5). In (5a) the described movement of the balloon takes place ‘onstage’ and is hence

FIGURE 13.1 Meaning as construal, with some of the relevant structures (adapted from
Langacker 2006).
COGNITIVE LINGUISTICS AND SEMIOTICS 273

a case of what Langacker calls objective construal. In contrast, it is not the profiled path
that is moving in (5b) but the ‘scanning’ of it, and given that this takes place ‘offstage’, the
meaning of the sentence involves a subjective construal.
(5a) The balloon is slowly rising in the sky.
(5b) The path slowly rises toward the summit.
As here only suggested due to space limits, but as elaborated by Verhagen (2007) and
Möttönen (2016a, 2016b, 2019) the concept of construal has considerable descriptive
and explanatory potential, under the condition that the two typical fallacies of CL
(subjectivism and conflating perception with signification) are redressed. Typically,
Langacker associates construal with individual-psychological ‘imagery’ (e.g. Langacker
1990). Even more problematic, in line with (current) neuro-reductionist trends, it is
assumed that such ‘structures described in qualitative terms ultimately consist in neural
processing’ (Langacker 2016: 473), thus placing their essence ultimately beyond
consciousness.
In an extensive meta-analysis and elaboration, Möttönen (2016a) shows that
such claims are fundamentally at odds with phenomenology, and consequently with
cognitive semiotics, given that the latter is strongly influenced by this philosophical
tradition (Sonesson 2007; Zlatev 2015). At the same time, Möttönen (2016a) argues
that a subjectivist (and even more so physicalist) interpretation of construal is not
required for Cognitive Grammar. Its emphasis on being a ‘usage-based’ theory in fact
implies considering construal as an intersubjective phenomenon. To accommodate
this, however, it is imperative to distinguish between (a) acts of perceptual
intentionality and other kinds experiential motivations, that may underlie but not do
not determine the use and interpretation of linguistic utterances such as those in (3)
and (4), and (b) such utterances themselves (Blomberg and Zlatev 2014; Zlatev and
Blomberg 2016).
The most obvious reason why trajector/landmark reversals and other ‘operations’
on an individual level cannot be determinative of meaning is the fact that language
always takes place in one social context or another. For example, one’s choice of which
referential expression to use in (2) will not so much depend on how specifically one needs
to characterize Donald Trump for one’s own sake, but for the sake of communication
(Möttönen 2016b). Verhagen (2007) correctly emphasizes that the ‘conceptualizer’
should not be equated with a single mind, but minimally include both the speaker and
the hearer. Hence, construal in language cannot be a private, purely mental process, but
needs to be seen as a pragmatic, communicative process, supervening on the experiences
of individual speakers and hearers. This is the transition of construal as non-signitive
intentional acts to construal as signitive, ‘meaning-intending’ acts in the sense of Husserl
([1900] 1970).
The notion of construal can be extended even further with the help of the
phenomenological concept of sedimentation, which denotes the process where, over
historical time, through numerous individual acts of meaning-making, relatively stable
intersubjectively shared structures emerge. As argued by Zlatev (2016), linguistic
meanings (senses) can thus be understood as ‘conventionalized, socially shared construals
of their intended referential objects’ (2016: 563). To take a famous example from the
history of semantics, the expressions Morning Star and Evening Star are not synonymous
even though they are co-extensional, as the contextual information pertaining to when
274 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

the referent is observed has become sedimented into the conventional semantics of the
respective expression. As described by (Möttönen 2016a: 161–2):
In this process, the multiple, intersubjectively-shared experiential designata, and the
particular ways in which they are experienced, come to define the content of the
expression. The designata are not only given as such, but also in their perspectival
appearance to subjects.
We return to the notion of sedimentation in the section Metaphor, but may for
now summarize that the CL concept of construal is indeed productive for (cognitive)
semiotics, under the condition that three kinds are both distinguished and interrelated:
(a) individual people (may) construe1 situations variously in conscious experience; (b)
speakers and hearers construe2 situations and referents in social interaction, motivated
but not determined by (a); (c) words and other linguistic expressions construe3 certain
situations and referents, to the degree that they are sedimentations of (b). These three
senses of the verb construe exist in everyday English, and this could be a reason, but
hardly an excuse, for commonly confusing them within CL.

ICONICITY
The concept of iconicity is possibly the most direct ‘borrowing’ in (cognitive) linguistics
from semiotics, and yet this is often not even acknowledged, along with terminological
slippage where iconicity becomes conflated with any form of ‘non-arbitrariness’, as noted
among others by Devylder (2018). CL overview books (e.g. Croft and Cruse 2004;
Dancygier 2017) tend to mention iconicity only in passing, though there is an informative
chapter on the topic in the first comprehensive CL handbook (van Langendonck
2007). On a general level, theorists of Cognitive Grammar (e.g. Langacker 2008: 209)
occasionally refer to iconicity as ‘correspondences’ between grammatical patterns and
their conceptual meanings. But as we argue below, this take on iconicity easily leads to
the two familiar problems of subjectivism and conflation of experiential motivations with
linguistic motivations.
It was Jakobson (1965) who introduced the Peircian concept of icon(icity) to linguistics,
albeit in a simplified way, in his ‘prelude to a theory of iconicity’ (De Cuypere 2008: 55),
where he questioned ‘the Saussurean dogma’ of the arbitrariness of the linguistic sign.2
In accordance with Peircian semiotics, Jakobson described iconicity as a resemblance-
based interpretative principle, which can only be understood in contradistinction with
two other complementary principles (or ‘semiotic grounds’): indexicality, based on
spatio-temporal contiguity, and symbolicity, based on shared agreement, or convention
(see Ahlner and Zlatev 2010). Further, Jakobson made it clear that these three semiotic
grounds are not mutually exclusive but can coexist (in different proportions) in the same
sign. For example, the English demonstratives this/that are (a) indexical as the requite
contiguity with a referent in the context, (b) symbolic as they are conventions in English
(as opposed to, e.g., the corresponding conventions tozi/onzi in Bulgarian), and (c)
arguably also iconic, as the vowel contrast [i/a] corresponds to the opposition [short/long]
distance from the speaker, and this correspondence has been shown to be non-random in
a large sample of languages (Johansson and Zlatev 2013).
But is such a correspondence between (elements of) the expression (‘form’) and
(aspects of) the meaning of the expression indeed a matter of resemblance, and thus
COGNITIVE LINGUISTICS AND SEMIOTICS 275

iconicity? A positive answer can be given, but only under two conditions. First, it is not
a matter of imagistic iconicity, where the expression is like a photograph or a direct
imitation of the referent, but of diagrammatic iconicity, where the resemblance takes the
form of an analogy: the relation A-B (in expression) corresponds to the relation A´-B´ in
experience. In the example of English demonstratives, the contrast between a close-front
vowel (with its higher acoustic frequency) and open-back vowel (with its lower acoustic
frequency) corresponds to the contrast between short distance and long distance. Second,
it is imperative that such ‘isomorphism’ does not only hold between ‘form’ and ‘content’
within language, but to extra-linguistic experience, if the similarity in question is to be
intersubjectively verifiable. This is precisely the take on linguistic iconicity as ‘structural
similarity between linguistic and non-linguistic entities’ adopted by Itkonen (2005: 101).
Iconicity concerns all levels of language, from the sounds of individual words to
grammar and discourse. On the lowest, there is phonic iconicity, commonly known as
(part of) the phenomenon of ‘sound-symbolism’, for example with English expressions
for actions and events such as (6).
(6) bang, bong, boom, buzz, crack, flip, flop, splash, squeak, zip . . .
Such onomatopoetic words are often mentioned as superficial counter-examples to
arbitrariness, only to be dismissed due to their supposed marginal numbers and role in
language; even van Langendonck (2007) writes that ‘compared to normal vocabulary,
onomatopoeia remains a marginal phenomenon in natural language’ (2007: 402). The
last decade has made it clear that such judgements are mistaken, in part due to the fact
that they are based on a ‘skewed database’ (Ahlner and Zlatev 2010: 305) of languages.
As pointed out by Perniss, Thompson and Vigliocco 2010: 3):
When we move outside the Indo-European language family, however, we find
that iconic mappings are prevalent and are used to express sensory experiences of
all kinds. Languages for which a large iconic, or sound-symbolic, lexicon has been
reported include virtually all sub-Saharan African languages . . .,some of the Australian
Aboriginal languages . . ., Japanese, Korean, Southeast Asian languages . . ., indigenous
languages of South America . . ., and Balto-Finnic languages.
Another reason for traditionally downplaying iconicity on this level is that
onomatopoeia is based on both (a) unimodal (sound-to-sound) resemblance and (b)
relatively imagistic (imitative) iconicity, while the massive amounts of iconicity on the
‘phonic level’ that are recognized in recent studies are both mostly (a´) cross-modal
(e.g. sound-to-shape) and (b´) diagrammatic as defined above (Dingemanse 2012;
Blasi et al. 2016; Jääskeläinen 2016). Even though such iconicity is subtle, and far
from obvious, it has been shown to offer advantages in terms of language acquisition
(Imai et al. 2012) and social transmission (Erben Johansson, Carr, and Kirby 2020),
implying that it is indeed perceived by speakers, even if tacitly. Jakobson (1965: 33)
recognized the ‘latent and virtual character’ of iconicity on this level, but claimed that
‘in syntax and morphology (both inflection and derivation) the intrinsic, diagrammatic
correspondence between the signans [i.e., expression] and signatum [i.e., content] is
patent and obligatory’. This statement appears to echo the way iconicity is understood
in Cognitive Grammar, predating it with some thirty years. However, as we illustrate
below, Jakobson’s ‘signatum’ does not correspond to Langacker’s notion of meaning as
mental imagery, or of construal as something purely subjective (section Construal) and
thus does not encounter the same problems.
276 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

A well-known example of iconicity on the level of morphology concerns the linguistic

category number as shown in (7); if singular and plural have different manifestations, it is
the plural form that (nearly always) has an extra morpheme, analogously to the ‘more’ in
experience. This type of linguistic iconicity is commonly known as ‘iconicity of quantity’.
(7a) boy vs. boy-s (English)
(7b) pojke vs. pojk-ar (Swedish)
(7c) momce vs. momce-ta (Bulgarian)
(7d) poika vs. poja-t (Finnish)
Another kind of iconicity on this level is the ‘iconicity of distance’, where ‘the distance
between expressions corresponds to the conceptual distance between the ideas they
represent’ (Haiman 1983: 782). In an extensive discussion of how such diagrammatic
iconicity can explain asymmetries in possessive constructions in Paamese (and other
languages), Devylder (2018) presents oppositions such as in (8), analysed as in Figure 3.2.
In a debate with Haspelmath (2008), Devylder points out that usage frequency (‘economy’)
cannot explain the distribution of possessive constructions, and that it is experienced
distance – in (8) pertaining to one’s body but in other cases pertaining to social and
emotional relations – that maps onto ‘linguistic distance’, with the indirect possessive
construction in (8b) also being used, for example, for mother-in-laws, while the direct
one in (8a) is used for mothers, etc.
(8a) vatu-k (Paamese)
head-1SG
(8b) maleles ona-k (Paamese)
lungs POSS-1SG
Moving on to the sentence level, iconicity of distance applies to many linguistic
constructions such as (9), where ‘absolute objective qualities like wooden and white are
closest to the noun. Relative objective qualities (size, e.g., little) are expressed further
away from the noun, and subjective qualities like nice are repressed still further. Furthest
away . . . in the determiner, since is serves only to pick out the referent’ (van Langendonck
2007: 409). This is a pattern that is potentially universal, with exceptions that can be
accounted of.
(9) those three nice little white wooden dolls
Again, to appreciate such iconicity, distance on the semantic side should be
understood as experiential rather than ‘physical’ or so. This is even clearer in (10),
where the contrast is between direct perception vs indirect perception (or evidence).
The examples in (10a) and (10b) could denote the same situation, but with different
construal, the first using a closer, and the latter a more distanced viewpoint. But (10b)
can also have the interpretation that the speaker did not at all hear the song him/herself,
but was given an indirect report of the fact. On the expression side, this difference
is mirrored by (10a) being a single clause construction, and (10b) consisting of two
clauses, with a subordinate finite verb.
(10a) I heard him sing/singing.
(10b) I heard that he sang.
COGNITIVE LINGUISTICS AND SEMIOTICS 277

FIGURE 13.2 Diagrammatic iconicity between embodied and linguistic distance in Paamese
possessive (from Devylder 2018, with permission from the author).

Moving on to the discourse level, and ‘iconicity of sequence’, the famous (11)
reportedly uttered by Caesar, implies both temporal and causal order of the denoted
events being mirrored by the order of the verb/clauses). In (12), taken from a well-known
Bulgarian poem by Hristo Botev, it is even clearer that the order in events is not causal
but experiential.
(11) Veni, vidi, vici (Latin)
­ ‘I came, I saw, I conquered’.
(12) Gora zašumi, vyatar povee . . . (Bulgarian)
‘Forest starts to sound, wind to blow’
We can generalize that in all these cases of linguistic iconicity, the resemblance
(mapping, correspondence, analogy, etc.) in question is between expression and a more
278 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

or less schematic experiential structure, and in most cases, this mapping is relational, so
that ‘more/less of’, ‘close to/away from’, ‘before/after’, etc., make sense only when two or
more linguistic and experiential elements form an analogical diagram as in Figure 13.2.
This serves to show that iconicity in language is ‘cognitive’ in the sense of requiring
conscious speakers/signers who can appreciate the subtle and relational resemblances
between linguistic and experiential structures. But it does not imply ‘cognitive’ in the
sense of subjective (‘in the head’) meaning.
Unfortunately, the latter is how iconicity is typically understood in CL, as illustrated by
Wilcox (2004). The author first assumes that ‘the iconic relation is between construals of
real-world scenes and construals of forms’ (2004: 123, our emphasis), with a subjectivist
understanding of construal (see section Construal). Even more problematically Wilcox
then defines cognitive iconicity as ‘a distance relation between the phonological and
semantic poles of symbolic structures’ (2004: 122): the ‘closer’ this distance, the more
‘cognitively iconic’ the construction should be. Thus, Wilcox ends up defining iconicity
in terms of invisible (internal, mental, conceptual) contiguity, i.e. indexicality, and even
more, does not provide criteria of how the latter could be independently established,
making the proposal quite circular.
What leads Wilcox to this problematic idea is the valid need to account for substantial
degrees of iconicity in signed languages like American Sign Language (ASL), despite the
fact that single signs are very often less iconic than corresponding gestures. He illustrates
this with the ASL sign SLOW, which is produced with a slow movement of one hand
along the other top of the other hand. But VERY SLOW is produced with ‘an initial
hold of the sign’s movement followed by sudden, rapid release’ (Wilcox 2004: 123),
with the result that VERY SLOW is produced faster than SLOW. This has been used
by some analysts to argue that signed languages are in fact arbitrary. Wilcox correctly
questions this claim, since the sign VERY also modifies all signs denoting qualities (e.g.
FAST, SMART) in essentially the same way, which makes the expressions of all such
phrases diagrammatically iconic: qualities without VERY will be expressed ‘normally’,
and those with VERY will have this extra very ‘sudden, rapid release’. But this is clearly
an intersubjective, sedimented sense in ASL, and there is hardly any need to evoke
any special kind of ‘cognitive iconicity’ (based on internal distance) to acknowledge its
iconic character.
The attitude displayed in the ‘cognitive iconicity’ suggested by Wilcox is symptomatic
for CL at large. As pointed out earlier, Langacker and others regard iconicity as a
generalized form of cognitive ‘isomorphism’ (van Langendonck 2007: 400), understood
as a relationship between two mental entities: internal phonological (or signed) form
and internal meaning. But if the claim of resemblance between these ‘poles’ is to be
non-vacuous, there must be non-linguistic criteria for establishing the structure of the
(experiential) meaning, which is why Itkonen (2005), among others, insists that what
iconic linguistic expressions correspond to are non-linguistic entities. In other words,
CL theorists once again commit the two fallacies of (a) meaning-subjectivism and (b)
confusing experiential motivations with linguistic semantics. In contrast, by making the
latter distinction clear, and by pointing out that iconicity holds not between ‘form’ and
‘meaning’, but between linguistic expressions and experiences of the kind that motivate
(but do not constitute) the intersubjective conventional construals of the expressions
in question, cognitive semiotic analyses are able to avoid both pitfalls (Blomberg and
Zlatev 2014; Zlatev and Blomberg 2016; Devylder 2018). Further, such a perspective
can easily be extended to metaphor, where it is even clearer that resemblance in question
COGNITIVE LINGUISTICS AND SEMIOTICS 279

is not between ‘form’ and ‘meaning’, but between different interpretations of the same
(linguistic) expression, as we explain in the following section.

METAPHOR
CL is famous for its research on metaphor, and in particular for the theory proposed
in the well-known book Metaphors We Live by (Lakoff and Johnson 1980), which
became eventually known as Conceptual Metaphor Theory (CMT, Johnson 1987,
2010; Grady 1997, 2007; Lakoff and Johnson 1999; Kövecses 2000; Gibbs 2017).
This theory redefined metaphor as a scheme of mental correspondences (a ‘mapping’)
from more concrete (e.g. SPACE, MOTION) to more abstract (e.g. TIME, EMOTION)
conceptual domains. As stated clearly by Grady (2007) ‘within Cognitive Linguistics the
term metaphor is understood to a pattern of conceptual association, rather than to an
individual metaphorical usage, or a linguistic convention’ (2007: 188). Metaphorical
expressions in language (or any other semiotic system) like (14) and (15) thus become
surface realizations of underlying ‘conceptual associations’.
(14) Let us forget the past, and look ahead!
(15) He fell into a deep depression.
Lakoff and Johnson (2003: 256) re-affirm the emphasis on ‘a stable, conventional
system of primary metaphors that tend to remain in place indefinitely within the
conceptual system and that are independent of language’. But one may reasonably wonder
how a ‘conventional system’ could exist in the mind/brain of individual speakers, given
that conventions are essentially social and normative phenomena (Itkonen 2008), and
how it could remain there ‘indefinitely’, given that neural connectivity is under constant
change (Edelman 1992). Furthermore, the definition of ‘a cross-domain mapping of
structure from a source domain to a target domain, where the two domains are regarded
as different in kind’ (Johnson 2010: 407) hinges on being able to individuate ‘conceptual
domains’, which is anything but a trivial matter (Croft and Cruse 2004). For example,
are the above-mentioned MOTION and EMOTION truly distinct domains (Zlatev and
Blomberg 2016)? The spectre of subjectivism reappears once again.
As can be expected, many from both within and outside CL have reacted to such
problems. Discourse metaphor theory (Cameron and Deignan 2006; Zinken 2007), has
brought back the attention to specific metaphorical expressions like drive X crazy, or bleed
(X) to death and how they become conventionalized in a language community as a result
of repeated use in social interactions. Even more radically, some theorists have focused
on the role of situated context and proposed that ‘metaphors come to exist only in the
moment’ (Müller 2016: 50). Further, given the difficulties of individuating ‘conceptual’
metaphor, and the need for intersubjectively valid criteria to decide what is metaphor
has prompted many to propose ‘procedures’ for identifying metaphors in text (Pragglejaz
2007). The debates surrounding CMT continue to rage and have even been characterized
as ‘metaphor wars’ (Gibbs 2017). We cannot hope to do them proper justice here, but
only show how cognitive semiotics could possibly resolve some of the debated issues.
From the perspective of semiotics, what is lacking in the CMT approach is first of all
an acknowledgement that metaphors are a particular kind of sign (process), and secondly,
that they are iconic signs. As seen above, proponents of CMT characterize metaphors
and mental ‘associations’, but as pointed out in the section Iconicity, associations are
280 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

at best indexical, not iconic processes. On the other hand, and as stated clearly by van
Langendonck (2007: 398): ‘A metaphor, in Peirce’s view, brings out the representative
character of a sign by representing a parallelism with something else.’3 Unlike images
and diagrams, in metaphors the resemblance in question is not between a linguistic (or
other) expression and some extra-linguistic object, related to its meaning (though not
the meaning itself, as argued in the section Iconicity) but between two interpretations
of the expression in question. A recent cognitive semiotic theory that capitalizes upon
this insight is the Motivation & Sedimentation Model (MSM; Stampoulidis, Bolognesi
and Zlatev 2019; Devylder and Zlatev 2020), which offers the following definition of
metaphor:

A metaphor is a (simple or complex) sign in a given sign system (or a combination of

systems) with
(a) at least two different potential interpretations (giving rise to tension),
(b) standing in an iconic relationship with each other, where
(c) one interpretation is more relevant in the communicative context, and
(d) can be understood in part by comparison with the less relevant interpretation.
(Stampoulidis 2021: 54)

To illustrate, we may consider a pictorial metaphor from the corpus of Greek street art
of Stampoulidis et al. (2019), shown in Figure 13.3. This may be interpreted as GREECE
IS SULLIED, provided the conventional metonymy (association) between national flag
and the country it represents. This pictorial expression/sign is (a) ambiguous between at
least two conflicting interpretations: the Greek flag and a roll of toilet paper, (b) standing
in a relatively imagistic iconic relation to one another. The question of relevance depends
on the context in which the metaphor is created and interpreted. Given that the street
artwork is a commentary on the political situation of Greece, (c) it is reasonable that
Greece is the topic (target) of the metaphor, which (d) is interpreted via the notion of
toilet paper, and its pejorative associations.
Concerning linguistic metaphors like that in (15) we can first establish that the
expression (type) fall into X has at least two different interpretation by means of a
‘polysemy test’ showing that a sentence like ‘A fell into X but did not (really) fall into X’
is not self-contradictory (Torstensson 2019; Geeraerts 2010). It is the figurative kind of
falling that is relevant in the context, and it is in part understood through comparison with
actual falling, given that there is an experiential similarity in their event structure: initial
stability, loss of stability, unpredictable but likely negative consequences, etc. A number of
studies have shown that this theoretical definition of metaphor leads to intersubjectively
verifiable operational definitions for different semiotic systems (Stampoulidis et al. 2019;
Torstensson 2019; Devylder and Zlatev 2020; Moskaluk 2020).
MSM is based on phenomenology (e.g. Merleau-Ponty 1962), and integral linguistics
(Coseriu 1985, 2000), which itself has roots in both phenomenology and (structural)
semiotics. In brief, MSM (see Figure 13.4) grounds meaning in the Embodied level, which
consists of non-linguistic cognitive and experiential processes like cross-modal perceptual
experience (Abram 1996), bodily mimesis (Donald 2001) and analogy making (Itkonen
2005). On the opposite side is the Situated level, which is that of actual social interaction,
spontaneous language use and artistic improvisation. This is where all particular linguistic
and extra-linguistic meaning-making take place, and where diversity and creativity
abound. Linking the two is the Sedimented level of historically derived, relatively stable
COGNITIVE LINGUISTICS AND SEMIOTICS 281

FIGURE 13.3 Greece Flag ≡ Toilet paper, leading to the metaphorical interpretation GREECE
IS SULLIED, given appropriate knowledge on the Sedimented and Situated levels (Stampoulidis
et al. 2019). Creator: Unknown. Photography: Georgios Stampoulidis © July 2015.
(Reproduced with permission from the author and copyright owner.)

linguistic and other social norms (Itkonen 2008; Zlatev and Blomberg 2019). It includes
shared beliefs and communicative and other communal conventions.
The most basic relation that links the levels is that of motivation, which grounds,
but does not determine, the meaning of the expression that is motivated. The second
relation, sedimentation (see section Construal), goes in the opposite direction, stabilizing
linguistic and other semiotic expressions, as well as other cultural phenomena, as a
repository of common knowledge for the community. These can now co-motivate
expressions on the Situated level, which are thus doubly motivated – not only by the
visceral experiences and non-linguistic thought processes on the Embodied level, but also
by the norms of the Sedimented level. The degree of metaphoricity of an expression on
the Situated level is, among other factors, proportional to the degree to which its use is
motivated by the Embodied level, and inversely proportional to the degree to which it is
282 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

motivated by the Sedimented level (Moskaluk 2020). Examples (14) and (15) would
thus qualify as relatively low on this scale, while the pictorial metaphor in Figure 3.4,
as well as (authentic) examples like (16), taken from a go-along interview with a Greek
street artist (Stampoulidis 2021), clearly display active analogy making, and hence the
predominant role of the Embodied level.
(16) You may think of people going to the gym, working out really hard to show off
their abs in the summer. Similarly, here, the people of art do the same. We show our
artistic abs. So, in essence, street art making is the same as going to the gym. And your
artworks are your abs. (Stampoulidis 2021, Paper 3: 31)
The cognitive semiotic MSM accommodates some of the insights from CL, including
the role of ‘mappings’, especially once it is acknowledged that this is identical with
the classical notion of analogy (Itkonen 2005), which as pointed out in the section
Iconicity also overlaps with that of diagrammatic iconicity. Further, the emphasis
on grounding language and other sign systems in bodily experience is shared by the
two approaches. But while MSM regards the mappings/diagrams and other cognitive
processes as motivations for sign use, CMT follows the usual pattern of CL of
conflating the two. By regarding metaphorical expressions and conventions as more or
less epiphenomenal reflections of the crucial cognitive infrastructure, CMT also cuts
off the link to social interaction and becomes subjectivist. In line with phenomenology,
MSM, on the other hand, emphasizes that even the Embodied level is not ‘in the head’,
but rather structured by embodied intersubjectivity and thus not solipsist (Zlatev and
Blomberg 2016; Möttönen 2016a, 2019). Further, MSM integrates elements of some
of the metaphor theories that have been proposed as alternatives to CMT mentioned
earlier. The Sedimented level accommodates shared socio-cultural knowledge and
conventions as emphasized by proponents of discourse metaphors, and the Situated

FIGURE 13.4 The Motivation & Sedimentation Model (MSM) with the upward arrows
signifying motivation, and downward arrow sedimentation (adapted from Stampoulidis et al.
2019; Devylder and Zlatev 2020).
COGNITIVE LINGUISTICS AND SEMIOTICS 283

level is indeed the level where metaphors ‘dynamically emerge, are elaborated, and
are selectively activated over the course of a conversation’ (Kolter et al. 2012: 221),
as argued by proponents of dynamic-enactive accounts (Müller and Kappelhoff 2018).
However, these could neither emerge nor be elaborated without the other two levels,
the Situated and the Embodied.4
In sum, metaphor continues to be a very hot topic, to use a highly sedimented but still
‘alive’ metaphor, and CL should be given credit for making it so. Its dominant theory
CMT, however, has been found increasingly problematic theoretically, empirically and
methodologically. And the main reason for this is once again the falacy of subjectivism:
placing metaphors in the (individual) head, rather than in communication, culture and
embodied intersubjectivity. We reviewed MSM as a contribution from cognitive semiotics
that clearly distinguishes between cognitive motivations and metaphor use, and links
them through a level of shared conventions, thus offering the blueprint for a peace treaty
in the ‘metaphor wars’.

MULTIMODALITY AND POLYSEMIOSIS

Somewhat later than other fields such as social semiotics (Kress 2009) and gesture and
sign language studies (Vigliocco, Perniss and Vinson 2014), CL has recently embraced
the notion of multimodality, with a recent handbook (Dancygier 2017) devoting a whole
part to ‘Language, Body and Multimodal Communication’. Unfortunately, as pointed
out by Devylder (2019: 149) in the review of this book, the term ‘is not defined and
approached cohesively within CL’. Devylder illustrates this claim clearly, with references
to particular chapters in the handbook:

Defining something as multimodal naturally implies that it involves at least two

modalities, or modes. That is about as far as the consensus goes in CL, and that is not
very far because we already do not know if we are talking about modes or modalities in
‘multimodality’, and what both terms mean. For Vandelanotte (p. 158), a modality is
a sensory channel (e.g., ‘the visual modality’), for Sullivan (p. 389–91) in line with the
CL tradition that uses the term ‘visual metaphors’ in contrast to ‘linguistic metaphors’,
modality has to mean that pictures and written text are two modalities (the combination
of which is thus ‘multimodal’), for Feyaerts et al. (p. 135), the term extends to just
about any aspect of a face-to face interaction. [. . . Thus,] when one reads a CL paper
on multimodality, one can expect the term to either mean: the combination of text and
image, the combination of gesture and speech, the combination of vision and hearing,
or a combination of all of the above and more.
(Devylder 2019: 149–50)

The problem is in part, as suggested above, that the term ‘multimodality’ has sprung
forth in different fields, from cognitive psychology to conversation analysis, and many of
these differ in terms of ontological and methodological commitments. While ‘modality’
commonly refers to a ‘channel’ of perception (in psychology), ‘mode’ is a very general
term, used for anything that could be meaningful in social interaction. This, however,
is problematic, as ‘there is no theoretical limit to the number of modes that may be
recognized in various socio-cultural contexts, and this leads to an abundance of modes
that are difficult to compare’ (Green 2014: 9–10).
284 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

At least some CL theorists recognize the terminological challenges with the term. For
example, Vandelanotte (2017: 161) distinguishes between ‘multimodality in the strict
sense’ for different sensory (and productive) channels and ‘in the loose sense’ for different
‘semiotic resources’ like text and image. Similarly, a prominent analyst of ‘multimodal
metaphor’ acknowledges that ‘what constitutes a mode is a much-debated and hitherto
unresolved issue’ (Forceville 2017: 27).
The problem, however, is more than terminological and reveals a deeper concern.
For example, Forceville continues the citation from above by stating: ‘I distinguish the
following modes: written language, spoken language, visuals, music, non-verbal sound,
gestures, olfaction, taste, and touch.’ This is indeed far from ‘cohesive’, given that it
includes (a) different semiotic systems like language, gesture and music, (b) different
sub-systems of language: spoken and written, and (c) different sensory modalities. It
is hard not to see the two CL fallacies of subjectivism and conflation of semiotic and
perceptual systems implicated here as well. First, purely perceptual (and thus non-
signitive) experiences are conflated with sign and signal use. Second, this is facilitated by
the general ‘cognitive’ bias to treat everything as essentially mental, or in other words,
subjectivism.
Recent work in cognitive semiotics (Louhema et al. 2019; Zlatev 2019) has once
again attempted to redress the problem by distinguishing types of signification from
types of experience, as a preliminary to investigating their interrelations. Such authors
re-introduce a familiar term from semiotics: semiotic system, but make it clear that this
should not be understood as a generalization based on language, which was arguably
the case in structuralism (Saussure [1916] 1960). First, semiotic systems can consist of
either signs, where expressions clearly denote (intentional) objects, or signals, which are
only associated with such objects (Zlatev, Zywiczynski and Wacewicz 2020). Human
language is a paradigmatic example of a sign system, while various animal alarm calls,
as well as spontaneous human facial expressions, are signal systems. Sign systems may
be defined as combinations of particular kinds of signs, along with their properties and
relations. Table 13.1 shows what are arguably the most fundamental sign systems in
human communication, with some of their central properties.
The semiotic system of language differs, depending on whether it is realized as speech,
writing or signing. In the case of writing, it is characterized by a high degree of permanence
of its expressions, at least in prototypical written media. In the case of the signed languages
of the deaf, with respect to production and perception, language adopts the features of
Gestures. Language has so-called ‘double articulation’: phonemes or graphemes combine
systematically to form meaningful morphemes, and this is arguably also the case for
some but not all signed languages, in particular those which have emerged more recently
(Sandler 2012). Its semiotic ground is predominantly conventional, even if iconicity and
indexicality are also present (see section Iconicity). Its ‘syntagmatic’ (sequential) relations
are characterized by a high degree of compositionality, where the meaning of a composite
sign is built up (at least in part), from the meanings of its constituent signs, and the rules
for combining these.
In the case of Gesture and Depiction, on the other hand, the predominant grounds are
iconicity and indexicality, even if conventionality is also important. The signs of Gesture
and Depiction can be analysed into phrases and units (Kendon 2004; Green 2014), but
these are not made up of minimal distinctive elements, and hence lack double articulation.
Further, these sign systems have much less systematic manners in arranging sequences of
signs, making it more difficult (but not impossible) to express complex messages such as
COGNITIVE LINGUISTICS AND SEMIOTICS 285

TABLE 13.1 The sign systems of Language, Gesture and Depiction, with some of
their properties (adapted and extended from Zlatev et al. 2020). Capital letters are
used here and in the text for the systems to highlight that the terms are used in a
specific
­ (technical) sense
Properties Sign systems
Language Gesture Depiction
Speech Writing Signing
Production Vocal Material Bodily Bodily Material

Perception Auditory Visual Visual Visual Visual

(Modality) (+Visual) (+Tactile) (+Auditory, (+Tactile)
+ Tactile)
Permanence Low High Intermediate Intermediate High

Double Yes Yes Yes/No No No

articulation
Semiotic Conventional Conventional Conventional Iconic, Iconic >
grounds > Iconic, > Iconic, > Iconic, Indexical > Indexical,
Indexical Indexical Indexical Conventional Conventional
Syntagmatic Compositional Compositional Compositional Linear Possibly
relations linear

narratives. In terms of permanence, Gesture (and Signing) is intermediate to Speech and

Writing, given that one can ‘hold’ gestural expressions in mid-air, when the need for
emphasis or audience attention requires it. The permanence of Depiction is similar to that
of Writing, with differences having to do with the ‘Material’ in which signitive acts are
produced (e.g. stone vs paper vs sand).
Communication through a single semiotic system (either a sign or signal system)
is by definition monosemiotic. But actual human communication, especially in face-
to-face contexts, is as a rule polysemiotic, combining multiple sign systems (as those
in Table 1) and signal systems, like facial expressions and postures. Much recent and
ongoing work in cognitive semiotics is formulated under the banner of polysemiotic
communication or polysemiosis for short. This includes work on intersemiotic
translation, i.e. transferring a message from one semiotic system to another, with or
without change of modality (Louhema et al. 2019), analysing the practice of ‘audio
description’ of films for blind viewers as intersemiotic translation (Diget 2019),
polysemiotic and monosemiotic street art (Stampoulidis et al. 2019), polysemiotic
‘path metaphors’ (Devylder 2020) and the analysis of the role of pantomime for the
evolution of language (Zlatev et al. 2020).
We lack the space to review this work here, but can point out its most relevant feature
for the present context: clearly distinguishing perceptual modalities and their combinations
(i.e. multimodality) and semiotic systems and their combinations (i.e. polysemiosis). The
classification in Table 13.1 allows us to clearly acknowledge that there is no one-to-one
286 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

relation between semiotic system and perceptual modality. Even considering only Speech,
it is typically perceived not only as sound, but visually as mouth movements, as shown by
the famous McGurk effect (Munhall et al. 1996). Various manifestations of Depiction,
such as sculptures and reliefs, afford perception through multiple modalities. From the
other side, vision is typically, or can be, involved in the perception of all sign systems.
Thus, we can see once more that cognitive semiotics helps de-conflate (perceptual)
experience from signification, in a first step, so as to investigate their interactions more
systematically in a second step.

CONCLUSIONS
In this chapter, devoted to the relation between the forty-year-old tradition of Cognitive
Linguistics (CL) and the much older traditions of semiotics, we chose to focus on four
theoretical concepts, which show both similarities and differences between the two
traditions: construal, iconicity, metaphor and multimodality. From the onset, we noted a
tension between the dominant subjectivist conception of meaning in CL, and the quest for
intersubjective structures in communicative and other kinds of acts of meaning making
in semiotics.
The relatively recent discipline of cognitive semiotics (Zlatev 2015; Stampoulidis
2021) combining concepts and methods from these traditions, as well as from
cognitive science (Konderak 2018) and phenomenology (Sonesson 2009), was
shown to be able to offer solutions to dilemmas for each of the four concepts. In
the case of construal, we pointed out the need to distinguish mental processes from
pragmatic interactions and conventional meanings. Concerning iconicity, we argued
that it should not be understood as a mapping between ‘mental form’ and ‘mental
meaning’, but as resemblance between intersubjectively observable expressions, and
independently perceivable (or inferable) non-linguistic events and experiences. Thus,
iconicity can indeed be shown to be prevalent on all levels of language, though not in
the same way nor to the same degree as in Gesture and Depiction, which are distinct
semiotic systems. Similarly, metaphor is from the cognitive semiotic perspective not a
‘mapping’ or ‘simulation’ in the subjective mind, but a complex iconic sign (process),
where the resemblance in question is between different interpretations of the relevant
signs, alongside more or less creative tensions between these, and the surrounding
context. Finally, the fact that CL currently analyses non-linguistic phenomena like
gestures, ‘memes’ and images in advertisements, cartoon, film, etc., is to be applauded,
but we argued that it is problematic to do so in terms of ‘multimodality’. This is so
both due to the ambiguity of the term and due to the latent subjectivism of CL, where
‘modes’ of expression become conflated with ‘modes’ or ‘modalities’ of thought and
perception.
In sum, we have offered a ‘diagnosis’ to CL in terms of the two fallacies of subjectivism
and expression/motivation conflation, along with prescribed medications from the
pharmacy of cognitive semiotics. For the sake of clarity, we have consistently contrasted
CL and cognitive semiotics. But given that the latter constitutes a synthesis of semiotics,
linguistics and cognitive science (as discussed in the introduction above), it is clear that no
sharp distinctions between CL and cognitive semiotics can be drawn. Both of the present
authors present their research also in CL conferences and publications, and we hope
that this chapter will spark further dialogue and collaboration between CL and cognitive
COGNITIVE LINGUISTICS AND SEMIOTICS 287

semiotics, despite their – at first glance – distant perspectives. We may conclude with the
final lines of a publication where the first author was asked to ‘predict’ the development
of CL in the future:
Having reviewed some of these differences, I have argued that methods and concepts
from phenomenology can help re-unite various strands in an epistemologically
pluralistic, truly experiential linguistics. My prediction would be that in the coming
decades, this would merge with broader fields that likewise adopt a phenomenological
perspective in the understanding of mind and meaning, such as enactive cognitive
science (Thompson 2007) and cognitive semiotics.
(Zlatev 2016: 570)

ACKNOWLEDGEMENTS
We wish to thank Johan Blomberg, Simon Devylder and Georgios Stampoulidis, as well as
the editor Jamin Pelkey, for many helpful comments on previous versions of this chapter.

NOTES
1 Reflected in Figure 13.1 are also the roles of trajector (tr) and landmark (lm), which the clouds
and the horizon respectively assume in (1). These notions are meant to capture the semantic
asymmetry internal to the profile, with the (grammatical) subject by default coinciding with
the trajector.
2 Jakobson can be said to have ‘abridged’ Peirce for a linguistic audience, and may have
involuntarily contributed to underestimating the concept of iconicity in linguistics. Nöth
(1999, 2008) can be consulted for a more elaborate application of Peircean ideas to language.
Nevertheless, we wish to acknowledge Jakobson (1965) both for its temporal primacy and
for the inspirational character of this short text, which fifty-five years after publication still
surpasses the way iconicity is dealt with in much current (cognitive) linguistic literature.
3 Peirce’s original definition (which is famously open to different interpretations) states
that metaphors are types of hypoicon ‘which represent the representative character of a
representamen by representing a parallelism in something else’ (Peirce c.1902: CP 2.277).
4 Somewhat related ideas within CL, with an explicitly diachronic (historical) focus, on how
linguistic structures are motivated by but not determined by experience have been proposed
by Heine (1997) and more recently by Schmid (2020). Some of these ideas have also been
fruitfully integrated within cognitive semiotics (e.g. Pelkey 2017, 2018).

REFERENCES
Abram, D. (1996), The Spell of the Sensuous: Perception and Language in a More-than-human
World, New York: Vintage.
Ahlner, F. and J. Zlatev (2010), ‘Cross-modal Iconicity: A Cognitive Semiotic Approach to
Sound Symbolism’, Sign System Studies, 38: 298–348.
Blasi, D. E., S. Wichmann, H. Hammarström, P. F. Stadler and M. H. Christiansen (2016),
‘Sound–meaning Association Biases Evidenced across Thousands of Languages’, Proceedings
of the National Academy of Sciences, 113 (39): 10818–23.
288 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Blomberg, J. and J. Zlatev (2014), ‘Actual and Non-actual Motion: Why Experientialist
Semantics Needs Phenomenology (and vice versa)’, Phenomenology and the Cognitive
Sciences, 13 (3): 395–418.
Brandt, L. (2013), The Communicative Mind: A Linguistic Exploration of Conceptual
Integration and Meaning Construction, Newcastle: Cambridge Scholars.
Brandt, P. A. (2004), Spaces, Domains, and Meaning: Essays in Cognitive Semiotics (European
Semiotics Series 4), New York: Peter Lang.
Cameron, L. and A. Deignan (2006), ‘The Emergence of Metaphor in Discourse’, Applied
Linguistics, 27 (4): 671–90.
Chomsky, N. (1990), ‘Language and the Problem of Knowledge’, in A. P. Martinich (ed.), The
Philosophy of Language, 675–93, Oxford: Oxford University Press.
Coseriu, E. (1985), ‘Linguistic Competence: What Is It Really?’ The Modern Language Review,
80 (4): xxv–xxxv.
Coseriu, E. (2000), ‘The Principles of Linguistics as a Cultural Science’, Transylvanian Review
(Cluj), 9 (1): 108–15.
Croft, W. and D. A. Cruse (2004), Cognitive Linguistics, Cambridge: Cambridge University
Press.
Daddesio, T. C. (1995), On Minds and Symbols: The Relevance of Cognitive Science for
Semiotics, Berlin: Walter de Gruyter.
Dancygier, B. (2017), The Cambridge Handbook of Cognitive Linguistics, Cambridge:
Cambridge University Press.
De Cuypere, L. (2008), Limiting the Iconic: From the Metatheoretical Foundations to the
Creative Possibilities of Iconicity in Language, Amsterdam: John Benjamins.
Deacon, T. (1997), The Symbolic Species, New York: Norton.
Devylder, S. (2018), ‘Diagrammatic Iconicity Explains Assymetries in Possessive Conctructions:
Evidence from Paamese’, Cognitive Linguistics, 29 (2); 313–48.
­Devylder, S. (2019), Review of Barbara Dancygier (ed.), ‘The Cambridge Handbook of
Cognitive Linguistics’ (2017), English Text Construction, 12 (1): 144–54.
Devylder, S. (2020), ‘Polysemiotic Metaphors of Intertribal Exchanges in Paamese Culture’,
Researching and Applying Metaphor (Virtual Conference), Hamar, Norway.
Devylder, S. and J. Zlatev (2020), ‘Cutting and Breaking Metaphors of the Self and the
Motivation and Sedimentation Model’, in A. Baicchi and G. Radden (eds), Figurative
Meaning Construction in Thought and Language, Amsterdam: John Benjamins.
Diget, I. S. K. (2019), Intersemiotic Translation from Film to Audio Description: A Cognitive
Semiotic Approach (MA Thesis), Lund University, Lund.
Dingemanse, M. (2012), ‘Advances in the Cross-linguistic Study of Ideophones’, Language and
Linguistics Compass, 6: 654–72.
Dingemanse, M., M. Perlman and P. Perniss (2020), ‘Construals of Iconicity: Experimental
Approaches to Form–Meaning Resemblances in Language’, Language and Cognition, 12 (1):
1–14.
Donald, M. (2001), A Mind So Rare: The Evolution of Human Consciousness, New York: Norton.
Dowty, D. R., R. Wall and S. Peters (2012), Introduction to Montague Semantics, Voume. 11,
Berlin: Springer Science and Business Media.
Edelman, G. M. (1992), Bright Air, Brilliant Fire: On the Matter of the Mind, Volume 14, New
York: Basic books New York.
Erben Johansson, N., J. W. Carr and S. Kirby (2020), ‘Cultural Evolution Leads to Vocal
Iconicity in an Experimental Iterated Learning Task’, in N. Erben Johansson (ed.), The
Building Blocks of Sound Symbolism, 126–72, Lund: Media Tryck.
COGNITIVE LINGUISTICS AND SEMIOTICS 289

Fauconnier, G. and M. Turner (2002), The Way We Think: Conceptual Blending and the Mind’s
Hidden Complexities, New York: Basic Books.
Forceville, C. (2017), ‘Visual and Multimodal Metaphor in Advertising’, Styles of
Communication, 9 (2): 26–41.
Gärdenfors, P. (1999), ‘Some Tenets of Cognitive Semantics’, in J. Allwood, P. Gärdenfors,
A. Jucker, J. Mey, H. Parret and J. Verschueren (eds), Cognitive Semantics: Meaning and
Cognition, 19–36, Amsterdam: John Benjamins.
Geeraerts, D. (2010), Theories of Lexical Semantics, Oxford: Oxford University Press
Geeraerts, D. and H. Cuyckens (2007), The Oxford Handbook in Cognitive Linguistics, Oxford:
Oxford University Press.
Gibbs, R. W. (2006), Embodiment and Cognitive Science, Cambridge: Cambridge University
Press.
Gibbs, R. W. (2017), Metaphor Wars, Cambridge: Cambridge University Press.
Giraldo, V. (2019), ‘Referential Iconicity in Music and Speech’, Public Journal of Semiotics, 9
(1): 41–77.
Grady, J. E. (1997), Foundations of Meaning: Primary Metaphors and Primary Scenes (PhD
Dissertation), University of California, Berkeley.
Grady, J. E. (2007), ‘Metaphor’, in D. Geeraerts and H. Cuyckens (eds), Handbook of
Cognitive Linguistics, Oxford: Oxford University Press.
Green, J. (2014), Drawn from the Ground: Sound, Sign and Inscription in Central Australian
Sand Stories, Cambridge: Cambridge University Press.
Haiman, J. (1983), ‘Iconic and Economic Motivation’, Language, 59 (4): 781–819.
Haspelmath, M. (2008), ‘Frequency vs Iconicity in Explaining Grammatical Asymmetries’,
Cognitive Linguistics, 19 (1): 1–33.
Heine, B. (1997), Cognitive Foundations of Grammar. Oxford: Oxford University Press.
Husserl, E. ([1900] 1970), Logical Investigations, New York: Humanities Press.
­Imai, M., M. Asano, M. Miyazaki, H. Okada, H. Yeung, K. Kitanjo, G. Thierry and S. Kita
(2012), ‘Sound Symbolism Helps Infants’ Word Learning’, in E. Cartmill, T. C. Scott,
M. Tamariz and J. Hurford (eds), The Evolution of Language: Proceedings of the 9th
International Conference on the Evolution of Language, 456–7, Singapore: World Scientific.
Itkonen, E. (2005), Analogy as Structure and Process: Approaches in Linguistics, Cognitive
Psychology and Philosophy of Science, Amsterdam: John Benjamins.
Itkonen, E. (2008), ‘The Central Role of Normativity in Language and Linguistics’, in J.
Zlatev, T. P. Racine, C. Sinha and E. Itkonen (eds), The Shared Mind: Perspectives on
Intersubjectivity, 279–306, Amsterdam: John Benjamins.
Johansson, N. and J. Zlatev (2013), ‘Motivations for Sound Symbolism in Spatial Deixis: A
Typological Study of 101 Languages’, Public Journal of Semiotics, 5 (1): 3–20.
Johnson, M. (1987), The Body in the Mind, Chicago: University of Chicago Press.
Johnson, M. (2010), ‘Metaphor and Cognition’, in D. Schmicking and S. Gallagher (eds),
Handbook of Phenomenology and Cognitive Sciences, 401–14, Dordrecht: Springer.
Jakobson, R. (1965), ‘Quest for the Essence of Language’, Diogenes, 13: 21–38.
Jääskeläinen, A. (2016), ‘Mimetic Schemas and Shared Perception through Imitatives’, Nordic
Journal of Linguistics, 39 (2): 15983.
Kendon, A. (2004), Gesture: Visible Action as Utterance, Cambridge: Cambridge University Press.
Kolter, A., S. H. Ladewig, M. Summa, C. Müller, S. C. Koch and T. Fuchs (2012), ‘Body
Memory and the Emergence of Metaphor in Movement and Speech’, in S. C. Koch, C.
Müller, M. Summa and T. Fuchs (eds.), Body, Metaphor and Movement (Advances in
Consciousness Research 84), 201–26, Amsterdam: Benjamins.
290 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Konderak, P. (2018), Mind, Cognition, Semiosis: Ways to Cognitive Semiotics, Lublin: UMCS
Press.
Kövecses, Z. (2000), Metaphor and Emotion, Cambridge: Cambridge University Press.
Kress, G. (2009), Multimodality: A Social Semiotic Approach to Contemporary Communication,
London: Routledge.
Lakoff, G. (1987), Women, Fire and Dangerous Things: What Categories Reveal about the Mind,
Chicago: University of Chicago Press.
Lakoff, G. and M. Johnson (1980), Metaphors We Live by, Chicago: University of Chicago
Press.
Lakoff, G. and M. Johnson (1999), Philosophy in the Flesh: The Embodied Mind and Its
Challenge to Western Thought, New York: Basic Books.
Lakoff, G. and M. Johnson (2003), Metaphors We Live By, 2nd edn., Chicago: University of
Chicago Press.
Langacker, R. W. (1987), Foundations of Cognitive Grammar: Theoretical Prerequisites,
Stanford: Stanford University Press.
Langacker, R. W. (1990), Concept, Image, and Symbol: The Cognitive Basis of Grammar,
Berlin: de Gruyter.
Langacker, R. W. (2006), ‘Subjectification, Grammaticization, and Conceptual Archetypes’,
in A. Athansiadou, C. Canakis and B. Cornillie (eds), Subjectification: Various Paths to
Subjectivity, 17–40, Berlin: de Gruyter.
Langacker, R. (2008), Cognitive Grammar: A Basic Introduction, Oxford: Oxford University
Press.
Langacker, R. W. (2016), ‘Working Toward a Synthesis’, Cognitive Linguistics, 27 (4): 465–77.
­Langacker, R. W. (2017), ‘Entrenchment in Cognitive Grammar’, in H. J. Schmid (ed.),
Entrenchment and the Psychology of Language Learning: How We Reorganize and Adapt
Linguistic Knowledge, 39–56, Berlin: Walter de Gruyter GmbH.
Levinson, S. C. (2003), Space in Language and Cognition: Explorations in Cognitive Diversity,
Cambridge: Cambridge University Press.
Louhema, K., J. Zlatev, M. Graziano and J. V. D. Weijer (2019), ‘Translating from
Monosemiotic to Polysemiotic Narratives: A Study of Finnish Speech and Gestures’, Sign
Systems Studies, 47 (3–4): 480–525.
Merleau-Ponty, M. (1962), Phenomenology of Perception, trans. C. Smith, London: Routledge.
Moskaluk, K. (2020), Dizziness of Freedom: The Influence of Maladaptive Anxiety on
Metaphorical Meaning-Making and the Motivation and Sedimentation Model (MA Thesis),
Lund University, Lund.
Möttönen, T. (2016a), Construal in Expression: Intersubjective Approach to Cognitive
Grammar, Helsinki: University of Helsinki.
Möttönen, T. (2016b), ‘Dependence of Construal on Linguistic and Pre-linguistic
Intersubjectivity’, Nordic Journal of Linguistics, 39 (2): 209–29.
Möttönen, T. (2019), ‘The Normative Basis of Construal’, in A. Mäkilähde, V. Leppänen
and E. Itkonen (eds), Normativity in Language and Linguistics, 125–50, Amsterdam: John
Benjamins.
Mouratidou, A. (2019), ‘Choice Awareness and Manipulation Blindness: A Cognitive Semiotic
Exploration of Choice-making’, Public Journal of Semiotics, 9 (1): 1–40.
Müller, C. (2016), ‘Why Mixed Metaphors Make Sense’, in R. Gibbs (ed.), Mixing Metaphor,
31–56, Amsterdam: John Benjamins.
Müller, C. and H. Kappelhoff (2018), Cinematic Metaphor: Experience–Affectivity–
Temporality, Volume 4, Berlin: Walter de Gruyter GmbH and Co KG.
COGNITIVE LINGUISTICS AND SEMIOTICS 291

Munhall, K. G., P. Gribble, L. Sacco and M. Ward (1996), ‘Temporal Constraints on the
McGurk Effect’, Perception and Psychophysics, 58 (3): 351–62.
Nöth, Winfried (1999), ‘Peircean Semiotics in the Study of Iconicity in Language’, Transactions
of the Charles S. Peirce Society, 35 (3): 613–19.
Nöth, Winfried (2008), ‘Semiotic Foundations of Natural Linguistics and Diagrammatic
Iconicity’, in K. Willems and L. De Cuypere (eds), Naturalness and Iconicity in Language
(Iconicity in Language and Literature 7), 73–100, Amsterdam: John Benjamins.
Oakley, T. (2009), From Attention to Meaning: Explorations in Semiotics, Linguistics, and
Rhetoric, Bern: Peter Lang.
Peirce, C. S. ([1866–1913] 1931–58) The Collected Papers of Charles Sanders Peirce, vols. 1–6,
C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University
Press. Cited as CP.
Pelkey, J. (2017), The Semiotics of X: Chiasmus, Cognition, and Extreme Body Memory,
London: Bloomsbury Academic.
Pelkey, J. (2018), ‘Upright Posture and the Meaning of Meronymy: A Synthesis of Metaphoric
and Analytic Accounts’, Cognitive Semiotics, 11 (1): 1–18.
Perniss, P., T. Thompson and G. Vigliocco (2010), ‘Iconicity as a General Property of
Language: Evidence from Spoken and Signed Languages’, Frontiers in Psychology, 1:
1–15.
Pragglejaz Group (2007), ‘MIP: A Method for Finding Metaphorically-used Words in
Discourse’, Metaphor and Symbol, 22 (1): 1–40.
Saeed, J. I. (2009), Semantics, Oxford: Oxford University Press.
Sandler, W. (2012), ‘Dedicated Gestures and the Emergence of Sign Language’, Gesture, 12:
265–307.
­Saussure, F. D. ([1916] 1960), Cours de Linguistique Générale/ Course in General Linguistics,
Paris and London: Payot and Duckworth.
Schmid, H. J. (2020). The Dynamics of the Linguistic System: Usage, Conventionalization, and
Entrenchment, Oxford: Oxford University Press.
Sonesson, G. (2007), ‘From the Meaning of Embodiment to the Embodiment of Meaning: A
Study in Phenomenological Semiotics’, in T. Ziemke, J. Zlatev and R. Frank (eds), Body,
Language and Mind, Vol. 1: Embodiment, 85–128, Berlin: Mouton de Gruyter.
Sonesson, G. (2009), ‘View from Husserl’s Lectern’, Cybernetics and Human Knowing, 16
(3–4): 107–48.
Stampoulidis, G. and M. Bolognes (October 2019), ‘Bringing Metaphors Back to the Streets: A
Corpus-based Study for the Identification and Interpretation of Rhetorical Figures in Street
Art’, Visual Communication, 1470357219877538: 1–35
Stampoulidis, G. (2021). Street Artivism on Athenian Walls: A Cognitive Semiotic Analysis of
Metaphor and Narrative in Street Art, Published PhD Dissertation, Lund University.
Stampoulidis, G., M. Bolognesi and J. Zlatev (2019), ‘A Cognitive Semiotic Exploration of
Metaphors in Greek Street Art’, Cognitive Semiotics, 12 (1): 1–20.
Stjernfelt, F. (2014), Natural Propositions: The Actuality of Peirce’s Doctrine of Decisigns,
Boston, MA: Docent Press.
Talmy, L. (2000), Toward a Cognitive Semantics, Vol. 1, Cambridge, MA: MIT Press.
Thompson, E. (2007), Mind in Life: Biology, Phenomenology and the Sciences of Mind,
Cambridge: Harvard University Press.
Torstensson, B. (2019), Metaphors and their Making: Bodily, Conventionally and Contextually
Motivated Metaphors in Inter- and Intra-Generational Conversations (MA Thesis), Lund
University, Lund.
292 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Tylén, K., R. Fusaroli, P. F. Bundgaard and S. Østergaard (2013), ‘Making Sense Together: A
Dynamical Account of Linguistic Meaning-making’, Semiotica, 194: 39–62.
van Langendonck, W. (2007), Iconicity the Oxford Handbook of Cognitive Linguistics, Oxford:
Oxford University Press.
Vandelanotte, L. (2017), ‘Viewpoint’, in B. Dancygier (ed.), The Cambridge Handbook of
Cognitive Linguistics, 157–71, Cambridge: Cambridge University Press.
Verhagen, A. (2007), ‘Construal and Perspectivisation’, in D. Geeraerts and H. Cuyckens (eds),
The Oxford Handbook of Cognitive Linguistics, 48–81, Oxford: Oxford University Press.
Vigliocco, G., P. Perniss and D. Vinson (2014), ‘Language as a Multimodal Phenomenon:
Implications for Language Learning, Processing and Evolution’, Philosophical Transactions
of the Royal Society B, 369 (1651): 1–7.
Wilcox, S. (2004), ‘Cognitive Iconicity: Conceptual Spaces, Meaning, and Gesture in Signed
Language’, Cognitive Linguistics, 15 (2): 119–47.
Zinken, J. (2007), ‘Discourse Metaphors: The Link between Figurative Language and Habitual
Analogies’, Cognitive Linguistics, 18: 445–66.
Zlatev, J. (1997), Situated Embodiment: Studies in the Emergence of Spatial Meaning,
Stockholm: Gotab.
Zlatev, J. (2012), ‘Cognitive Semiotics: An Emerging Field for the Transdisciplinary Study of
Meaning’, Public Journal of Semiotics, 4: 2–24.
Zlatev, J. (2015), ‘Cognitive Semiotics’, in P. Trifonas (ed.), International Handbook of
Semiotics, 1043–67, Dordrecht: Springer.
Zlatev, J. (2016), ‘Turning Back to Experience in Cognitive Linguistics via Phenomenology’,
Cognitive Linguistics, 27 (4): 559–72.
­Zlatev, J. (2019), ‘Mimesis Theory, Learning and Polysemiotic Communication’, in M. Peters
(ed.), Encylcopedia of Educational Philosophy and Theory, Dordrecht: Springer.
Zlatev, J. and J. Blomberg (2016), ‘Embodied Intersubjectivity, Sedimentation and Non-actual
Motion Expressions’, Nordic Journal of Linguistics, 39 (2): 185–208.
Zlatev, J. and J. Blomberg (2019), ‘Norms of Language: What Kinds and Where From? Insights
from Phenomenology’, in A. Mäkilähde, V. Leppänen and E. Itkonen (eds), Normativity in
Language and Linguistics, 69–101, Amsterdam: John Benjamins.
Zlatev, J., P. Zywiczynski and S. Wacewicz (2020), ‘Pantomime as the Original Human-specific
Communicative System’, Journal of Language Evolution, 5 (2): 1–19.
 CHAPTER FOURTEEN

Cognitive Science and

Semiotics
GÖRAN SONESSON

For the purpose of this chapter, semiotics and cognitive science will be considered to be
two distinct research traditions, each of which forms a temporal sequence made up of
problems formulated from a particular perspective, solutions proposed to these problems,
as well as new issues resulting from these solutions, and so on indeterminately. In this
view, a research tradition can sometimes, at some places, be consolidated into a discipline,
but before that, and after that, and at other places, it continues to exist as a network
of (generations of) scholars, who are connected because of sustaining these issues and
implementing some of these solutions. The perspective defining semiotics is the nature of
meaning, broadly conceived, which may involve how meaning is produced and conserved
by means of different vehicles (some or all of which may be signs, depending on how this
term is defined; see Sonesson 2010), and how it develops in children, and in the evolution
of human beings and other animals. The perspective of cognitive science is best described
as being the nature of consciousness; although, as is well known, many of its exponents
hoped to get rid of this pesky notion, substituting something easier to apprehend, such as
the computer or the brain. It is also often interested in the uses to which consciousness
(or whatever stands in for it) is put in the real world, as well as its history of evolution
and development.
As a research tradition, semiotics is very old, at least going back to Greek Antiquity,
having reached a second apogee in Medieval Scholastics, and a third one during the
Enlightenment, both periods of which abound in ‘treatises of signs’, before Peirce
and Saussure toiled with it, in their separate ways, during the penultimate turn of
the century, and the French structuralists made it fashionable during a short time
span about seventy years ago. Cognitive science is much younger, but it is made
up of disciplines, some of which, as research traditions, have rich pedigrees, which
is true of psychology, philosophy, biology, linguistics and anthropology, and some
which are very recent, such as artificial intelligence and neuroscience. Even more
recent is cognitive semiotics, which aims to put together the perspectives of semiotics
and cognitive science, and to resolve their separate, but (according to this tradition)
connected, problems, by means of adding up their solutions and pooling their
resources.
294 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

THE ENCOUNTER BETWEEN COGNITIVE

SCIENCE AND SEMIOTICS
Given that both semiotics and cognitive science are diversified research traditions, we need
to go into some details about them both, before we can start untangling the events leading
up to their encounter and the ensuing results. In this section, I suggest that three stages
of development can be found within the constellation of disciplines termed cognitive
science. They are termed ‘stages’, because they emerged in a temporal sequence, but this
is not denying that many representatives of cognitive science have made halt at stage 2,
let alone stage 1. Since this whole set of books is about semiotics, less need be said about
(classical) semiotics (see vol. 1: History and Semiosis). Still, it is necessary to underscore
where possible points of contact with cognitive science could be expected to occur. The
third subsection will give a primary sketch of the ways this encounter has played out so
far. Some of these scenarios will then be fleshed out in subsequent sections.

The stages of cognitive science

Although the first ‘popular’ presentation of cognitive science bore the title, The Mind’s
New Science (Gardner 1985), the original purpose of this conglomerate of disciplines was
actually to do away with the mind, in the sense of consciousness, and instead substitute
the computer. This first stage comprised in the tripartite scheme for the development
of cognitive science delineated by Sonesson (2009: 115 ff.) corresponds to the good
old days of artificial intelligence, from ‘symbolic computation’ as initiated by Herbert
Simon to connectionism, originally associated with the work of David Rumelhart and
James McClelland. Without any essential change to the basic epistemology, the computer
metaphor was later exchanged for that of the brain, epitomized by Daniel Dennett’s
‘brain-in-the-vat’. The hardiest exponents of this neurological reductionism might be
Paul and Patricia Churchland, but there can be no doubt the most influential thinkers of
this stripe, at least in relation to the humanities (including semiotics), have been George
Lakoff and Mark Johnson.
In this scheme, the second stage starts out from the idea that, to function and to make
sense, the computer, and by implication the mind, has to be ‘situated’: in other terms,
it needs a context. If we restrict this context to the human body (or what is equivalent
to it for the computer), we arrive at the notion of embodiment according to Lakoff and
Johnson (1999). If then we take into account that, to human beings (and as it can be
simulated in a computer), the representation of the body in the brain can be extended
to the body in the world of real human experience, it is not too difficult to take the
next step allowing the context to comprehend much more than the body. When Andy
Clark and David Chalmers (1998) proceeded to include notebooks and everything else
which might conserve information outside of the human organism in the notion of
extended mind, their argument was exactly the same as that originally used to identify
the computer and the mind: they are supposed to be functionally equivalent, in other
words, substitutable. Pursuing the computer-brain metaphor to the end, it is possible
to conclude, with Hutto and Myin (2013), that no content, or as they sometimes say,
no meaning, is necessary, for neither the extended nor the un-extended mind, except
(perhaps) in the case of language. So far, this historical account may roughly correspond
to the distinction, made by Lakoff and Johnson (1999: 75ff.), between first- and second-
generation cognitive science.
COGNITIVE SCIENCE AND SEMIOTICS 295

Starting out from Clark and Chalmers, we will arrive, by way of several intermediaries
which we will neglect at present, in Mark Rowlands’ (2010) terms, to ‘the mind embodied,
embedded, enacted, and extended’. Rowlands is out to establish whether or not all these
addenda really are required, and to what extent they overlap. That, however, is not our
present concern. From our point of view, the real issue, as we will see, is whether any
of the ‘4E’ are really functionally equivalent to the un-extended mind. The notion of
embodiment, in a more complicated sense than that of Lakoff and Johnson, epitomized
by one hand feeling the other, was suggested much earlier, by Edmund Husserl, and
later, as is more well known, by Maurice Merleau-Ponty. As for the extended mind more
generally, it has much more of a pedigree, although it might only have been given a
distinct status by Friedrich Hegel and his followers, and later by Husserl and the Neo-
Kantians. Within recent semiotics, this theme has also been very much the focus of Jurij
Lotman’s (1979) work. This is not to say that the current discussion of extended mind
has not contributed new perspectives on this old issue, which are worthwhile exploring.
The problem is, however, that no kind of extended mind can ever be equivalent to the
un-extended mind, because they all need the mind, not only to be produced, but also to
be interpreted. In the example given by Clark and Chalmers (1998), Otto’s notebook is
meaningless, if he (or somebody else) does not write down the information, and if he does
not remember (in particular since he is supposedly an Alzheimer’s patient) to look it up.
The extended mind has to be extended from somewhere (Sonesson, 2021a).
Although it is not pursued by most exponents of cognitive science, the third stage is
of particular interest to us, because of its overlap with recent development in (cognitive)
semiotics. It has resulted in a renovation of the study of consciousness, where Husserl-
inspired phenomenology has been prominent, although there have also been contributions
from within the tradition of British-style linguistic philosophy. In spite of being anathema
to much of mainstream cognitive science, notions such as agency, intentionality,
consciousness, empathy and intersubjectivity are typical of ‘consciousness studies’, such
as practised, for instance, by Evan Thompson (2007), Shaun Gallagher (2005) and Dan
Zahavi (2005). A special mention should be made of Anthony Chemero (see Käufer and
Chemero 2015) for bridging 4E cognitive science and phenomenology, including in the
latter, the ecological psychology of James Gibson (as also suggested in Sonesson 1989).
Since we are interested, in this chapter, in the encounter of semiotics and cognitive
science, it is important to note two other characteristics of cognitive science, which results
from it being, unlike semiotics so far, a constellation of already-existing sciences: first
of all, thanks, notably to psychology, cognitive science employs experimental methods,
which has very rarely been the case in semiotics. In the second place, because of the
inclusion of linguistics and anthropology, cognitive science has posed questions about
the origin of language and, marginally, some other semiotic resources, with reference to
both child development and the origin of the human species, which have been a challenge
to semiotic theory. This is particularly true of the work of two scholars whose basis is in
neuroscience: Terrence Deacon (1997), who has had explicit recourse to terms borrowed
from Peircean semiotics, and Merlin Donald (1991, 2001), who doesn’t use any semiotic
terms, but who broaches issues in other ways familiar to anyone involved with semiotics.

Semiotics as a research tradition

Since semiotics is the main subject of this book, we will not try to cover that ground here.
Suffice it to say that, at the last mid-century, Saussure gained a prodigious following, in
296 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

particular as his notions were relayed by Louis Hjelmslev, in the French structuralists,
whose most prominent representative turned out to be the Italian Umberto Eco. Focusing
on one of the contradictory strands in Saussure’s posthumously published lectures,
linguistics was described as an autonomous science, thus being enjoined to ignore any
knowledge gained from other disciplines. Widening the scope to all kinds of semiosis,
the French structuralists, and, more insistently, the Greimas school, claimed the same
privilege for semiotics.
­Sonesson (1978) suggested in his dissertation that rather than an advantage, this
amounted to a handicap. Unsurprisingly, his voice was not taken into account at the time,
and there is no reason to think that many other protests did not go unheard in the same
period. In Germany, nevertheless, such an extension of semiotics was realized concretely
in a number of books by Walter Koch (1971) and his then disciple Winfried Nöth (1977).
Koch (1986) even broached the subject of semiotic evolution. Although this cannot be
considered an encounter of semiotics with cognitive science, which was hardly thought
of at the time, it certainly constituted one of the failed attempts of semiotics to go into
the interdisciplinary mode. In view of the importance of experiments to contemporary
cognitive semiotics, it is worthwhile noting some early experiments, realized within a
specifically defined semiotic framework. Thus, René Lindekens (1971) carried out
experiments aiming to show that even photographic images were, to some extent, non-
iconic, since the shading of the picture approximated the real-world situation to the
same extent that its contrast departed from that situation, and vice versa. In a similar
vein, Hartmut Espe (1983) realized experiments which purported to show that the same
photograph differently contrasted carries different affective import transferred to the
subject of the picture: the girl appears more or less beautiful, the landscape more or less
melancholic, and so on. But these were isolated instances of experiments being realized
within semiotics.
Among those protesters whose voices were more widely heard, Paul Bouissac may
have been the first to give a wider scope to semiotics. In his interpretation, semiotics is a
kind of ‘meta-analysis’, which ‘consists in reading through a large number of specialized
scientific publications, selected among the published literature in one or several domains
of inquiry, and of relating the partial results within a more encompassing model than the
ones that are held by the various specialists concerned’ (Bouissac 1999: 4). While this is a
very good description of interdisciplinary approaches in general, it is insufficient to spell
out what is specific to this particular interdisciplinary approach. Sonesson (2010, 2016)
suggested that the particular point of view taken by semiotics on these different domains
of study pertains to the nature of meaning: more specifically, how meaning is produced,
conveyed and experienced in different cases.

Approaches to cognitive semiotics

Those of us who were present at the congresses of the IASS during the 1990s, have a
distinct memory of Thomas Sebeok, who loomed large at those congresses, repeatedly
pointing out to us that the difference between semiotics and cognitive science basically
consisted in the latter, but not the former, having discovered the required formula to
receive founding for its research. In his review of semiotics in the United States, Sebeok
(1991: 2) claims that cognitive science is (at worst) ‘a historically untutored and – dare
I use so pejorative a word – pretentious relabelling of an ancient multidisciplinary
enterprise’ corresponding to what we have described above as the semiotic tradition of
COGNITIVE SCIENCE AND SEMIOTICS 297

thought. Ironically, Sebeok goes on to claim that the central notion of both traditions is
that of representation. There is no space here to explore the polysemy of this term, also
within the two traditions: suffice it to say that, at the time when Sebeok was writing his
paper, cognitive science used the term in order to avoid talking about meaning; and since
then, major trends within both semiotics and cognitive science have rejected the notion of
representation, mostly without attending to the polysemy of the term, in favour of some
version of active meaning-making, termed sign process, enaction or the like.
The term ‘cognitive semiotics’ may first have been used, notably by Juan Magariños
de Morentin (at least since 1999)1 as a label for Peircean semiotics. According to
Claudio Paolucci (2021: 2), Umberto Eco (1999) also used this term with respect to
Peirce’s semiotics. As far as I can tell, however, Eco (1999: 13) only claims that Peirce
was the sole person whose theory was ‘semiotic, cognitive, and metaphysical all at the
same time’. Such a usage is beyond our present concern, since it has nothing to do with
the encounter between semiotics and cognitive science. Nor will it be possible, in the
following, to account for all those scholars who have used the term in the required
sense. In what follows, we will concentrate on contributions to the epistemological
issues arising from such an encounter (see Konderak 2018). It is intriguing, nevertheless,
that a least two scholars who want to marry cognitive science with semiotics, Nathan
Houser (1996) and Claudio Paolucci (2021), understand the latter in terms of the
Peircean brand.
Perhaps the first explicit proposal for organizing an encounter between semiotics and
cognitive science was formulated by Thomas Daddesio (1995), which, at the time, seems
to have had zero impact. In Daddesio’s view, semiotics tends to restrict its attention
to static structures, whereas cognitive science is involved with the way structures are
processed by consciousness. As pointed out elsewhere (Sonesson 2016), this is a gross
oversimplification. At the time, Daddesio had no means of knowing that, some years
later, beginning with Clark and Chambers, cognitive science would trade very much into
structures; but his description of semiotics suggests that he basically identified it with
French structuralism, ignoring, most notably, the Prague School of semiotics, which was
very much inspired by phenomenology, while also insisting on the social character of
semiosis. Nevertheless, Daddesio was quite right in pinpointing the correlation between
intersubjective structures (the Saussurean langue) and subjective access (language as
‘performance’, not in the sense of Chomsky but in that of psycholinguistics).
It was probably Per Aage Brandt who first used the term ‘cognitive semiotics’ (dated
by Brandt 2020 to the 1990s) to designate an approach fundamentally preoccupied with
questions of linguistic semantics, which, from the very beginning, had been conceived
within the framework of Greimasean semiotics (including the later contributions of René
Thom and Jean Petitot), but which took on new inspiration from cognitive linguistics,
notably as practised by Lakoff and Johnson (e.g. 1999) as well as Fauconnier and Turner
(e.g. 2002). At the time, Brandt was active at Aarhus University in Denmark, but he later
went to Case Western Reserve University in Ohio, where he joined Merlin Donald and
Todd Oakley. Remaining in Aarhus, Peer F. Bundgaard may have been the first to publish
an anthology bearing the title, in Danish, ‘Cognitive Semiotics’ (Bundgaard et al. 2003),
which, however, apart from texts by Brandt, Bundgaard, Stjernfelt, Thom and Petitot,
contained translations of papers mostly by exponents of cognitive linguistics. Frederik
Stjernfelt (2007), who spent some time in Aarhus, but later returned to Copenhagen,
is a formidable scholar, who has made important contributions to semiotics and
phenomenology, both in the sense of Peirce and that of Husserl. In recent years, Aarhus
298 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

has been more notable for realizing many experimental studies, partly with a semiotic
agenda (see the section below ‘The Contribution of Experimental Semiotics’).
Quite independently of the Aarhus connection, another approach to cognitive semiotics
emerged from the convergence of the lifetime work of two scholars at Lund University,
who only began to collaborate at the turn of the last century: Göran Sonesson, who, after
his linguistics studies in Lund, spent ten years working in France and in Mexico, after
which he returned to Sweden to establish semiotics as a subject of doctoral studies; and
Jordan Zlatev, who, after studies in cognitive science and linguistics, was employed at
the Department of Linguistics in Lund. Zlatev and Sonesson initiated their collaboration
with a faculty-internal project concerned with language, gesture and pictures in a
developmental and evolutionary perspective, then proceeding to form the Swedish part in
the EU project ‘Stages in the Evolution and Development of Sign Use’. Subsequently, they
together the lead of a six-year-long programme, entitled ‘Centre for Cognitive Semiotics’,
involving around thirty scholars (not counting the students), many from linguistics, but
also from several other human sciences. From these experiences emerged an idea of
cognitive semiotics as a specific way of merging semiotics and cognitive science which
will be the main focus of the following sections.

SOME ISSUES OCCASIONED BY THE ENCOUNTER

According to the first issue of the journal Cognitive Semiotics, published in 2007, cognitive
semiotics aims to integrate ‘methods and theories developed in the disciplines of cognitive
science with methods and theories developed in semiotics and the humanities, with the
ultimate aim of providing new insights into the realm of human signification and its
manifestation in cultural practices’. The ambition to take on the legacies of both semiotics
and cognitive science is unproblematic, to the extent that they overlap or complement
each other. The case is different when there seem to be contradictions between these
two heritages, or perhaps between different strands within them. In the latter case,
methodological and epistemological resolutions are called for. Since Zlatev and Sonesson
seem to have been the only scholars actively concerned in sketching an epistemology for
cognitive semiotics, the paragraphs which follow will attend to their proposals. This is
not to deny that, in the end, cognitive semiotics may develop in quite different ways.

The part played by linguistics

According to Jordan Zlatev (2015), linguistics should be considered, along with cognitive
science and semiotics, to be the third element making up cognitive semiotics. It may be
objected that linguistics is already one of the established disciplines incorporated into
cognitive science, and that it is also, in principle, though rarely as an academic fact, a part
of semiotics. It will be remembered that Saussure reinvented semiotics (his ‘semiology’) as a
wider domain of research which should not only include the study of language as a part, but
which was also called upon to take the latter at its template. This precept was unfortunately
taken by most representatives of French structuralism as a pretext for projecting models
and patterns which might be adequate for describing language to other semiotic means.
The result could only be a serious distortion of the description of other semiotic resources,
as was notably the case in pictorial semiotics (see Sonesson 1989, 2016).
There are, however, other reasons for singling out linguistics. One of these reasons,
and an important one, has to do with the epistemological status of (cognitive) semiotics.
COGNITIVE SCIENCE AND SEMIOTICS 299

The problematic position of the humanities, in a context in which the natural sciences
already were the touchstone of scientificity, has been debated many times, but two such
occasions are particularly pertinent in the present context, because they allowed, to a
certain extent, the human (and social) sciences to gain a certain acceptance. The first of
these rounds, the so-called Methodenstreit, which took place at the end of the nineteenth
century, involved Heinrich Rickert, Wilhelm Windelband, Max Weber and Wilhelm
Dilthey (see Beiser 2011), while the second round, which occurred in the middle of the
twentieth century, was initiated by Hans-Georg Gadamer, Jürgen Habermas and Paul
Ricoeur (see Radnitzky 1970; Ricœur 1983–5; Habermas 1985, 1995). The outcome of
the first round was to describe the natural sciences as being nomothetic, that is, aiming to
establish general laws, while the humanities were supposed to be ideographic, that is, their
goal was to describe particular (historically situated) facts. The main contribution of the
second round was to complicate the initial conditions, by including the social sciences,
which, in the meantime, stood out as a different kind of science.
In the intermission between the two rounds, Ernest Cassirer (1945: 89) observed that,
during the first debate, ‘the fact that there is such a thing as human speech and that there
is such a thing as linguistics was never mentioned’. He did not hesitate to qualify this as
‘a very regrettable fact, a sin of omission that could not fail to have its consequences’. It
is easy to agree with this judgement. While linguistics is clearly, in this taxonomy, one of
the human sciences, it is basically involved with laws and regularities, synchronically and,
at least in part, diachronically (Sonesson 2018b). Cassirer did not live to discover that,
also in the second round of the Methodenstreit, the peculiarity of linguistics was ignored.
Without at the time being acquainted with Cassirer’s pertinent observation, Sonesson
(1989, 2016, 2018b), taking semiotics to be a generalization of linguistics as a type of
science, suggested both were fundamentally nomothetic, just as the natural sciences, but
that, on the other hand, they were focused on qualities, like the traditional humanities,
unlike the natural sciences, which were basically interested in quantities.
This is a sense in which linguistics may serve as a paragon for the other semiotic
sciences: in being predominantly nomothetic and qualitative. According to this
conception, pictorial semiotics has to construct a model for the analysis of pictures, and
then apply this model in the same way to a whole corpus of pictures (which is supposedly
defined by some explicit criteria as being one and the same corpus), and if you observe
discrepancies between the model and the corpus, you have to change the model. This has
never seriously been done, no doubt in part because it would be a tedious work, which
can only properly be realized with the help of computer algorithms (see Sonesson 1989;
Reyes and Sonesson 2019).
There is another sense in which linguistics may well play a fundamental part in the
development of cognitive semiotics. There is a rich tradition in linguistics for studying
the content side of the sign, that is, semantics, even if this part of the tradition was in
part curtailed already by structuralism, and then more decisively by generative linguistics.
Semantics writ large, it could be said, is semiotics. The pioneers of this kind of study
were no doubt Jost Trier and Leo Weisgerber, whose studies of the semantic field were
adapted to structuralist premises by Roman Jakobson, Louis Hjelmslev, Helmut Gipper,
Eugenio Coseriu, Bernard Pottier, the early A. J. Greimas and others (see Diodato 2019;
Coseriu 1977; Greimas 1966). After the hibernation of semantics within the framework
of generative grammar, George Lakoff’s (1969) first attempt to strike it out on his own,
under the label of ‘generative semantics’, establishing connections with psychology and
fuzzy logic, as well as Charles Fillmore (1983) studies in case grammar, constituted, in
300 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

retrospect, a break with the Chomskyan paradigm, as well as a return to an earlier linguistic
tradition. The later work by Lakoff, and his collaborators, as well as the contributions
by Ronald Langacker and Leonard Talmy, while perhaps not taking much account of
the earlier tradition, has certainly set up models which have been inspiring, not only
within linguistics proper, but also for many scholars aiming to do cognitive semiotics (see
Geeraerts and Cuyckens 2007).
It is important to take into account these pioneer works within semantics, but, at
the same time, there is a danger of generalizing too much from models possibly valid in
linguistics to models for other semiotic resources.

With a little help from phenomenology

Phenomenology, as we have seen (in an earlier section entitled ‘The Stages of Cognitive
Science’), became an important contributor to cognitive science in its third stage. Although
the basic tenets of phenomenology and much of contemporary semiotics may seem to be
emphatically opposed, there have always been many connections between them, starting
out with Husserl’s early text bearing the title, ‘Semiotik’. The entire model elaborated by
the Prague semiotic circle was based on phenomenology (Mukařovský 1974; Sonesson, in
press). Within Peirce’s semiotics, a distinct branch bore the name phenomenology, later
rebaptized ‘phaneroscopy’, although it continued to carry almost the same definition as
Husserlean phenomenology (Sonesson 2013; 2017). There is an obvious substratum of
phenomenology, or more exactly, existentialism in much of French semiotics, already in
the early work of Greimas (1966).
Apart from being a domain where some parts of cognitive science and some parts of
semiotics overlap, phenomenology has a very important part to play in cognitive semiotics.
Since this overlap is otherwise rather limited, phenomenology is already required to
mediate between the constructs of cognitive science and semiotics. It has to investigate
whether both research traditions are really involved with the same thing, even though
they use the same term, and whether they may be attending to the same subject, although
they use different terms. Either way, phenomenology serves to explore the potentialities
of enriching one conceptual framework using concepts and findings of the other. Since
there are many brands of semiotics as well as of cognitive science, phenomenology should
also take on the hard task of negotiating between their different claims and assumptions.
In experimental studies, furthermore, it has to help formulating the tasks set up in such
a way that they are relevant to the behaviour of real life, as well as connecting the results
of the experiment to the situation in the experienced Lifeworld. Before more can be said
about this issue, we have to consider the importance of experiments to cognitive semiotics,
as well as to delve deeper into the nature of the semiotic sciences, more commonly known
as the human and social sciences.

The contribution of experimental semiotics

Even those, within semiotics, such as Koch, the early Nöth, and later Sonesson, who
considered semiotics to be in the business of bridging traditional disciplines, taking into
account concepts and findings from different disciplines, reviewed from an overarching
semiotic perspective, did not realize any experiments of their own. Those semioticians
who did, like Lindekens and Espe (see the section ‘Semiotics as a Research Tradition’)
did not offer any rationale for doing so. Experiments have turned out to be an important
COGNITIVE SCIENCE AND SEMIOTICS 301

ingredient in cognitive semiotics, at least partly resulting from the encounter with cognitive
science, and have played an important part in the approaches to cognitive semiotics, both
in Aarhus and in Lund (see the section ‘Approaches to Cognitive Semiotics’).
The term, ‘experimental semiotics’ is now normally applied to an experimental
paradigm, which, according to Gallantucci and Garrod (2011: 1), proposes to study
‘novel forms of communication which people develop when they cannot use pre-
established communication systems’. In spite of Galantucci’s claim to the contrary, this
label has been used before, at least twice, to designate, on the one hand, the experimental
study of perceptual meaning, as opposed to conventional meaning-making, and, at the
other extreme, the study of complex social situations. In relation to Galantucci’s bare
bone approach, both the minimalist procedure centring on perception and the maximalist
manner including all of human sociability clearly involve human meaning-making,
although subjecting it to experimental study in different ways (see Sonesson 2019).
In the Galantucci-style bare bone approach, subjects are placed in a situation in
which they are required to communicate, without using language, and in which they
are, depending on the particular experimental situation, allowed the use of pictures
or gestures, with or without some particular filter being applied to the employment of
these means. This is no doubt a promising experimental procedure, and it has been used,
not only by Galantucci and his close collaborators, but also by Kristian Tylén and his
collaborators in Aarhus (e.g. Christensen et al. 2016). It should not be forgotten, however,
that all experimentally manufactured situations are artificial in relation to the world of our
experience, the Lifeworld, and this applies even more to such stripped-down situations.
Cognitive semiotics in Lund has so far mostly pursued a more classical psychological
approach, although starting out from features of more obvious relevance to semiotic
theory. This involves creating a much richer experimental situation, which is, relatively
speaking, closer to Lifeworld experience, and which thus may be used to mimic real-world
situations in evolution and child development (e.g. Zlatev et al. 2013; Hribar et al. 2014;
Sonesson and Lenninger 2015; Lenninger et al., 2020). Even though these experiments are
no doubt closer to ordinary life, they remain very distant from it, since they aim to control
a small number of features in the environment, in order to study the ensuing outcome.
Even in this case, then, experiments have to be complemented by other approaches to
bridge the space between the artifice of experimental situations and the Lifeworld.

Cognitive semiotics in relation to the human and social sciences

If we except the third phase of cognitive science, which is influenced by phenomenology,
the domineering ideology defining cognitive science is physicalist; that is, it takes for
granted that the world as described by natural sciences is more real and/or basic that the
one experienced by human beings. Many semioticians have no doubt shared in such a
physicalist worldview, but, since semiotics is concerned with meaning, which cannot, as
such, be described in physical terms, this must, in the end, be a self-defeating approach.
Giambattista Vico ([1710] 1988: 45) famously formulated the Verum-Factum principle,
according to which we, as human beings, have access to the truth about anything we, or
other human beings, have created, that is, culture in general, and all specific cultures,
whereas knowledge of other things, such as nature, can only be more indirectly, and
laboriously, gained.
Without any reference to Vico, the semiotician Luis Prieto (1975: 215 ff.) has suggested
that, contrary to the reigning positivist epistemology, our access to the subject matter
302 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

of the human and social sciences (which he terms the semiotic sciences) is direct, and
thus objective, while our access to the subject matter of the natural sciences can only be
indirect and therefore subjective. Prieto observes that the sound, as a material object, may
have innumerable properties, but the phoneme, which is the subject matter of linguistics,
is defined by its functional relations to the other plane of the sign, and thus has a limited
number of clearly defined features. Nevertheless, there is a problem with this claim, not
only because our knowledge even of what we ourselves have created may be somewhat
obscure, but more importantly, because, even in linguistics, we will have to go below the
level of what we as language users are ordinarily aware of, beyond the phoneme to the
phonological features, and so on (Sonesson 1989: 28 ff.). Put in the traditional terms of
hermeneutics, we may say that, after coinciding with the user in his or her understanding
of the phoneme, the semiotician goes on to explain the conditions of possibility of this
understanding on the level of distinctive features. If so, semiotics involves the knowledge
of the user but also something more.
There are basically two procedures for going beyond what is directly given to human
beings as users of meaning (see Sonesson, 2021b). One of these procedures, which can
be characterized as an iconic operation, is particular to the semiotic sciences, as suggested
by Vico and Prieto. The second one, which can be described as being indexical, is
fundamentally the same as that employed by the natural sciences. Here it must be clear
that these terms are not used to describe different kinds of signs, and not even, in the first
case, a kind of ground (see Table 14.1). According to Sonesson’s (2013) meta-analysis
of Peirce’s different phenomenological characterizations of grounds, iconicity basically
corresponds to the idea of ‘something being there’, that is, a direct perceptual experience.
Indexicality, according to the same meta-analysis, involves something being added to
what was first perceived, by some more indirect means.
Along the lines of what will henceforth be called the Vichean-Prietean supermaxim, it
may be supposed that, since nature is opaque to us, we can only gain knowledge about
it using indexical operations, but that the semiotic sciences, however paradoxical that
may seem, can use iconic operations to go beyond first appearances, while also needing,
in specific cases, to have recourse to indexical operations. The first procedure is only
possible because of what Husserl has described as the sedimentation of acts in the stream of
consciousness. According to Husserl (1901, 1966), all our acts of consciousness (including
those which have an external manifestation) can only have a meaning because they take
for granted the ‘passive synthesis’ of earlier experiences, building on numerous other acts
of consciousness, some of which took place in our earlier life (genetic sedimentation),
and many others which resulted from the experience of earlier generations of human

TABLE 14.1 The relations between principles, grounds and signs in the present
interpretation of Peirce (Sonesson 2010)
Firstness Secondness Thirdness
Principle (Firstness) Iconicity — —
Ground Iconic ground Indexicality = Symbolicity =
(Secondness) indexical ground symbolic ground =
Sign Iconic sign (icon) Indexical sign (index) Symbolic sign
(Thirdness) (symbol)
COGNITIVE SCIENCE AND SEMIOTICS 303

beings (generative sedimentation). Since the sedimented results of these acts are what help
make sense of our current interaction, we may be able to reactivate these acts, using the
procedures of phenomenology. Combining Husserlean and Peircean phenomenology, we
could say that this is a way of delving deeper into iconicity.
There are two, very different, reasons why indexical procedures are needed. In the
first case, some facts are simply not accessible to the human mind, and we are reduced
to proposing relatively arbitrary hypotheses, which can be more or less well founded, to
the extent that they serve to connect a greater or lesser extent of nodes of observations,
whether these observations are made with the naked eye, with some, relatively speaking,
simple extension of the eye such as microscopes and telescopes, or more sophisticated
devices such as PET-scanning, fMRI and the like. In any case, these are probes into a
matter which remains basically inaccessible to us, which is not to deny that the history
of the (natural) sciences of the last 500 years has been the story of the extension of our
ability to delve deeper into this black matter. The second case is different, but it leads
to a similar conundrum. In spite of all our efforts to reanimate the sedimented layers of
our consciousness, whether they were laid down genetically and generatively, our access
to their meaning is often blocked. Using a term which has a history, since, after being
coined by Ludwig Feuerbach, it was adopted and adapted by Karl Marx, and thus by all
his followers, we might say that some kinds of sedimentations resist reanimation, because
they have become reified. This is why some approaches to the study of human, and in
particular, social, life have adopted the indexical procedures characteristic of the natural
sciences. A case in point is economics, which is certainly opaque to human consciousness,
although it is clearly a human creation. Although the term was coined for different reasons,
economics deserves to be called ‘the dismal science’, because its hypotheses most of the
time turn out to be wrong. The difference between natural science and economics is that,
in the latter case, we do not have to wait for aeons to pass, because the predictions are
rapidly falsified by close-time historical events. That is why economics and comparable
approached to social life are, in Prieto’s sense, as subjective as the natural sciences.

COGNITIVE SEMIOTICS ON ITS OWN

The purpose of the present section is to draw some conclusions from the preceding
section, going beyond the encounter between semiotics and cognitive science, and taking
into account the offer, more or less clearly assumed, by both these approaches to serve
as a way of integrating the diverse human and the social sciences, and thus, by the same
token, clarifying their relation to the scientific enterprise as a whole.

The domains of study

Jordan Zlatev (2008) has suggested that the domain of study covered by cognitive
semiotics can be divided into several levels, each with its corresponding subjects, worlds
and signification. Meaning, in the broader sense of the term (well beyond the notion of
sign), can be understood as the relationship between a subject and his/her environment
which is determined by a value. On a biological level, this value is life, more specifically
the maintenance and reproduction of life, and the subject is the organism. This is what
was described by Jakob von Uexküll (1973) as the ‘functional cycle’. The next level,
which Zlatev calls the natural Lifeworld (in the sense of what pertains to nature), has as its
subject the minimal ego and is governed by the value of sentience. As Zlatev identified this
304 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

latter value with intentionality, in Husserl’s sense, as it originally appears in perception, it

can be understood as consciousness, in the broadest sense of the word.
Above the level of the minimal Ego is situated the cultural Ego, the subject of the
cultural Lifeworld, which is determined by a value that Zlatev calls meaning. As we have
already used this term to describe the relationship between a subject, an environment and
a value, we will call culture the specific value of the third level. What is new here is life in
a community, also known as sociability. Zlatev originally introduced the sign at this level,
in the narrow sense of Sonesson (1989): a unit of two items which are differentiated from
each other, one of which is directly present but not in focus, while the other is in focus,
but not directly experienced. Sonesson has long argued, mostly orally (but see Sonesson
2015a) that there must be an intermediate level, which is not made up of signs, but which
is still intersubjective and contains behaviours that set one society apart from another.2
Zlatev’s fourth level, which is for Sonesson the fifth (and now also of Zlatev 2015),
consists for Zlatev of language, but it could also be generalized to include all kinds of
semiotic resources which are in any way systemic, whether they depend mostly on rules of
substitution, which is the case with language and formalized gesture systems, or on rules
of transformation, as is the case with pictures and some kinds of gestures (see Table 14.2).
Many animals, notably primates, but also some birds, clearly have cultures, in the
minimal sense of the term. It remains difficult to determine whether some animals are
capable of using signs or not, but systems of signs seem to be the exclusive property
of human beings. In any case, there are many other reasons at present for conceiving
sociability as being a particular rung on the semiotic hierarchy. It seems very probable
that group selection is at least as important in evolutionary biology as individual selection,
and that the specificity of human culture depends largely on the combination of natural
and cultural selection rendered possible by group selection (see Richerson and Boyd
2005; Dunér and Sonesson 2016).

Evolution and development and anything in between

According to a scenario for human evolution designed by Merlin Donald (1991), which
has been very influential in cognitive semiotics, episodic memory, the memory for single
situated happenings, is something that human beings share with many other animals.
Mimetic memory, however, involves using the body to record behaviour, and is restricted
to human beings and their antecessor species. Donald’s third stage, mythic memory,
corresponds to language, so designated because, according to Donald, it involves the

TABLE 14.2 ‘The semiotic hierarchy’: Five types of selves, worlds and
significations
­ (adapted from Sonesson 2015a, as an elaboration of Zlatev 2009)
Level Subjects Worlds Significations Value
1 Organism Umwelt Biological Life
2 Minimal Ego Natural Lebenswelt Intentional Sentience
3 Cultural Ego Cultural Lebenswelt Cultural Culture
4 Signic Ego Universe of signs Signic Sign
5 Semiotically Universe of discourse Systemic Sign system
specific ego
COGNITIVE SCIENCE AND SEMIOTICS 305

construction of narratives, no doubt initially used to recount myths, and thought by

Donald to be at least one of the reasons why language evolved. In this scenario, language
would seem to be a limiting case: the capacity for language might well have an evolutionary
origin, but the development of particular languages is part of history. It is in the fourth
stage, however, which Donald calls theoretic culture, that history comes of its own.
Pictures, writing and theories are the three human products typically manifesting this
stage according to Donald. In later texts, Donald (2010) has used the term ‘exograms’ to
pinpoint the specificity of this stage, which consists in making memory available outside,
and relatively independently, of human beings, both as to their body (as in mimetic
memory) and as to their consciousness (as in mythic memory).
Within Donald’s mimetic stage, a number of important things, not properly
distinguished in his scheme, seem to happen. First there is the use of tools in a systematic
way, which may be thought of as memory embodied in the limbs. If tool use is going to
gain currency in a community, imitation must take place, and imitation would seem to
require the capacity to pick up (cognize) the type, given a number of tokens – or, in other
words, extracting it from a sequence of episodes. Thus, in order to imitate an observed
use of a hammer, which is a token of the act of hammering, one must be able to abstract
the type from this individual act and the particular physical and motivational context in
which they take place. It is only once you conceive the act of hammering as a type that
you can transpose hammer usage to other contexts, in this way giving rise to a second
token, and a third, etc. (see Dunér and Sonesson 2016). According to Donald, this is also
the stage in which gestures begin to be used, but if so, this requires something more than
the mere fitting of the token to the type, the discovery of signs, in the sense of a unit
consisting of two items which are differentiated from each other, one of which is directly
present but not in focus, while the other is in focus, but not directly experienced.
In fact, in terms of the domains of meaning distinguished above (see the section ‘The
Domains of Study’), already the episodic stage seems to go well beyond the mere Umwelt,
entering the natural Lifeworld, but the mimetic stage appears to cover all the distance
from the natural Lifeworld to the cultural Lifeworld and further to the universe of signs.
On the other hand, many animals clearly live in a pre-episodic Umwelt, because they do
not need to have any experience of the here-and-now: the tick, for instance, has to realize
its acts in a certain temporal sequence, but not going back and forth in time (see Sonesson
2015b). The import of the third stage clearly is sociability becoming an important
factor in the creation of meaning. This is not to suppose, along the lines of Rousseau’s
‘philosophical history’, that human beings once lived alone, like the orangutans, or even
were the orangutans, but only that this factor gained more importance, and was more
decisive, once cultural evolution came to predominate over natural evolution.
The fourth stage also has its problems, which concern, in particular, the case of
pictures. There is no obvious reason to suppose that the advent of picture production
had to attend the capacity to preserve memory outside of human bodies and minds. If
pictures were created earlier in history, they may have employed more perishable media,
such as sand or human skin. In fact, there are historically and ethnographically testified
examples of drawings customarily being made on the ground (for a recent example, see
Green 2014; cf. Sonesson 2015b).
Not even the tentative to reanimate generative sedimentations is of any help here. All
approaches to evolution are bound to encounter the hurdle of memory reification. Studies
of evolution are fundamentally reduced to using indexical procedures, in a situation in
which the indexicalities available for making the connection are very sparse. Although we
306 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

have long since been warned to project scales of evolution and child development onto
each other, this is basically our only extant recourse. Even if the analogy doesn’t hold, the
projection of Donald’s scale onto child development, independently made by Kathrine
Nelson (2007) and Zlatev (2008), may still be of interest, and, while it also depends
largely on indexical procedures, it nevertheless has many more data points to rely on.
Something similar is true about the study of the behaviour in animals similar to human
beings, most notably the great apes. Both these approaches have been used at the Centre
for Cognitive Semiotics at Lund University (e.g. Zlatev et al. 2013; Hribar et al. 2014;
Sonesson and Lenninger 2015; Lenninger et al. 2020). Whether or not these analogies
hold, the results should be interesting in their own right.
Studies of ontogenesis and psychogenesis should not make us forget about microgenesis,
so termed by Heinz Werner, but first studied, using the term ‘Aktualgenese’, by the
Gestalt psychologist Friedrich Sander (see Sonesson 1989: 149; Rosenthal 2004). The
term microgenesis designates the dynamic unfolding and differentiation on a brief time
scale of a percept, a thought, an object of imagination or an expression. Most studies
have, from the beginning, involved perception, but there is no reason to limit the notion
to this domain of experience. Although Sander’s and Werner’s original studies were
experimental, microgenesis also lends itself easily to a phenomenological approach, as
testified by the pioneering work of Aron Gurwitsch (1964), which largely overlaps with
earlier Gestalt psychology.

Methodologies
In many papers, Sonesson has used the phrase ‘the dialectics of phenomenology and
semiotics’ to describe how phenomenology is first called upon to construct an experimental
situation, in such a way that it is relevant to the behaviour of real people, and then
is required to relate the results of the experiment to the situation in the experienced
Lifeworld, all of which forms a circle which may be repeated many times over (see the
section ‘With a Little Help from Phenomenology’). More broadly, Zlatev (2015: 1059)
describes ‘methodological triangulation’ as a characteristic of cognitive semiotics, which
involves the integration of what he calls first-, second- and third-person methods as applied
to any domain, where, however, first- and second-person methods have epistemological
priority. Here the types of methods should, naturally, be understood in analogy to the
three persons of the personal pronouns in Indo-European languages, also construed by
Zlatev as being, respectively, subjective, intersubjective and objective.3 In subordinating
third-person methods, in this sense, to first- and second-person methods, cognitive
semiotics goes against the grain of cognitive science, except in what was characterized
above as its third stage.
This is, of course, not subjectivity and objectivity, as these terms are understood by Prieto
(see the section ‘Cognitive Semiotics in Relation to the Human and Social Sciences’). To
avoid confusion, the terms ipseity, dialogicity and neutrality will be used in the following
(see Table 14.3). More importantly, Sonesson has suggested that we need to go a step
further by distinguishing the modes of access employed and the phenomena on which they
operate, both of which may pertain to the first, second and third person (see Table 14.3).
Both in the case of introspection and in that of the phenomenological reduction, the mode
of access is necessarily (in spite of Dennett 2003) the first person; but in introspection,
we try to discover something about the particular person realizing the operation,
whereas phenomenology is concerned with mapping out the invariant structures of
COGNITIVE SCIENCE AND SEMIOTICS 307

TABLE 14.3 The distinction of ipseity, dialogicity and neutrality in the phenomena
accessed and the modes of access (adapted from Sonesson [submitted], as an
extension of Zlatev 2015: 1059)
Phenomena accessed
Ipseity Dialogicity Neutrality
(First person) (Second person) (Third person)
Ipseity Introspection Empathy Phenomenology
(First person)
Modes of access

Dialogicity Participant observation Dialogue Interactive

(Second person) description
Neutrality Behaviouristic description Interview Experimentation
(Third person) (‘heterophenomenology’)

consciousness, which is to say that it aims to establish some neutral structures. From the
point of view of their modes of access, then, both operations rely on ipseity, but only the
first operation simply aims to access a phenomenon of ipseity.
We can have third-person reports of introspection, which is what Daniel Dennett (2003),
confusingly, calls ‘heterophenomenology’, but a more fruitful third-person approach
which pertains to the third-person domain is experimentation. Both phenomenology and
experimentation aim to attain neutral phenomena, but experimental approaches opt for
a neutral mode of access, while phenomenology uses an access pertaining to ipseity. In
part, experimentation and phenomenology may not be able to attain the same neutral
structures. In part, however, they do, but then phenomenology uses an iconic approach
(in the sense of the section ‘Cognitive Semiotics in Relation to the Human and Social
Sciences’), while experimentation is reduced to an indexical modus operandi. Since
both methods have their intrinsic problems, it is certainly worthwhile employing both
approaches.

STATE-OF-THE-ART AND CURRENT PRIORITIES

In the preface to a Conference volume, Zlatev, Sonesson and Konderek (2016: 9) claimed
that cognitive semiotics could no longer be considered merely an ‘emerging’ discipline,
but that, on the contrary, it already existed. Let us look at the evidence. On the positive
side, we have two institutional facts. An international journal on cognitive semiotics,
(Journal of) Cognitive Semiotics – Multidisciplinary Journal on Meaning and Mind,
was initiated by scholars mostly connected to Aarhus semiotics in 2007. Several issues
were published by Peter Lang Verlag, and, after an interregnum consisting of online
publications, the journal was founded anew in 2014, in collaboration between scholars
from Lund University, Aarhus University and Case Western Reserve University. It has
since then been published by Mouton de Gruyter. The current board consists of Peer
F. Bundgaard, Göran Sonesson, Todd Oakley, Merlin Donald and Bruno Galantolucci.
This revival is connected to a second institutional fact, the creation of the International
Association for Cognitive Semiotics (IACS), founded in 2013, and which has since then
308 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

held three successful conferences, in Lund (2014), Lublin (2016) and Toronto (2018),
while a fourth one, which was planned to occur in Aachen in 2020, had to be postponed
because of the coronavirus, but was finally held online in 2022.
On the negative side, experience from the papers submitted to the journal and
presentations at the conferences suggest that the specificity of cognitive semiotics, as
delineated above, and more broadly as an encounter between cognitive science and
semiotics, has still not been widely assimilated by most scholars. It goes without saying
that the epistemology of cognitive semiotics does not have to be exactly the one sketched
above. But many scholars still have to understand that cognitive semiotics is not identical
to classical semiotics or traditional cognitive science, let alone to linguistics. This brings
us to the first priority for this kind of research, which is precisely for those who aspire to
be part of it to assume the whole intricacy of the conditions attending on the encounter
between the two research traditions.
Another current priority is no doubt to work for a full-scale employment of findings,
models and methods from both interdisciplinary approaches and the traditional disciplines
which they both strive to integrate. There are still models, methods and findings, both
in semiotics and in cognitive science, which have not been fully assumed within the new
research tradition. Finally, another priority should be to achieve a better cross-penetration
of the tool kit and other paraphernalia offered by both traditions. This observation is in
need of a gloss: as it is presented above, cognitive semiotics, unlike most of cognitive
science, and like some parts of semiotics, phenomenology and the traditional human
sciences, privileges an approach from within the Lifeworld, e.g. iconic operations, or, in
other terms, employing first- and second-person modes of access. An encounter between
semiotics and cognitive science may be organized, and to some extent has been done so,
by taking one’s point of departure in traditional cognitive science; but, so far, there do
not seem to exist any epistemological considerations attending to such an encounter.

CONCLUSION
In this chapter, we have presented cognitive science and semiotics as two research
traditions, the encounter of which has resulted in cognitive semiotics. The goal of the
latter is to assimilate the methods, models and findings of both these research traditions
onto a single pool of knowledge. With this aim in view, we have considered a few
milestones in the history of semiotics as a tradition of thought and of cognitive science
as a rather recent, still variegated, paradigm, set up to unify a number of already-existing
sciences. With this point of departure, we have examined what cognitive semiotics is, and
could be, considered as a way of banking on the successes and failures of both traditions
of thought, thus pulling the resources of both traditions together. Any results of such an
exploration can of course only have a preliminary value.
Such an encounter as that between cognitive science and semiotics cannot be realized
without certain decisions having been taken as to the overall epistemology of the ensuing
research tradition. Just as epistemological determinations have come out differently in
the history of semiotics as well as that of cognitive science, as we have seen above, these
issues have been settled in different ways by different advocates of cognitive semiotics.
Since they seem to have been the only scholars to have explicitly and pointedly set up
(partly overlapping) epistemological stakes, Jordan Zlatev and Göran Sonesson have been
our guides in the exposition of cognitive semiotics above. This is not to deny that other
scholars, who have recently published books carrying the mention ‘cognitive semiotics’
COGNITIVE SCIENCE AND SEMIOTICS 309

in their title, have chosen a somewhat different path: thus, Konderak (2018), while
accepting much of the Sonesson-Zlatev lore, in the end puts in a plea for computational
modelling, and Paolucci (2021) bases his cognitive semiotics on Peircean semiotics, with
a touch of late Eco, combined with radical enactivism. As for Brandt (2020), his cognitive
semiotics remains very much a child of French structuralism (even though Brandt, at the
end of his book, claims cognitive semiotics is the very opposite of structuralism).
The justification for focusing on the notion of cognitive semiotics as conceived by
Zlatev and Sonesson (apart from the fact that this entry is written by one of these scholars)
is that, instead of simply importing an epistemology from some pre-existing doctrine,
such as that of Peirce or Greimas, they have attempted to elaborate an epistemology
specifically for cognitive semiotics, taking into account the emphasis on meaning in
semiotics and the plurality of methods in cognitive science. According to this conception,
phenomenology is presumed to be the link between the two earlier research traditions,
since it could be considered the philosophical approach most concerned with meaning,
which is central to semiotics and, also, although often repressed, to cognitive science.
According to our pacesetters, the trajectory of meaning and thus the domains studied by
cognitive semiotics run from the organism to the conjunction of sign systems in society.
Cognitive semiotics uses methods taking their departure in the first, second and third
person, which it tries to combine; though, given its point of departure in meaning, third-
person methods only have an ancillary function. Unlike Zlatev, Sonesson does not want
to mention linguistics as a third ingredient of cognitive semiotics, because it is already
included in the former two. Still, also in his conception, linguistics has a fundamental part
to play in cognitive semiotics. It is the epitome of cognitive semiotics, not for the reason
suggested by Saussure and the French structuralists, but because it fails to conform to the
pattern indicated for scientific disciplines delineated during the classical Methodenstreiten,
according to which the natural sciences are dedicated to setting up rules and regularities,
which they preferably formulate in terms of quantities, while the cultural sciences are
concerned with individual facts, which they study in qualitative approaches. Breaking
this pattern from the start, linguistics is involved with setting up rules, which are mainly
of a qualitative nature. In this respect, linguistics is a paragon for some approaches to
semiotics, and certainly for cognitive semiotics.

NOTES
1 As testified by his website, still persisting after his untimely death in 2010: http://www.
magarinos.com.ar
2 We can certainly argue whether consciousness can appear before living in society, and
whether we should not identify level 2 and 3, but I believe that, in the sense of a perceptual
consciousness, consciousness is much more elementary than society.
3 These three kinds of methods should not be confused with Peirce’s notions of Firstness,
Secondness and Thirdness. In fact, there is only a partial overlap (see Sonesson 2018b).

REFERENCES
Beiser, F. C. (2011), The German Historicist Tradition, Oxford: Oxford University Press.
Bouissac, P. (1999), ‘Le savoir sémiotiques’, Degrés, 25 (100): c1–c15.
­Brandt, P. A. (2020), Cognitive Semiotics: Signs, Mind and Meaning, London: Bloomsbury
Academic.
310 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Bundgaard, P., J. Egholm and M. Skov, eds (2003), Kognitiv semiotik: en antologi om sprog,
betydning og erkendelse, København: Haase.
Cassirer, E. ([1945] 1972), ‘Structuralism in Modern Linguistics’, in Word 1: 99–120. Quoted
from B. Malmberg (ed.), Readings in Modern Linguistics, 8th edn, 78–98, Stockholm:
Scandinavian University Books.
Christensen, P., R. Fusaroli and K. Tylén (2016), ‘Environmental Constraints Shaping
Constituent Order in Emerging Communication Systems’, Cognition, 146: 67–80.
Clark, A. and D. Chalmers (1998), ‘The Extended Mind’, Analysis, 58 (1): 7–19.
Coseriu, E. (1977), Principios de semántica estructural, Madrid: Gredos.
Daddesio, T. (1995), Of Minds and Symbols, Berlin: Mouton de Gruyter.
Deacon, T. (1997), The Symbolic Species, New York: Norton.
Dennett, D. (2003), ‘Who’s on First? Heterophenomenology Explained’, Journal of
Consciousness Studies, 10 (9–10): 19–30.
Diodato, F. (2019), ‘The Neo-Humboldtian Lexical Field Theory. Origin, Reception and
Perspectives’, Paradigmi, 2: 259–74.
Donald, M. (1991), Origins of the Modern Mind. Three Stages in the Evolution of Culture and
Cognition, Cambridge: Harvard University Press.
Donald, M. (2001), A Mind So Rare, New York: Norton.
Donald, M. (2010), ‘The Exographic Revolution: Neuropsychological Sequelae’, in L.
Malafouris and C. Renfrew (eds), The Cognitive Life of Things: Recasting the Boundaries of
the Mind, 71–9, Cambridge: McDonald Institute Monographs.
Dunér, D. and G. Sonesson, eds (2016), Human Lifeworlds: The Cognitive Semiotics of
Cultural Evolution, Frankfurt am Main: Peter Lang.
Eco, U. (1999), Kant and the Platypus: Essays on Language and Cognition, trans. A. McEwen,
London: Secker & Warburg.
Espe, H. (1983), ‘Empirische Analyse visueller Zeichen’, in M. Krampen (ed.), Visuelle
Kommunikation und/oder verbale Kommunikation, 92–121, Hildesheim and Berlin: Olms
Verlag.
Fauconnier, G. and M. Turner (2002), The Way We Think: Conceptual Blending And The
Mind’s Hidden Complexities, New York: Basic Books.
Fillmore, C. (1968), ‘The Case for Case’, in W. Bach and R. Harms (eds), Universals in
Linguistic Theory, 1–88, New York: Holt, Rinehart, and Winston.
Galantucci, B. and S. Garrod (2011), ‘Experimental Semiotics: A Review’, Frontiers in Human
Neuroscience, 5 (11): 1–15.
Gallagher, S. (2005), How the Body Shapes the Mind, Oxford: Clarendon.
Gardner, H. (1985), The Mind’s New Science, New York: Basic books.
Geeraerts, D. and H. Cuyckens (2007), The Oxford Handbook of Cognitive Linguistics, Oxford:
Oxford University Press.
Green, J. (2014). Drawn from the Ground, Cambridge: Cambridge University Press.
Greimas, A. J. (1966), Sémantique structurale: Recherchede méthode, Paris: Larousse.
Gurwitsch, A. (1964), The Field of Consciousness, Pittsburgh: Duquesne University Press.
Habermas, J. (1985), Zur Logik der Sozialwissenschaften: Erweiterte Ausgabe, Frankfurt am
Main: Suhrkamp.
Habermas, J. (1995), Theorie des kommunikativen Handelns, Frankfurt am Main: Suhrkamp.
Houser, N. (1996), ‘Semiotic as Cognitive Science’, Cruzeiro Semiotico, 22/25: 139–50.
­Heis, J. (2018), ‘Neo-Kantianism’, The Stanford Encyclopedia of Philosophy (Summer
2018 edn), ed. E. N. Zalta, https://plato.stanford.edu/archives/sum2018/entries/neo-
kantianism/
COGNITIVE SCIENCE AND SEMIOTICS 311

Hribar, A., J. Call and G. Sonesson (2014), ‘From Sign to Action. Studies in Chimpanzee
Pictorial Competence’, Semiotica, 198: 205–40.
Husserl, E. (1901), Logische Untersuchungen, Tübingen: Niemeyer. Fünfte Auflage 1968.
Husserl, E. (1966), Analysen zur passiven Synthesis: aus Vorlesungs- und
Forschungsmanuskripten 1918–1926, Husserliana: Gesammelte Werke. Bd 11. Haag:
Nijhoff.
Hutto, D. and E. Myin (2013), Radicalizing Enactivism: Basic Minds without Content,
Cambridge, MA: MIT Press.
Käufer, S. and A. Chemero (2015), Phenomenology: An Introduction, Cambridge: Polity Press.
Koch, W. (1971), Varia Semiotica, Hildesheim: Olms Verlag.
Koch, W. (1986), Evolutionäre Kultursemiotik, Bochum: Brockmeyer.
Konderak, P. (2018), Mind, Cognition, Semiosis, Lublin: Maria Curie-Skłodowska University
Press.
Lakoff, G. (1969), On Generative Semantics, Bloomington: Indiana University Linguistics Club.
Lakoff, G. and M. Johnson (1999), Philosophy in the Flesh, New York: Basic Books.
Lenninger, S., T. Persson, J. van de Weijer and G. Sonesson (2020), ‘Mirror, Peephole and
Video: The Role of Contiguity in Children’s Perception of Reference in Iconic Signs’, Front,
Psychol., 14 July 2020. https: //doi.org/10.3389/fpsyg.2020.01622
Lindekens, R. (1971), Eléments pour une sémiotique de la photographie, Paris: Didier/Aimav.
Lotman, J. (1979), ‘Culture as Collective Intellect and Problems of Artificial Intelligence’,
Russian Poetics in translation, 6: 84–96.
Mukařovský, J. (1974), Studien zur strukturalistischen Ästhetik und Poetik, München: Hanser.
Nelson, K. (2007), Young Minds in Social Worlds, Cambridge: Harvard University Press.
Nöth, W. (1977), Dynamik semiotischer Systeme, Stuttgart: Metzlersche Verlagsbuchhandling.
Paolucci, C. (2021), Cognitive Semiotics: Integrating Signs, Minds, Meaning and Cognition,
Berlin and New York: Springer.
Prieto, L. (1975), Essai de linguistique et sémiologie générales, Genève: Droz.
Radnitzky, G. (1970), Contemporary Schools of Metascience, Gothenburg: Akademiförlaget.
Richerson, P. J. and R. Boyd (2005), Not by Genes Alone, Chicago: University of Chicago
Press.
Ricœur, P. (1983–5), Temps et récit, 3 vols, Paris: Seuil.
Rosenthal, V. (2004), ‘Microgenesis, Immediate Experience and Visual Processes in Reading’,
in A. Carsetti (ed.), Seeing, Thinking and Knowing, 221–43, Dordrecht: Kluwer Academic
Publishers.
Rowlands, M. (2010), The New Science of the Mind: From Extended Mind to Embodied
Phenomenology, Cambridge, MA: MIT Press.
Sebeok, T. A. (1991), Semiotics in the United States, Bloomington: Indiana University Press.
Sonesson, G. (1978), Tecken och Handling, Lund: Doxa.
Sonesson, G. (1989), Pictorial Concepts, Lund: Lund University Press.
Sonesson, G. (2009), ‘The View from Husserl’s Lectern: Considerations on the Role of
Phenomenology in Cognitive Semiotics’, Cybernetics and Human Knowing, 16 (3–4):
107–48.
Sonesson, G. (2010), ‘Semiosis and the Elusive Final Interpretant of Understanding’, Semiotica,
179 (1–4): 145–258.
­Sonesson, G. (2013), ‘The Natural History of Branching: Approaches to the Phenomenology of
Firstness, Secondness, and Thirdness’, Signs and Society, 1 (2): 297–326.
Sonesson, G. (2015a), ‘Le jeu de la valeur et du sens’, in A. Biglari (ed.), Valeur. Aux
fondements de la sémiotique, 85–100, Paris: L’Harmattan.
312 BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Sonesson, G. (2015b), ‘From Remembering to Memory by Way of Culture: A Study in

Cognitive Semiotics’, Southern Journal of Semiotics, 5 (1): 25–52.
Sonesson, G. (2016), ‘Epistemological Prolegomena to the Cognitive Semiotics of Evolution
and Development’, Language and Semiotic Studies, 2 (4): 46–99.
Sonesson, G. (2017), ‘Mastering Phenomenological Semiotics with Husserl and Peirce’, in K.
Bankov and P. Cobley (eds), Semiotics and Its Masters, vol. 1, 83–104, Berlin: de Gruyter.
Sonesson, G. (2018a), ‘Meaning Redefined: Reflections on the Scholastic Heritage Conveyed
by John Deely to Contemporary Semiotics’, The American Journal of Semiotics, 34 (1–2):
65–86.
Sonesson, G. (2018b), ‘Beyond the “Tragedy of culture”: In-between Epistemology and
Communication’, The American Journal of Semiotics, 33 (3–4): 141–80.
Sonesson, G. (2019), ‘The Psammetichus Syndrome and Beyond: Five Experimental
Approaches to Meaning-making’, The American Journal of Semiotics, 35 (1–2): 11–32.
Sonesson, G. (2021a), ‘The Relevance of the Encyclopaedia: From Semiosis to Sedimentation
and Back Again’, in N. S. Drăgan (ed.), Differences, Similarities and Meanings. The Interplay
of Differences and Similarities in Communication and Semiotics, Berlin: de Gruyter.
Sonesson, G. (2021b), ‘Nothing Human Is Foreign to Me – Except Some of It. From the
Semiotic Tradition to Cognitive Semiotics by way of Phenomenology’, Degrés, 186–7:
b1–b23.
Sonesson, G. and S. Lenninger (2015), ‘The Psychological Development of Semiotic
Competence: From the Window to the Movie by way of the Mirror’, Cognitive
Development, 36: 191–201.
Stjernfelt, F. (2007), Diagrammatology: An Investigation on the Borderlines of Phenomenology,
Ontology, and Semiotics, Dordrecht: Springer.
Thompson, E. (2007), Mind in Life, Cambridge: The Belnap Press.
Vico, G. ([1710] 1988), On the Most Ancient Wisdom of the Italians, Unearthed from the
Origins of the Latin Language, Ithaca and London: Cornell University Press.
Von Uexküll, J. (1973), Theoretische Biologie, Frankfurt/M: Suhrkamp.
Zahavi, D. (2005), Subjectivity and Selfhood, Cambridge: The MIT Press.
Zlatev, J. (2008), ‘The Co-evolution of Intersubjectivity and Bodily Mimesis’, in J. Zlatev, T.
Racine, C. Sinha and E. Itkonen (eds), The Shared Mind: Perspectives on Intersubjectivity,
215–44, Amsterdam: John Benjamins.
Zlatev, J. (2009), ‘The Semiotic Hierarchy: Life, Consciousness, Signs and Language’, Cognitive
Semiotics, 4: 169–200.
Zlatev, J. (2015), ‘Cognitive Semiotics’, in P. Trifonas (ed.), International Handbook of
Semiotics, 1043–67, Dordrecht: Springer.
Zlatev, J., E. Alenkaer Madsen, S. Lenninger, T. Persson, S. Sayehli, G. Sonesson and J. van
de Weijer (2013), ‘Understanding Communicative Intentions and Semiotic Vehicles by
Children and Chimpanzees’, Cognitive Development, 28: 312–29.
Zlatev, J., G. Sonesson and P. Konderak (eds) (2016), Meaning, Mind and Communication:
Explorations in Cognitive Semiotics, Frankfurt am Main: Peter Lang.
 INDEX

abduction 57–8, 128–9, 134, 139 n.7 Bresciani, F. E. 91

Actor Network Theory (Latour) 52 Buber, M. 123
adaptation 156 Bühler, K. 187, 190
Adorno, T. W. 50 Bulwer, J. 185
agenda-setting theory 54 Bundgaard, P. F. 10, 119 n.10
Aktualgenese 306 Bunge, M. 91
American Sign Language (ASL) 278 ­Byrne, T. 104
Anderson, B. 74–5
anticipatory system 95–6 Calbris, G. 186
Aquiar, D. 150 Cannizzaro, S. 4
Arnheim, R. 193 canon of use 218–20
Arnold, M. 50 Cantril, H. 47–8, 53
artificial intelligence (AI) 57–9, 204, 294 Capozzi, R. 137
artificial semiotic system 96–7 Carnap, R. 5
augmented reality (AR) 59 Casasanto, D. 119 n.10
Augustine, St. 10, 126 Cassirer, E. 15–16, 19, 299
Austin, J. L. 116 categorial signs 110–11
authority 49, 169, 174, 260 Centre for Contemporary Cultural Studies
autocommunication 155 (CCCS) 50–1
Centre for Mass Communication Studies
Bakhtin, M. 49, 59, 152–3, 232 (CECMAS) 47, 49
Barthes, R. 5–6, 47, 49, 55–7, 152, 154, 163, Chalmers, D. 294–5
170, 217, 239 Cheney-Lippold, J. 53
base space 173–4 Chesterman, A. 147
Bastin, G. 156 chiasm model 28–9, 31–3, 35
Bateson, G. 17–19, 25 Chomsky, N. 51
Baudrillard, J. 59 Cienki, A. 192, 198
Becoming a Translator (Robinson) 150 Ciula, A. 69, 72, 74
Bildung (Gadamer) 130 Clark, A. 294–5
binary analogue model 37–8 Clark, H. H. 193
biosemiotics 2, 9, 68, 74, 78–80, 150, 156, classic media semiotics 55–7
261–4 closed system 92
blending mode 173–4, 176–8 Cobley, P. 78
Boehm, G. 73 cognition
bonding icons 239–40 cognitive assemblages 55
Botev, H. 277 cognitive psychology 107, 116–17, 232,
Bouissac, P. 296 283
Bounce 225–6 cognitive revolution 7
Boutet, D. 192 and iconicity 270, 278
boyd, d. 67 cognitive linguistics (CL) 7, 116, 187, 269–71,
Brandt, L. 180 n.28 286–7
Brandt, P. A. 7, 10, 179 n.12, 180 n.27, 297, construal 271–4
309 iconicity 274–9
314 INDEX

metaphor 279–83. See also conceptual cultural Lifeworld 304–5

metaphor theory (CMT) cultural text 154
multimodality and polysemiosis 283–6 Culture and Anarchy (Arnold) 50
cognitive science 293, 307–9 curvature 224
approach 296–8 cyberculture 57–8
domain of study 303–4 cyberspace 49, 53, 58–9, 61–2
evolution and development 304–6
and experimental semiotics 300–1 Daddesio, T. 297
human and social sciences 301–3 Danesi, M. 2, 9, 72, 74
linguistics 298–300 Dawkins, R. 60
methodologies 306–7 Deacon, T. 295
phenomenology 300 Debrun, M. 94
semiotic tradition 295–6 decodification 128
stages of 294–5 decoding 23, 27, 37, 55
cognitive semiotics 7, 12, 195, 270–1, 280, deduction 139 n.7
282, 284, 286, 297–8, 300–9 Deely, J. 33, 127, 129, 140 n.12, 261–2
Cohen, J. 222 De Kerckhove, D. 61
Colino, S. 222 Deledalle-Rhodes, J. 149
Comay, R. 20 Deleuze, G. 52, 55
communication 15–16, 258–9 Dennett, D. 307
function 110, 117, 190, 233 depiction 17, 28, 37, 169, 175, 284–6
Gusdorf discourse model 22–4 Derrida, J. 56, 126
­human discourse model 19–22 deterministic system 93
human observer model 17–19 Devylder, S. 274, 276, 283
and information 16 diagrammatic iconicity 197–8, 275–7, 282
matrix 28–31, 34–5, 38–40 diagrams 68, 73, 167–8, 170, 172, 179 n.14,
scholastic trivium discourse model 24–5 179 n.17, 197–8
trope model 27–33, 36, 39–40 dialogicity 306–7
competence 15, 37, 60, 75–6, 78–9 dialogue functions 129
complex system 93–4 digital humanities (DH) 67–70
compositionality 93, 96–7, 108, 112, 233, ­biosemiotic approach 78–9
237–8 ecosemiotics 79–80
computational approach 96–7 embodiment 74
conceptual meaning 164, 166, 172, 274 existential graphs 73–4
conceptual metaphor theory (CMT) 279, 282 iconicity 72–3
consciousness 7, 9–10, 12, 15–17, 59, 91, industry 77
113–14, 117–18, 119 n.12, 125, 128, literacy 74–6
130–3, 170, 273, 293–5, 297, 302–5, multimodality 70–1
307 translation 71–2
construals 11, 70, 72, 75, 104, 108, 113, 115, digitalization 67, 71, 73–7
203–4, 270–6, 278, 281, 286 digital literacy 59, 75
contiguity 198–200 Dilthey, W. 132
conventionality 201 discourse analysis 231–3
conventional system 279 discursive practices 221
Cornips, L. 263 dismal science 303
Courtés, J. 151–2 Dockhorn, K. 130
Crisis of the European Sciences and Domani 137–8
Transcendental Phenomenology, The Donald, M. 52, 55, 295, 304–6
(Husserl) 135 D’Ottaviano, I. M. L. 91
critical discourse analysis (CDA) 6 double articulation 284
Croce, B. 254 Duteil-Mougel, C. 152
cultivation theory 54 dynamical systems 92, 97
INDEX 315

ecosemiotics 68, 79–80 genuine signs 104, 106, 111–12

Eco, U. 10, 22, 59, 136, 139 n.4, 139 n.10, Gerbner, G. 54
153–4, 156 gesture 183–5
idiolect 136 co-speech 190–2, 194–5
modes of sign production 135 coverbal 191
Numero Zero 123, 137–8 empirical research 202–4
paramount 136–7 first wave 185
postmodernity 129 kinesic, visual and imagined facets
Theory of Semiotics, A 133–7 192–3
Efron, D. 185 multi-functionality and polysemy 190–1
Eide, Ø. 69, 72, 74 Peircean approaches to 195–202
Ekman, P. 185 pointing 190–1
ELAN 202 second wave 185–6
Elleström, L. 72–4 and semiotic theory 195
Elliot, A. 222 sign formation 193–4
emblematic gestures 189–90 sign language 194–5
embodied intersubjectivity 282 third wave 186–7
embodied level 280–2 verbo-gestural sign process 188–90
embodiment 17, 74, 131, 295 Gibson, J. 295
encoding 27, 37 global discourse 60
enunciation 111, 164, 169–70, 174, 176, 178, global semiotics 261–2, 264
179 nn.14, 18–19 goal-based behaviour 96
Espe, H. 296 Goethals, G. 150
‘Evolutionary Love’ (Peirce) 139 n.9 Gorlée, D. 149–50, 157
evolutionary system 94–5 Gottlieb, H. 145
existential graphs 71, 73–4, 76, 78 Grady, J. E. 279
expansion 224 Gramsci, A. 51
experimental semiotics 300–1 Greimas, A. J. 10, 33, 40, 103, 118, 147,
expressions 105–6, 109 151–2, 163, 165, 167
Grondin, J. 4, 124–6, 139 n.3
Fabbri, P. 153, 157 Gross, L. 54
Fairclough, N. 6 Guattari, F. 52, 55
Ferraris, M. 124–7, 139 n.4 Gurwitsch, A. 306
­Fetzer, J. H. 87 Gusdorf, G. 22–4
Feuerbach, L. 303
Fillmore, C. 299 Hadlington, L. 53
Fiore, Q. 48 Halliday, M. A. K. 11, 146, 215, 217, 223,
Floreano, D. 87 231–4, 248
Floridi, L. 77 ­hardcore reality 262, 264
Foucault, M. 6, 23, 25, 32, 56, 220 Harris, R. 254–65
framing 237 Hartama-Heinonen, R. 147, 150
Frankfurt School 50–1 Hartley, J. 67, 75, 77, 80
Friesen, W. 185 hashtag 51, 60
functional cycle 303 Haviland, J. B. 186–7
fusion 152 Hawreliak, J. 56–7
Hayles, K. 55
Gadamer 124–7, 129–35, 137–8, 139 n.3 Hegel, F. 295
play, notion of 131–2 hegemony theory 51
Galantucci, B. 301 Heidegger, M. 4, 22, 24, 34–5, 37–8, 40, 58,
Gamer, E. -C. 156 124–6, 130–2
Garrod, S. 301 Heine, B. 287 n.4
general systems theory 90–2 Herman, E. 51
316 INDEX

hermeneutics 10, 123–7, 129–30, 132–3, 137, inter-personal communication 17, 27

302 interpersonal meaning 233, 237, 246
heterocommunication 155 interpictorial translation 156
heterophenomenology 307 intersemiotic translation 155–7
Hinnell, J. 202–3 intertextuality 151–2
Hippocrates 8–9 intimating function 106, 109–10, 115–16
History of Hermeneutics (Ferraris) 126 intimation 106, 111
Hjelmslev, L. 103, 163, 178 n.6, 239, 296 intra-group communication 17, 27
Hodge, B. 215 intra-personal communication 17, 27
Hoffmeyer, J. 80 intrapictorial translation 156
Holenstein, E. 103, 115 Invasion from Mars, The: A Study in the
Horkheimer, M. 50 Psychology of Panic (Cantril) 47
Houser, N. 297 ipseity 306–7
human beings 15–17, 32, 34, 40, 48, 59, 74, isomorphism 135, 196, 275, 278
95, 105, 110, 126, 128, 131, 258–9, Itkonen, E. 275, 278
262, 264, 293–4, 301–2, 304–6
human discourse model 19–22 Jakobson, R. 2, 10, 15, 17, 22–3, 25–6,
humanism 126, 256 71, 103, 143–7, 150–7, 199, 201,
humanity 50, 77, 264–5 270, 274–5, 287 n.2. See also
human observer model 17–19 communication
Husserl 4, 10, 21–4, 27, 31–3, 37, 103–7, chiasm model 28–9, 31–3
109–15, 118 nn.2–3, 119 nn.11–12, elements and functions 27–31
125, 135, 273, 295, 297, 300, 302–4 logic and semiotic squares 33–4
phenomenological semiotics 115–18 scientifically cautious 151
Hutton, C. 260, 294 Jenkins, H. 58
hypertextuality 129 Johnson, M. 279, 294–5
Hypodermic Needle Theory (HNT) 53–4 Jones, P. 6
Jourdain effect 1–3, 7–8
iconicity 57, 71–4, 78–9, 196, 203, 270,
274–80, 282, 284, 286, 287 n.2 Kant, I. 130–1
of distance 276 Katz, E. 53–4
of quantity 276 Keane, W. 255–6
iconic operation 302 Kees, W. 7
iconization 169, 239 Keller, R. 253
icons 16, 25, 72, 88–9, 128, 139 n.8, 149, Kendon, A. 186, 189, 192–3
167–9, 196–7, 199, 239–40 Koch, W. 296
ideational meaning 233, 239 Konderak, P. 309
Iedema, R. 241 Kourdis, E. 156
imagistic iconicity 275 Kress, G. 215, 217, 231–3, 237–9
indexicality 199–200, 203, 284 Kristeva, J. 32, 151–2
indications 105–6, 110–11, 113 Kukkonen, P. 148
Information and Communication Kull, K. 79, 150, 156
Technologies (ICTs) 77
information value 237 Lacković, N. 75
Ingarden, R. 119 n.13 Lakoff, G. 279, 294–5, 299–300
Ingram, H. J. 240 Landowski, E. 165, 167
­integral linguistics 280 Langacker, R. W. 272–3, 275, 278
integrationism 6, 11, 254, 256, 261–5 Langlois, G. 52
intelligibility 37–8, 40 Language of Displayed Art, The (O’Toole) 217
inter-group communication 18, 27 language-processing 107
intericonicity 156 Lanigan, R. L. 9, 29, 32–3, 38, 133
International Association for Cognitive Lasswell, H. D. 50
Semiotics (IACS) 307–8 ­Latour, B. 51–2
INDEX 317

Lazarsfeld, P. F. 53–4 metaphor 20, 22, 32–3, 56, 137, 157, 173–5,
Lazzarato, M. 52 187, 190–1, 197–201, 269, 271, 274,
Leavis, F. R. 50 279–86
Le Bourgeois gentilhomme (The Bourgeois Methodenstreit 299, 309
Gentleman, Molière) 1 metonymy 20, 22–3, 33, 174–5, 199–200,
Lederer, M. 152 203
Lemke, J. L. 248 mimetic memory 304
Levinas, E. 125 ­Mind’s New Science, The (Gardner) 294
Lévi-Strauss, C. 58 Minsky, M. 57
Levy, A. 51 Mitchell, W. J. T. 73
Lindekens, R. 296, 300 Mittelberg, I. 192, 196, 199, 201, 203
linguistic metaphors 280, 283 Molière 1
linguistics 298–300 monomodal model 74, 198, 215, 241
linguistic signs 69, 105, 111–15, 118, 119 Morgenstern, A. 192
n.10, 119 n.12, 156, 184, 188, 190, Morris, C. 5, 7
201, 253–5, 257, 260, 265, 274 Moscow-Tartu School. See Tartu-Moscow
Lippmann, W. 50 School
liquidity 225 Mossop, B. 145
literacy 9, 74–7, 80 motivation 281
local discourse 60 Motivation & Sedimentation Model (MSM)
Logical Investigations (Husserl) 103–4, 110, 280, 282–3
113, 118 n.1 Möttönen, T. 273
Lotman, J. 153–5, 295 Müller, C. 187, 193, 198
Lotman, Y. 20, 77, 79 multimodal discourse analysis (MDA) 231–5
Lupyan, G. 119 n.10 Clarifai algorithm 242, 247–8
digital approaches 241–8
McCarty, W. 79 IBM Watson algorithm 241–2, 247–8
McCombs, M. 54 Just Terror 236–8, 240–1, 243–5
McGurk effect 286 K-modes algorithm 242, 245
machinic phylum 52 semiotic sciences 248–9
McLuhan, M. 9, 48–9, 52, 59, 61–2 multimodality 4–5, 11, 68, 70–1, 73–5,
McNeill, D. 186, 196–7 188–91, 215–17, 227, 231–2. See also
Marais, K. 150 polysemiosis
Marx, K. 303 affordances 219–20
mass culture 49–50 canon of use 218–20
meaning-fulfilling acts 106–11, 116–17 discourses 220–1
meaning-giving acts 105–6, 108–10, 115–16 discursive practices 221
Mechanical Bride (McLuhan) 49 food packaging 221–3
media 47–8, 61–2 inventory 223–4
classic 55–7 language 192–5
convergent 58 and polysemiosis 283–6
mass 47–51, 53–5, 57–8 resources and choices 217–18
new media 51–3, 55, 57–61 semiotic materials 218–20
origins 48–9 textures 224–7
psychological approach 53–5 music 171–2
Medium is the Massage, The: An Inventory of Myin, E. 294
Effects (McLuhan and Fiore) 48–9 mythic memory 304–5
memes 60–1 Mythologies (Barthes) 55
mental software 55
Merleau-Ponty, M. 22, 25, 32, 34–5, 37, 104, Nadin, M. 95
117, 295 natural-artificial 225
meta-analysis 273, 296, 302 natural Lifeworld 303, 305
metafunctions 233–5, 242, 246–8 Nelson, K. 306
318 INDEX

Nesselroth, P. W. 56 postmodernity 129

neutrality 306–7 post-structuralism 56
new media 51–3, 55, 57–61 pragmatics 163
Noelle-Neumann, E. 54 of meaning 166–9
Nolfi, S. 87 of music 171–2
non-categorial sign 111 semiosis and diegesis 165–6
non-deterministic system 93 and semiotics 163–5
non-linguistic sign 10, 111, 151, 157, 255, of tropes 172–7
257, 265, 275, 278, 280–1, 286 Prieto, L. 301–2, 306
Norris, C. 2 propaganda theory 51
Nöth, W. 296 Public Opinion (Lippmann) 50

objective construal 272–3 Queiroz, J. 150

­Observer Model of Human Communication,
The (Ruesch and Bateson) 17 radical indeterminacy 260
O’Halloran, K. 68–9, 71, 77, 234 Rastier, F. 152
Ong, W. 60 Reading Images (Kress and Van Leeuwen) 217
open system 92 referential meaning 164, 166, 172
organization 25, 91–2, 94–6, 98 n.1 regularity 224–5
orientation 224 ­relevance 37–8, 40, 72, 104, 110–12, 116,
O’Toole, M. 217, 231–4 133, 280
relief 225
Palmer, R. E. 124–5, 130, 139 n.2 Ricoeur, P. 127
Paolucci, C. 297, 309 Rifkin, J. 79
Papert, S. 57 rigidity 225
Pareyson, L. 133, 139 n.4, 140 n.14 Robinson, D. 148–50
Paris School of Semiotics 10, 103, 144, 151–3, Rosenblatt, F. 57
232 Rosen, R. 94–5
Pasquinelli, M. 57–8 Rowlands, M. 295
Peirce, C. S. 10–11, 15, 196, 199, 201, 261 Ruda, F. 20
Firstness 95, 128, 302 Ruesch, J. 7, 17–19, 25
to gesture 195–202
ground 195–6 salience 237
interpretant 25, 89, 149, 195, 261 transfer 54
logic typology 22 Sander, F. 306
process semiotics 88–9 Saussure, F. de
representamen 25, 129, 140 n.13, 195 communication model 263
Secondness 95–6, 128, 302 différence 55–6
Thirdness 95–6, 128, 302 human capacity 254
translation 143–4, 148–50 idealism 254, 263
Pennycook, A. 259 integrational linguistics 255–6
Perspectives Model of Communication schemata 201
(Lanigan) 29 Schleiermacher, F. 132
Petrilli, S. 127–8, 139 nn.5–6, 11, 157 Schmid, H. J. 287 n.4
phenomenology 300 scholastic trivium discourse model 21, 24–5
phonetics 163–4, 171 Schütz, A. 37
phonic iconicity 275 SCI-MX 225–6
polysemiosis 283–6 Sebeok, T. A. 2, 7, 33, 78–9, 139 n.5, 157,
polysemiotic approach 145, 285 261, 263–5, 296–7
Ponzio, A. 127–8, 139 nn.5, 11 secondary orality 60
Popovič, A. 158 n.1 sedimentation 273, 280–2
p-organization 91–2, 94 Selfish Gene, The (Dawkins) 60
INDEX 319

self-maintaining system 90 signs 4, 11, 87–8, 104–10. See also linguistic

self-organized system 92, 94–5 signs
Semantics of Science, The (Harris) 257, 262–3 and activities 256–61
Semio-linguistic Research Group. See Paris categorial 110–11
School of Semiotics co-speech gesture and 194–5
semiosis 2–6, 92, 96–7, 123–4, 128–30, definition 88
132–8, 149–56, 163–4, 204, 248, 296. genuine 104, 106, 111–12
See also polysemiosis gestural 167, 184, 186–94
agency in 8, 134 humanity of 264–5
and cognition 118 meaning formation 117–18
connotative 167 non-categorial 111
diagrammatic 168 non-linguistic 10, 111, 151, 157, 255, 257,
dialogic, dialogue and 124, 137–8 265, 275, 278, 280–1, 286
ethics in 128 ontologies 253–6
hermeneutics and 130 Peircean 195–202, 258, 261, 264
human acts of 58 picture and 131
intersemiosis 151–2, 154–6, 231 production 128, 134–8, 149
linguistic 164 signal 110–13
as meaning making 69 surrogational 260
mediated 49, 52 system 284–5
meta-semiosis 178 n.6 verbo-gestural 188–90
multimodal 183, 188, 283, 285 Simon, H. 294
and music 172, 231 simulation 96–8, 107–9, 116–18
patterns of 52 situated level 280–1
Peircean 87–8, 143, 149, 261, 269 Snell-Hornby, M. 145, 147
­and process 89, 150 social media 60–1, 248–9
recursion of 177 social semiotics 68, 70–1, 74, 215–17, 220–1,
restricted 116 231–5, 239, 248–9, 283
rhetorical 164 Sonesson, G. 155–6, 196, 294, 296, 298, 302,
social 297 304, 306, 308–9
and translation 149–50 s-organization 91, 94
unlimited, infinite 134, 139 n.1, 261 Sources of Hermeneutics (Grondin) 125
urban 67 Sowa, J. F. 73
Semiotic Animal (Deely) 129 ­spiral of silence theory 54
‘Semiotic Modalities in Translation Causality’ Stecconi, U. 150
(Chesterman) 147 Stjernfelt, F. 118, 297
semiotic(s) 146. See also biosemiotics; Streeck, J. 187, 193
cognitive semiotics; ecosemiotics; subjective construal 273
semiosis; signs; social semiotics supercategories 262–3
generations 147–8 Sütiste, E. 145, 155
ideologies 253–7 Sweetser, E. 180 n.27
materials 218–20 symbols 89, 96, 131, 135, 139 n.8, 167–70,
reflexivity 255 201, 239, 274–5
systems 87 System Functional Theory (SFT) 233–4, 239
Semiotics and the Problem of Translation systemic functional multimodal discourse
(Gorlée) 149 analysis perspective (SF-MDA) 234
Semiotics Unbounded (Petrilli and Ponzio) systems theory 89–92
127
semiotranslation 149–50 Tan, S. 234
sensus communis (Gadamer) 130–1 Tarasti, E. 147
Shaw, D. 54 Tartu-Moscow School 10, 144, 153–5, 232
signal systems 284 Tawheed gesture 238, 240
320 INDEX

textual meaning 154, 233, 246, 248 utterance 136, 153, 164–7, 169, 176–7, 180
theoretic culture 305 n.21, 185–6, 188, 190–2, 196, 198,
Theory of Linguistic Signs, A (Keller) 253 202–4
3D motion capture (MoCap) 203–4
Torop, P. 155 Vandelanotte, L. 283–4
Torres-Martínez, S. 150 van Langendonck, W. 275, 280
total translation 155 Van Leeuwen, T. 215, 217, 224, 231–3,
Toury, G. 143, 145 237–9
transduction 156–7 Veale, T. 204
translatability 151 Verhagen, A. 273
translation 71–2, 143–4 Vichean-Prietean supermaxim 302
intersemiotic 155–7 Vico, G. 130, 136, 301–2
Moscow-Tartu School 153–5 Vimercate 137–8
Paris School 151–3 Visual Rhetoric (VR) 56–7
Peircean account of 148–50 von Bertalanffy, L. 89–90
research 146–8
scholars 144–6 ­weight 224
transposition 151 Werner, H. 306
trivium model 21, 24–5, 33 Wicke, P. 204
trope model 27–33, 36, 39–40, 172–7 Wiener, N. 19
Truth and Method (Gadamer) 126, Wignell, P. 234, 239–40
130–3 Wilcox, S. 270, 278
Turns of Translation Studies, The: New Wundt, W. 185
Paradigms or Shifting Viewpoints?
(Snell-Hornby) 145 Yoka, C. 156

Uexküll, J. von 79, 262, 303 Zethsen, K. 145

uses and gratifications theory 54 Zlatev, J. 201, 273, 298, 303–4, 306, 308–9
Uspenskij, B. -A. 154 Zwaan, R. 107–8
321
322
323
324