Multimodality

Having come across the word Multimodality in Eric Champion’s book, I am now distracted from that work, while I follow this particular White Rabbit through the galleries of Museum Studies. Right now, I am reading Tiina Roppola’s book, Designing for the Museum Experience, which seems to be the most worked through discussion of multimodality in museums and cultural heritage sites.

But first, let us enquire at the nexus of all knowledge. Wikipedia (today at least) describes multimodality thus:

In its most basic sense, multimodality is a theory of communication and social semiotics.

However, this page also says “This article has multiple issues. Please help improve it or discuss these issues on the talk page. This article may be too technical for most readers to understand. This article needs attention from an expert in Media. The specific problem is: lead difficult to read for those not already familiar with the subject.” So… lets try and pull together something more understandable here.

Semiotics is something we can all get our heads around (I hope?) being the study of how we use signs (like words) to stand for things (like objects or ideas). So for example, the word “banana” can stand for the curved yellow/green object that we can find in the fruit bowl. But, the object itself, the banana, can also be a sign for the Caribbean islands, or tropical climates, or multinational food conglomerates.

Now, to quote Roppola (page 51): “Rather than a traditional Western view privileging language, a multimodal lens equally values the written and spoken word, still and moving images, sound and music, spatial and architectural arrangements, gesture and and gaze, and any other culturally-patterned ‘modes’ of expression. Modes are considered semiotic when used by given communities ‘with discernable regularity, consistency and shared assumptions about their meaning potentials.'” (Here, she is quoting Gunther Kress “What is Mode” in The Routledge Handbook of Multimodal Analysis (2009).)  That said, even language itself is multimodal, consider the same word spoken with different tones of voice, for example. And tone isn’t the only thing that changes the meaning of a word in conversation, consider the impact of gesture, gaze, spacial relations and even costume.

Roppola observes that museum interpretation is thus multimodal, with the visitor’s gaze switching back and forth between object and label, drawing meaning from juxtaposition of objects, and how they are arranged in space, as well as sound effects and audiovisual. But this begs a couple of significant questions:

Isn’t this just “multimedia semiotics”? What’s the difference between multimedia and multimodality?

Here, Roppola is on shakier ground. Though she makes the point that multimedia is often understood only as computer based multimedia, and that “a book is a medium as much as a computer screen” this doesn’t seem a strong enough distinction to justify using a statement such as “a museum exhibition is a multimodal experience” in preference to “a museum exhibition is a multimedia experience.” Especially as Champion claims that computer based games are (or can be) multimodal.

According to this article by Iedema, one thing that differentiates multimodality from mere multimedia is “first, the de-centring of language as favoured meaning making; and second, the re-visiting and blurring of the traditional boundaries between and roles allocated to language, image, page layout, document design, and so on.” Iedema argues that a multimodal approach to the study of discourse is essential, whatever the medium of communication, unless the design choices are so limited (by for example, publication in an academic journal) that only the discourse is distinctive.

So it seems that museums are not uniquely multimodal, compared to any other discourse, but they may be more multimodal than some, because of all the design choices, made around constructing the discourse. And of course we should be clear that, really the multimodality is inherent in the analysis of the museum, not in its nature.

However, Iedema specifically addresses museums displays “as semiotic constructs through the deliberations of writers, filmmakers, planners and builders. It is this kind of perspective which is important for revealing, describing and understanding representation as a truly multimodal construct, embodying not merely the sounds and images which we see, but also all the semiotics, the coincidences and the compromises which played a role in its inception.” He calls this a process of “resemiotisation.”

Iedema is the sort of writer that eschews common words, when one of his own creation might be used instead – ‘supra-logogenetic’ anyone? – but I think I understand where he is coming from…

Back to Tiina Roppola, who argues that although researchers have applied multimodal analysis to museums, “most of this work omits the the study of those who are ultimately making meaning: museum visitors.”