The Universe Brain GIF by by Percolate Galactic

Textual literacy differs from visual and aural literacy in the sense that there is no genetic basis for it. While we are already born with the capacity to see, hear, and feel, we have to invest a considerable amount of time and effort to learn to read and write (Lutkewitte, 2020, p. 139). The fact that the ability to produce and process text is not given may account for the high status that has been attributed to textual literacy(Lutkewitte, 2020, pp. 17-18). Our learning processes, however, clearly involve multiple literacies that are at work at the same time. That we are not able to grasp at once how simultaneous multiple processes interact, tells us something about the limits of our intellectual capacity to understand how complex systems work. Within the paradigm of knowledge of science it is a known principle that we have to reduce this complexity to be able to understand it. The German philosopher Henrich Rickert attributes scientists the important pedagogical task of translating complex systems into understandable terms. Rickert sees the world as a ‘heterogenous continuum’, with one phenomenon flowing smoothly into another without clear boundaries. To be able to ‘know’ the world, scholars have to set boundaries to this continuum and simplify the appearance of its components. This means that even if we are looking at a scientific work or representation, it is always a simplification of a phenomenon (Knobl, 2015).

Below you can see an example of such a simplification: the way in which the functioning of our senses in interaction with specific areas in our brain is represented.

Brain regions by Biologist. Click image to enlarge.

In real time, hearing and vision do not function independently from each other. We do have neurons that can only process one sense at a time in the temporal and occipital lobe. Yet, the way in which we perceive the world around us is the result of the capacity of our brain to integrate different kinds of stimuli through neurons that respond to multiple stimuli (Spence, Senowski and Röder, 2009). This takes place in so-called ‘convergence zones’. The integration of unimodal stimuli coming from our ears, eyes, and skin ensures that our experience is not fragmented, but perceived as a unitary representation of the world (Lachs, 2021). This process is called ‘multisensory integration’.

Multisensory integration

Watch the clip in which PhD student Naomi Haffner from the University of Bath (UK) explains how the perception of reality is influenced by the interaction of our senses. She first illustrates how our ears can be tricked by our eyes with the McGurk effect.

Naomi Haffner on the McGUrk Effect

Social processing

Her second example of multisensory integration shows how crucial the mix of sensory experiences and modes of communication is in contexts of social processing. We attribute meaning to another person’s behaviour through the use of our sight, hearing, smell and touch, that are able to translate several modes of communication - pitch, scent, facial expression, bodily movement, speech and bodily contact - into meaning.

Reducing complexity to better understand how processes work

The reason for explaining how our brain integrates stimuli from diverse senses in a course about multimodality is to underline the fact that processes of perception and communication always involve multiple actions that occur at the same time. We single out each of the five senses when learning about how we perceive the world, and we juxtapose multimodality to monomodality in order to reduce the complexity of how these processes work and lower the threshold for understanding. But a unimodal stimulus alone cannot produce meaning, just as monomodal modes of communication are non existent. There is only Rickert’s ‘ heterogenous continuum’ from which we can isolate particular elements that interact to better understand and explain how they work.
Let’s try and project this principle to a case study:

"Imagine that you are at a party, drinking, chatting, looking around, balancing on your high heels, glancing at a mirror to see how your make-up and dress look, waiting for someone you know to turn up. Within this context you notice the scent of a perfume."

We now single out the processes that are set in motion by the stimulus of the perfume - and ignore all the other things that are going on around you at the party. The scent is captured by olfactory sensory neurons situated in the cavity of our nose and is then transmitted to our brain (Olfactory system, wikipedia, 2021). But it can only become meaningful by integrating it with what we see and ask ourselves:

• whether can we identify a person or a piece of clothing with the memories that it triggers;
• have we ourselves ever used it and do we like it;
• do we know someone who uses it, what is our relation to that person;
• do we know what the bottle and label look like;
• can we remember an advertisement about it?

And then compare with the feelings it triggers, such as attraction, inspiration, relaxation, repulsion, and our knowledge of it - the name or brand, where it can be bought, whether it is expensive. All this takes place at the same time.

To get an impression of the many processes that are ongoing in our brain, besides processing the stimuli of our 5 senses, have a look at this GIF:

Animation Brain GIF by  the Explainer Studio, 2021

Thinking of communication and perception in terms of a series of continuous complex processes can make us more conscious of the fact that we may have tacit aural, visual and multimodal skills that are involved in these processes, but that we usually take them for granted. The move from a textual to a multimodal information culture may increase the relevance of these skills in such a way that teaching of visual and aural literacies could become an essential part of the standard curriculum of the 21st century (Kress, 2003).

References

• Haffner, N. [https://www.senselesspsych.com]. (2020, December 14). Better Together: Combining the Senses in Social Processing [Video]. YouTube. https://youtu.be/2889YSjGaos
• Lutkewitte, C. (2020). writing in a technological world [E-book]. https://doi.org/10.4324/9780429507014
• Knobl, W (2015). The origins of social science and the problem of conceptualizing ‘Modernity’/’Modernities’ In: R. Weber & S. Trakulhun, (Eds.), Delimiting Modernities: Conceptual Challenges and Regional Responses, Lexington books (pp. 79–96). https://idiscover.lib.cam.ac.uk/permalink/f/16u99e0/44CAM_ALMA21299544580003606
• Kress, G. R. (2010). Multimodality : a social semiotic approach to contemporary communication.
• Lachs, L. (2021). Multi-modal perception. In R. Biswas-Diener & E. Diener (Eds), Noba textbook series: Psychology.Champaign, IL: DEF publishers. Retrieved from http://noba.to/cezw4qyn
• Olfactory system. (2021, May 25). In https://en.wikipedia.org/wiki/Main_Page. https://web.archive.org/web/20210525035316/https://en.wikipedia.org/wiki/Olfactory_system
• http://theexplainerstudio.com/. (2021, December 1). Animation Brain GIF By The Explainer Studio [Gif]. Https://Giphy.Com. https://media.giphy.com/media/mqhvMx0PlB6qg5LN1d/giphy.gif
• Spence, C., Senkowski, D., & Röder, B. (2009). Crossmodal processing. Experimental Brain Research, 198. https://doi.org/10.1007/s00221-009-1973-4
• Szymik, B.(2011, May 09). What's Your Brain Doing?. ASU - Ask A Biologist. Retrieved June 8, 2021 from https://askabiologist.asu.edu/brain-regions