Tag Archives: Binding (neuropsychology)

Sound married to vision is the completely normal and sometimes entertaining way of perceiving the world

I’ve got to laugh on the odd time that I read a description of synaesthesia that makes it sound like some kind of mental disorder or abnormality of sensory perception. Sure enough, synaesthesia concurrents are perceptions of sensory experiences that are not triggered by things happening outside of the mind. They are experiences (not always sensory) triggered by activity happening within the mind (just like the meanderings of your normal constant train of thought). A synaesthete can experience sound as a synaesthesia concurrent triggered by a visual experience (the synaesthesia inducer), and might also experience a visual concurrent triggered by a sound inducer. Coloured music and visual animations that make sound are commonly-reported experiences in people who are, to borrow a phrase from Galton, “sane persons”. It might sound psychedelic to a non-synaesthete, but it is not far at all from normal perception, because life is full of events in which movement or some other visual stimuli is accompanied by a sound sensory stimuli: clapping hands, wind that rustles leaves, lips that speak, impacts that bang, or an explosion that is huge visually, sonically and physically. This pairing of sound and sight is so much a part of normal perception (in humans and other creatures) that it is commonly exploited in live entertainment.

I’ve had the opportunity to work in the past in the live entertainment industry and I also recently enjoyed the rehearsals of the Arcadia musical and special effects spectacular show currently at Elizabeth Quay in Perth. I know that there is a most startling loud roaring sound through the stage speakers that is typically created to coincide with a visual effect of an explosion of flames. It’s like some bloke presses a button somewhere and all hell breaks loose for a second or two. Arcadia uses this flame-roar sound to add sonic spectacle to the flame-thrower, and in the past while working I’ve also heard that sound used in complete isolation from music in a sound check of another spectacular stage show. At the risk of ruining the magic, I’m revealing that the sound that goes with the flames is an artistic artifice. I guess that any real sound that the flame effect makes has been judged to be not sufficiently loud and spectacular enough, and a suitably awesome sound (a recording of what I can only guess) was created to go along with the visual effect of flames from hell. I think this shows just how important crossmodal experiences are to live entertainment shows that are based on spectacular sensory experiences. The sound must equal the visual spectacle.

There are also many other ways in which sound and sight are linked in stage shows and special effects in entertainment. Musicolour lighting effects have been around since my Dad created disco equipment in our lounge using it back in the 1970s, and similar but much more developed lighting effects can be seen in the body of the Arcadia spider. Technology is not always required to artificially marry lighting and sound, as the amazing red and blue man electricity show features electrical discharges that look like tamed lightning that naturally give off a crackling sound along with the white light. But then again, I’m now wondering whether that sound is for real. Anyway, it’s wonderful, mad, sensory fun. I love it!

Postscript January 2017

I think the phenomenon of “quiet fireworks” adds more support to my point that spectacular public entertainment special effects often include the deliberate timing of sound and visual effects to happen at the same time to create a form of artistic synaesthesia, because while fireworks typically have bangs and flashes at the same time, the bang part of the spectacle is not essential or inseparable.

Arcadia Spider

Red man and blue man

Flaming Spider

Brain Games on SBS

This TV series is so much worth viewing if you are interested in psychology, including optical illusions, visual perception and many other interesting things. http://www.sbs.com.au/ondemand/video/506325059936/brain-games-power-of-persuasion

Synaesthesia-related current and upcoming arts events in Australia

MONA in Tasmania will be revisiting the theme of synaesthesia in Synaesthesia+, a musical, visual and gustatory festival of the psychological phenomenon. It is happening this weekend and tickets will set you back quite a lot.

In Perth, Western Australia PICA have been hosting an exhibition of sound art, What I See When I Look at Sound, featuring the works of artists Lyndon Blue, Lauren Brown, Matthew Gingold, Cat Hope and Kynan Tan. This show will be on until the end of this month and it is free, or at least we didn’t get charged when we went to look and listen to it a while ago.

You might think from considering the title of the exhibition that it might have the theme of synaesthesia, and indeed the works are described each as a “synaesthetic offering”, but actually I believe that the theme of the exhibition, “the relationship between looking and hearing” is actually about binding, which is a broader term that can encompass normal or average sensory perception and also some types of synaesthesia that are similar to or more consciously-experienced variants of normal mental sensory perception. I think this exhibition is about binding more than it is about synaesthesia. If a multi-sensory arts event was “about synaesthesia” I’d expect to see lots of colour and hear music and maybe see or feel letters of the alphabet, or see calendars suspended in space, and maybe even experience smells and flavours. I might look at a “synaesthesia art” painting and as a direct result “feel” motion or “hear” rhythms.The painting Upward by synaesthete artist Vassily Kandinskii or the painting Broadway Boogie Woogie by probable synaesthete artist Piet Mondrian are both pretty clear examples of what I mean by synaesthesia art. I have written about both artists previously in posts at this blog.

Binding is a term used in psychology, the philosophy of mind, neuroscience and cognitive science. It is certainly related to synaesthesia and is central to scientific understanding of synaesthesia as a phenomenon in neuroscience, but it isn’t the same thing. As far as I understand binding is about the perception of the many different sensory characteristics of an object or an event as a unified thing or event. A clear example would be the installation Filament Orkestra by Matthew Gingold. It grabs and holds attention and causes reflection even though the idea is no more complicated than (simple) sound and (plain white) light being presented (or not presented) both at the same points in time. I found the effect to be quite reminiscent of flamenco dancing and tap dancing, which I guess shows how the sensory binding of sight and sound is an engaging effect that is used in a diverse range of art forms, high arts and popular arts, modern and traditional, even including firework displays. Have you ever had the experience of viewing from an elevated location a fireworks display that is happening a distance away, and the wind is blowing in such a direction that the sound waves never reach where you are standing, so that the sight has no soundtrack? It’s the strangest thing to see (and not hear).

According to some online festival programs, tomorrow (Saturday August 16th 2014), as a part of the Perth Science Festival which is a part of National Science Week there will be a free event in the Central Galleries at PICA titled Sounds Symbols and Science at 1.00pm, which will be “a special live concert of “Cat Hope’s End of Abe Sade in the What I See When I Look at Sound exhibition”” and this will somehow involve digital graphic notation, which is a concept that very much overlaps with many synaesthetes’ experiences of listening to music, including my own at times, so I’m happy to categorize this planned event as synaesthetic, which is more than enough to provoke my curiosity.

http://www.pica.org.au/view/Sounds%2C+Symbols+and+Science/1891/

https://www.facebook.com/events/686307634740051/

http://www.scienceweek.net.au/perth-science-festival/

http://www.scitech.org.au/events/1583-perth-science-festival

Defining synaesthesia and some interesting research findings – a lecture by a leading Aussie synaesthesia researcher

Below is a link to a webpage that has a video of Associate Professor Anina Rich from Macquarie University delivering the Paul Bourke Lecture 2014 and answering questions afterwards. Some other speakers have a few things to say before her lecture. Associate Professor Anina Rich is the winner of the 2013 Paul Bourke Award for Early Career Research.

http://www.assa.edu.au/events/lectures/bourke/2014/38

Don’t forget the parietal lobe – the connections are interesting

If you have been reading this blog for a long time you’d know I’ve been trying to figure out which parts of my brain are responsible for my synaesthesia and related experiences. I’ve found that the right fusiform gyrus is a part of the brain that comes up over and over again, in relation to synaesthesia and also face recognition I experience many types of synaesthesia and also have achieved scores in face recognition tests consistent with being a super-recognizer, so this combination seems significant, and despite a lack of any evidence from other case studies linking synaesthesia with superior ability in face recognition, I still think it is a possible relationship that should be scientifically investigated, especially in light of a pattern of associations which I believe suggests that synaesthesia might be a neuropsychological condition that could be seen as the opposite of Benson’s syndrome, which is a type of dementia that involves a loss of visual perception, apparently including a loss of face recognition ability. While synesthesia is generally an inborn developmental condition, and Benson’s or PCA a neurodegenerative condition with a typical onset late in life, I’ve still got to wonder whether inborn factors contribute towards Benson’s. While Benson’s is considered to be a variant of Alzheimer’s, I don’t think anyone knows why it causes deterioration in different areas of the brain as are affected by Alzheimer’s, apparently the same parts of the brain (at the rear) that appear to be enhanced or hyperactive in my brain, and I also doubt that anyone knows why Benson’s has an onset earlier than Alzheimer’s disease. I’m sceptical of the idea that Benson’s is just Alzheimer’s of the back-end of the brain. I suspect that immune system elements microglia and complement might be central to an explanation for Benson’s syndrome. Reading Dr B. Croisile’s paper about Benson’s I’m struck by the many very strange effects of Benson’s on perception, and I wonder at the ways in which a study of it might inform science about  the workings of the brain. I think it is at least as interesting as synaesthesia, which attracts a lot of attention from researchers. Apparently people with Benson’s cannot imitate movements. Does this mean that the mirror-neuron system which so many neuroscientists have gotten so excited about is located at the rear of the brain? I note that the inferior parietal cortex is one of the parts of the brain that are thought to house mirror neurons.

When I set out to write this post I had actually planned to write about a fairly recent review journal paper focusing on recent research about the most common and well-known types of synaesthesia: coloured hearing, coloured graphemes and time units in space synaesthesia. I really like the paper cited below by Professor Karsten Specht from the University of Bergen in Norway, and I’d recommend it to anyone who wants to learn about the latest knowledge about synaesthesia from just one paper. I only have a couple of gripes about he paper. I wouldn’t describe synaesthesia as “rare” as Specht does. Ward, Sagiv and Butterworth wrote in 2009 that around 12% of the population have number forms, and that estimate doesn’t surprise me. Synaesthesia in general can’t be rare if it includes one type that isn’t rare. Time-space synaesthesia or number forms is one type of synaesthesia which the synaesthete can have but not suspect that it is synesthesia, or anything out of the ordinary, so I’d guess it could be very much under-reported and under-estimated. My other gripe with Professor Specht’s paper is this bit; “In recent years, several studies have attempted to investigate whether synaesthesia is primarily a perceptual or conceptual phenomenon.” I think Specht is here presenting the reader with a false dichotomy. In some of the types of synaesthesia and related phenomena which I experience sensory perception, memory and conceptual thinking are connected with synaesthesthetic linkages, so I doubt that there is much point in trying to characterise synaesthesia as one or another type of phenomenon. I was very excited when I read the book Beyond Human Nature by philosopher Jesse Prinz. Professor Prinz argued that we think in mental images rather than in language. He wrote that “It used to be thought that the back part of the brain is used for perceiving and the front is used for thinking. But we now know that the back part of the brain, where most of the senses are located, is very active when people think. Moreover, we know that the front part of the brain does not work on its own, but rather coordinates and reactivates sensory patterns in the back. Recent evidence from Linda Chao and Alex Martin has shown that reading activates the same areas as looking at pictures, suggesting that we visualize what we read.” In a post that I wrote a while ago I described involuntarily “seeing” in my mind’s eye visual images of landscapes and building interiors from imagination and memory while listening to an autobiographical audio-book. I thought it was probably related to synaesthesia, but it appears that everyone’s brain illustrates text with images when reading. Perhaps synaesthetes do this to a greater degree or in a way that is more available to conscious awareness.

Anyway, back to Specht’s paper. Having read it I now suspect that the parts of my brain that are bigger or better connected or more active or something are: the right fusiform gyrus (including the FFA), the left parietal lobe including the left intraparietal sulcus, the right inferior parietal lobe, the hippocampus (I’m sure is involved with IMLM) and the parahippocampal gyrus. I’d guess that these are the places where interesting things are happening. It appears that the role of the parietal lobe in synaesthesia has been understated in the past. It is now thought that synaesthesia does not solely involve the cross-activation of two different sensory areas (as if it was ever that simple!), but it also requires a “binding” process to happen in the parietal lobe. There is no underestimating the importance of this binding.

If you are as interested in synaesthesia and bits of the brain as I am, you might also like to read a much longer journal paper by Rouw, Scholte and Colizoli that was published last year. It is available in full text at no cost, but I don’t think it covers non-colour types of synaesthesia. Details can be found below. One part of the parietal lobe mentioned in that paper, which is cited by a few studies as involved with synaesthesia is the inferior parietal lobule (IPL, Brodmann areas 39 and 40). It is also known as Geschwind’s territory because the neurologist Geschwind predicted in the 1960s that the parietal lobe played a role in language, and was proven right when the IPL was found to include a second connection between Broca’s area and Wernicke’s area, which are of central importance in language. The IPL is very interesting as a part of the brain involved in synaesthesia because according to a 2004 article in New Scientist magazine the IPL matures at a late age, between the ages of five and seven years, which just happens to be time in life when children typically learn the ability to read and write, and it is also the age range in which some children develop grapheme-colour synaesthesia. I find this very interesting because in my family we have at least three closely related grapheme-colour synaesthetes who are unusually high achievers in reading and writing in testing and academic achievement. Two of these synaesthetes were early readers and also talented at language learning. What’s the betting that some gene that alters the development of the IPL is behind this? The author of the most interesting little science magazine article that brought me this news, Alison Motluk, is herself a synaesthete. Is it just a coincidence that a journalist with a well-connected brain has pointed out a number of interestingly related facts that are connected around the conceptual hub of the inferior parietal lobule?

Specht, Karsten Synaesthesia: cross activations, high interconnectivity, and a parietal hub. Translational Neuroscience. Volume 3 Number 1 (2012), 15-21, DOI: 10.2478/s13380-012-0007-z
http://www.springerlink.com/content/512306132j162437/

Croisile, Bernard Benson’s syndrome or Posterior Cortical Atrophy. Orphanet. September 2004. http://www.orpha.net/data/patho/GB/uk-Benson.pdf

Ward, Jamie, Sagiv, Noam and Butterworth, Brian The impact of visuo-spatial number forms on simple arithmetic. Cortex. Volume 45 Issue 10Pages 1261-1265 (November 2009). http://www.cortexjournal.net/article/S0010-9452(09)00213-5/abstract

Rouw, Romke, Scholte, H. Steven, Colizoli, Olympia Brain areas involved in synaesthesia: A review. Journal of Neuropsychology. Special Issue: Synaesthesia. September 2011 Volume 5 Issue 2 p.214-242. Article first published online: 16 SEP 2011 DOI: 10.1111/j.1748-6653.2011.02006.x  http://onlinelibrary.wiley.com/doi/10.1111/j.1748-6653.2011.02006.x/full

Motluk, Alison Two links good for kids’ language comphrehension. New Scientist. Issue 2478. December 18th 2004. p.12. http://www.newscientist.com/article/mg18424784.300-second-link-discovered-in-human-language-circuit.html