New article in Nature Scientific Reports: https://www.nature.com/articles/s41598-021-01797-z#Sec2
Abstract: Cross-modal integration is ubiquitous within perception and, in humans, the McGurk effect demonstrates that seeing a person articulating speech can change what we hear into a new auditory percept. It remains unclear whether cross-modal integration of sight and sound generalizes to other visible vocal articulations like those made by singers. We surmise that perceptual integrative effects should involve music deeply, since there is ample indeterminacy and variability in its auditory signals. We show that switching videos of sung musical intervals changes systematically the estimated distance between two notes of a musical interval so that pairing the video of a smaller sung interval to a relatively larger auditory led to compression effects on rated intervals, whereas the reverse led to a stretching effect. In addition, after seeing a visually switched video of an equally-tempered sung interval and then hearing the same interval played on the piano, the two intervals were judged often different though they differed only in instrument. These findings reveal spontaneous, cross-modal, integration of vocal sounds and clearly indicate that strong integration of sound and sight can occur beyond the articulations of natural speech.
Shortcuts to latest articles:
Body Data for Music Information Retrieval Tasks - May 2021
Computational analysis of melodic contour and body movement - November 2019
Analyzing Free-Hand Sound-Tracings of Melodic Phrases January 2018
Exploring melody and motion features in “sound-tracings”
My recently defended PhD is about melodic cognition using motion-capture and motion-annotations as the main tool to elicit our understanding of melodic contour.This project aims at studying the relationship between melodic contour perception and body motion. The interaction of bodily responses to rhythmic content are fairly well understood in terms of physical motion, entrainment and synchronization. In terms of pitch material, several models have been suggested regarding the embodied nature of conceptual space as well as in the form of physical motion. In this project, we take melody as the central unit and analyse the gestural responses to melodic contour. The project will include a study of theoretical rules for well-formed melodies from various cultures. The project aims at developing methods for computational analysis of motion capture data for sound tracings. An exploration of the relationships of these to contours, and contour typologies, and additional work in the interaction of melodic contour with sound and timbre features form a part of this project.
My thesis investigates the relationships between visual and auditory modalities through the medium of North Indian Classical Music. Intermodal binding and unitization can be studied via natural body gestures and hand movements in improvised music. In this thesis, we present experiments to analyze the dependance relationships between musical parameters and body gestures through data from pitch analysis, motion capture and video processing. We provide evidence for a cognitive model for music-motion binding. We investigate the typology and categorization of these gestures and perception of monodic musical shapes amongst musicians and non musicians. We also implement models to convert these gestures back into sound, using rule based grammars derived from raga models.