Above, a video visualizing two of our participants’ ECG responding to music in real-time.
Yellow and green lines represent their ECG’s, red the level of synchrony between them.
The audio is a live recording of the experiment and is synced to the plots.
A GitHub repository with all source code can be found here.
The last few years, there has been a growing movement of doctors, scientists, music therapists and psychologists investigating the use of music in medicine to help patients deal with, among others, pain, depression and possibly even Alzheimer’s disease. As Joanne Loewy, director of the Armstrong Center and co-editor of the journal Music and Medicine, states: “There’s just something about music — particularly live music — that excites and activates the body. Music very much has a way of enhancing quality of life and can, in addition, promote recovery.”
So as a researcher at the JADS, I am proud to have been asked to participate in the “Music as Medicine” project – a project launched last year, by a large and diverse consortium of partners, each bringing their extensive expertise to the table:
- Deloitte – Data Consultancy & Regulatory advice
- Erasmus MC – Deep domain knowledge on health & music
- IMEC – Stress sensors
- Jheronimus Academy of Data Science – Data science
- Rotterdams Philharmonisch Orkest – Music
- Totem Open health – Open ECG wearable
The goal of the project is to
- Actively contribute in the quest to apply music in medicine
- By exploring the positive impact of music on the human body in a live setting
- Gathering data through a variety of digital wearables
In this, the set-up of the study is both open-ended and exploratory, with the intention to gain new insights in how people respond in different (physiological and psychological) ways to various forms of music.
A first basic exploratory experiment that made use of the “Bobbi,” Totem’s Open Health ECG sensor, was performed last December:
Though the experiment was relatively limited in scope, it did inform the team that it could be useful to complement the Bobbi with other wearables – such as those of IMEC. And that, to be able to get a better grasp on the interactions between the music and subjects’ psychophysiological responses, it would be important to synchronize the music with all physiological measures, preferably with millisecond accuracy.
1 Exploratory working hypothesis
Happily, in an exploratory experiment of this size, there is ample room to evaluate working hypotheses aplenty. Would it, for instance, be possible to infer the emotional states of our participants in response to changes in the tonality of the music through changes in Heart Rate Variability? Do alternating segments of music indeed slow the heart, while more stable sections speed it up?
Still, as I have always been fascinated by how people are able to non-verbally coordinate and share emotions and behaviors, here I decided to focus on the compelling suggestion that people who share emotional attachments also tend to share their physiological states. It is unclear why this may be the case, though one theory is that music and dance have co-evolved biologically and culturally to serve as a technology of social bonding. Which leads to my exploratory working hypothesis: might the heart rates of our participants synchronize while they listen to music? Such synchrony is not all that easy to establish, as two or more cyclic signals will by definition sync up now and then. But if we do find some form of heightened synchronization, I can think of a great many new experiments to further investigate this hypothesized link between music and social bonding! Do people who score lower on empathy also show less heart rate synchronization? Do different types of music (jazz, classical, dance) have different effects on various forms of social bonding or empathy – and how do these relate to measures of heart rate synchrony? Can music be of use to help people bond again, for instance in couples therapy? Can we use heart rate measurements during a therapy session to measure how the couple is doing? But let me not get ahead of myself – let’s first see whether there is any indication of synchrony at all!
So, to be able to synchronize an audio recording of future experiments with the Bobbi (or other) ECG wearables in a simple, cheap and effective way, I came up with a friendly new gadget: the “Bobbi Sync Device.”
This Bobbi Sync Device consists of a basic Arduino / MOSFET based circuit1 controlling three strong, of the shelf electromagnets (Intertec ITS-MS-5030 400 N 12 V). It can generate a series of short, precisely timed2 magnetic pulses — which are automatically registered by any Bobbi’s3 placed on or near the device.
This setup enables us to synchronize up to hundreds of devices at a time — pre-empting the need for complex, and often fragile, Bluetooth or WIFI based synchronization protocols.
As a bonus, the Bobbi Sync Device can also, among others, beep loudly, perfectly in sync with each magnetic pulse. These beeps can then be recorded, after which the resulting audio recording can be synced with Bobbi’s ECG measurements as well. For the current setup, we recorded the whole experiment, from start to finish, with an audio recorder app on a Samsung Android smartphone4.
To complete our setup, we repeat our “beep and pulse protocol” at the end of our trial, which enabled us to compensate for any potential linear drift in the internal clocks of any Bobbi.
1 Arduino source code can be found here.
2 By default ten 500 ms square wave pulses at 500 ms intervals.
3 Or any other magnetometer equipped wearable or MCU based dev board — that is, most of them!
4 Starting the recording some time before the first ten beeps, and stopping some time after the final beeps.
3 The experiment
The sync device was tested during the “Music as Medicine” group’s second pilot experiment, at Mariënburg, Den Bosch (the data science campus of JADS).
This time around, eight participants (all team members) were outfitted with both the Bobbi wearable two lead ECG device and some cool IMEC sensors:
- A wristband measuring skin conductance, skin temperature, and movement.
- A wristband measuring heart rate through photoplethysmography (PPG)
The participants were then seated in the Marienburg chapel, where they listened to classical music selected by Andre Heuvelman – who, as a bonus, gave a solo performance of 10 minutes after the experiment.
4.1 The audio recording
Let’s first take a look at the MP3 audio recording (128kbps MP3 or WAV version on request) of our Music as Medicine session. The downloadable version has been precut and covers the session right from the first “start” beep up until the last of the final beeps. This yields a recording of just over an hour, or 1:08.59.023, to be precise. The recording has the following structure:
The three musical sections of the experiment consist of the following compositions:
- Ravel’s Pavane pour une infante défunte followed by the Nocturne of Mendelssohn’s A Midsummer Night’s Dream.
- Fauré’s Pavane followed by the first part of the prelude to his Pelléas et Mélisande.
- The third movement of Haydn’s Symphony No. 44.
When zooming into the final 9.5 seconds of the recording, that is, to between 1:08.49.523 and 1:08.59.023, it is clear that our Bobbi Sync Device’s beeping did its job:
4.2 The Android serial log
As an additional safeguard, we also time-stamped all beeps/pulses3. The resulting log files can be found here:
- Serial log data start — first of ten start beeps/pulses of 500 ms each starting at 11:05:15:811
- Serial log data end — first of ten end beeps/pulses of 500 ms each starting at 12:14:04:629
3 By controlling our Arduino with Eltima’s Serial Port Monitor Pro, which offers time stamped logging.
4.3 Bobbi ECG data
Now let’s take a look at our CSV data files, containing ECG, the magnetometer x/y/z signal (which registers our Sync device’s pulse) and more.
The files can be downloaded here:
Information on the data structure of these files, and some general tips on how to analyze Bobbi ECG data can be found here.
Regrettably, most of the Totem Bobbi’s we received were accidentally not updated to their latest firmware version. With the result that some of them did not record at all, and only one of our Bobbi recordings was complete. We will, of course, remedy this before our next experiment.
However, for the Bobbi’s that did record, the synchronization protocol worked beautifully! To demonstrate this, let’s zoom in to the first 20 seconds of the experiment. Here we see the magnetic pulse as registered by the magnetometer in the top panel – and no ECG, as the Bobbi’s ECG leads were not yet attached at this point in time:
Next, let’s move to some random point in time right in the middle of our experiment. In the top panel, the magnetometer now registers the earth magnetic field. But take a look at that ECG signal in the bottom panel: upside-down, but also clean and well defined!
So now we have an audio recording and the Bobbi data files, all cut to size and containing clear start and stop sync pulses. Next step: running some first exploratory analyses.
For our analysis, I started out by cleaning the data and extracting the heartbeats with an exploratory R script (available on GitHub):
To get a better feel for how heart rates and music actually relate to each other in real time, I then imported the now cleaned and enriched audio and data files into Sonic Visualizer. an application for the interactive analysis of the contents of music audio files. Sonic Visualizer allowed me to produce the video at the top of this post. And to ascertain once again that the sound and the Bobbi data are now indeed perfectly aligned, as the following screenshot proves:
Of course, our exploratory analysis would not be complete without a first test for ECG synchrony. There are many measures of synchrony, but for now, I decided to use both our participants extracted heartbeats and test them for spike train synchrony. By importing the heartbeat data in Python, I was able to make use of the PySpike package: a Python library for the numerical analysis of spike train similarity. Its core functionality is the implementation of the ISI-distance and SPIKE-distance as well as SPIKE-Synchronization. The resulting plots, juxtaposed with the music:
As can be seen in the averaged SPIKE-Sync profile plot (the final one), there seems to be some heartbeat synchronization overall – which grows towards the end, then lowers again. Very intriguing – but also very preliminary and possibly random. So I am now keeping my fingers crossed for this pattern to hold up in our next pilot, with more participants, and more up-to-date Bobbi’s’!