Music as Medicine: Sync

E-health, Psychology, R, Research, Statistics • •


Above, a video visualizing two of our participants’ ECG responding to music in real-time.
Yellow and green lines represent their ECG’s, red the level of synchrony between them.
The audio is a live recording of the experiment and is synced to the plots.
A GitHub repository with all source code can be found here.

The last few years, there has been a growing movement of doctors, scientists, music therapists and psychologists investigating the use of music in medicine to help patients deal with, among others, pain, depression and possibly even Alzheimer’s disease. As Joanne Loewy, director of the Armstrong Center and co-editor of the journal Music and Medicine, states: “There’s just something about music — particularly live music — that excites and activates the body. Music very much has a way of enhancing quality of life and can, in addition, promote recovery.”

So I was happy to be asked to participate in the “Music as Medicine” project – a project launched last year, by a large and diverse consortium of partners:

The goal of the project is to

  • Actively contribute in the quest to apply music in medicine
  • By exploring the positive impact of music on the human body in a live setting
  • Gathering data through a variety of digital wearables

In this, the set-up of the study is both open-ended and exploratory, with the intention to gain new insights in how people respond in different (physiological and psychological) ways to various forms of music.

A first basic exploratory experiment that made use of the “Bobbi,” Totem’s Open Health ECG sensor, was performed last December:

Though the experiment was relatively limited in scope, it did inform the team that it would be useful to complement the Bobbi with other wearables – such as those of IMEC. And that, to be able to get a better grasp on the interactions between the music and subjects’ psychophysiological responses, it would be important to synchronize the music with all physiological measures, preferably with millisecond accuracy.

2 Setup

To do so, it would be necessary to synchronize an audio recording with the Bobbi’s (or other) ECG wearables signal in a simple, cheap and effective way. Here, we came up with a friendly new gadget: the “Bobbi Sync Device.”

    

The Bobbi Sync Device consists of a basic Arduino / MOSFET based circuit1 controlling three strong, of the shelf electromagnets (Intertec ITS-MS-5030 400 N 12 V). It can generate a series of short, precisely timed2 magnetic pulses — which are automatically registered by any Bobbi’s3 placed on or near the device. Which enables us to synchronize up to hundreds of devices at a time — pre-empting the need for complex, and often fragile, Bluetooth or WIFI based synchronization protocols. As a bonus, the Bobbi Sync Device can also beep in sync with each magnetic pulse. These beeps can then be recorded, after which the resulting audio recording can be synced with Bobbi’s ECG measurements as well. For the current setup, we recorded the whole experiment, from start to finish, with an audio recorder app on a Samsung Android smartphone4.

1 Arduino source code can be found here.
2 By default ten 500 ms square wave pulses at 500 ms intervals.
3 Or any other magnetometer equipped wearable or MCU based dev board — that is, most of them.
4 Starting the recording some time before the first ten beeps, and stopping some time after the final beeps.

3 The experiment

The sync device was tested during the “Music as Medicine” group’s second pilot experiment, at Mariënburg, Den Bosch (the data science campus of JADS).

This time around, eight participants (all team members) were outfitted with both the Bobbi wearable two lead ECG device and some cool IMEC sensors:

The participants were then seated in the Marienburg chapel, where they listened to classical music selected by Andre Heuvelman – who, as a bonus, gave a solo performance of 10 minutes after the experiment.

4 Data

4.1 The audio recording

Let’s first take a look at the MP3 audio recording (128kbps MP3 or WAV version on request) of our Music as Medicine session. The downloadable version has been precut and covers the session right from the first “start” beep up until the last of the final beeps. This yields a recording of just over an hour, or 1:08.59.023, to be precise. The recording has the following structure:

The three musical sections of the experiment consist of the following compositions:

  1. Ravel’s Pavane pour une infante défunte followed by the Nocturne of Mendelssohn’s A Midsummer Night’s Dream.
  2. Fauré’s Pavane followed by the first part of the prelude to his Pelléas et Mélisande.
  3. The third movement of Haydn’s Symphony No. 44.

When zooming into the final 9.5 seconds of the recording, that is, to between 1:08.49.523 and 1:08.59.023, it is clear that our Bobbi Sync Device’s beeping did its job:

4.2 The Android serial log

As an additional safeguard, we also time-stamped all beeps/pulses3. The resulting log files can be found here:

  • Serial log data start — first of ten start beeps/pulses of 500 ms each starting at 11:05:15:811
  • Serial log data end — first of ten end beeps/pulses of 500 ms each starting at 12:14:04:629

3 By controlling our Arduino with Eltima’s Serial Port Monitor Pro, which offers time stamped logging.

4.3 Bobbi ECG data

On completion of our recording, we acquired the data from all Bobbies in the form of CSV data files. Each file contains ECG, the magnetometer x/y/z signal (which registers our Sync device’s pulse) and more.

Three of these Bobbi CSV files can be downloaded here:

Information on the data structure of these files, and some general tips on how to analyze Bobbi ECG data can be found here.

Regrettably, most of the Totem Bobbi’s we received had not yet been updated to the latest firmware version. With the result that only one of our Bobbi recordings was 100% complete.

However, for the Bobbi’s that did record, the synchronization protocol worked beautifully. To demonstrate this, let’s zoom in to the first 20 seconds of the experiment. Here we see the magnetic pulse as registered by the magnetometer in the top panel – and no ECG, as the Bobbi’s ECG leads were not yet attached at this point in time:

Next, let’s move to some random point in time right in the middle of our experiment. In the top panel, the magnetometer now registers the earth magnetic field. But take a look at that ECG signal in the bottom panel: upside-down, but also clean and well defined.

So now we have an audio recording and the Bobbi data files, all cut to size and containing clear start and stop sync pulses. Next step: running some first exploratory analyses.

5 Analyses

For our analysis, I started out by cleaning the data and extracting the heartbeats with an exploratory R script (available on GitHub):

To get a better feel for how heart rates and music actually relate to each other in real time, I then imported the now cleaned and enriched audio and data files into Sonic Visualizer. an application for the interactive analysis of the contents of music audio files. Sonic Visualizer allowed me to produce the video at the top of this post. And to ascertain once again that the sound and the Bobbi data are now indeed perfectly aligned, as the following screenshot proves:

Of course, our exploratory analysis would not be complete without a first test for ECG synchrony. There are many measures of synchrony, but for now, I decided to use both our participants extracted heartbeats and test them for spike train synchrony. By importing the heartbeat data in Python, I was able to make use of the PySpike package: a Python library for the numerical analysis of spike train similarity. Its core functionality is the implementation of the ISI-distance and SPIKE-distance as well as SPIKE-Synchronization. The resulting plots, juxtaposed with the music:

As can be seen in the averaged SPIKE-Sync profile plot (the final one), there seems to be some heartbeat synchronization overall – which grows towards the end, then lowers again. Very intriguing – but also very preliminary and possibly random. So, for now, I am now keeping my fingers crossed for this pattern to hold up in a next pilot, with more participants, and more up-to-date Bobbi’s’.

Leave a Reply