Grateful data

UNIVERSITY PARK, Pa. — While data is often used to form a line on a graph describing a phenomenon — think points on a chart — what happens when it comes to life as a musical score?

For Mark Ballora, a professor of music technology, sonifying data not only makes perfect sense, it also offers a new avenue for experiencing and understanding data. In his case, cosmological phenomena.

“I take the line and turn it into a melody,” Ballora said. “The idea of finding music in nature and science appeals to me on a poetic level.”

Ballora is one of the Penn State faculty members and graduate students who will be offering hands-on demonstrations July 13-15 at The Art of Discovery, Penn State’s booth at the Central Pennsylvania Festival of the Arts. The booth, next to Willard Building on the University Park campus, will feature free workshops for children and adults ranging from seeing 3-D printers in action and experiencing 360-degree viewfinders to watching glass candy get made.

For the full schedule, read more about the booth here. To talk with Ballora and experience data as sound, drop by the booth from 5:30 p.m. to 7:30 p.m. Thursday, July 13.

Among Ballora’s projects is working with former Grateful Dead percussionist and ethnomusicologist Mickey Hart and 2006 Nobel Laureate George Smoot on the DVD project “Rhythms of the Universe.” Ballora created the sonifications for the project, which explores the cosmos and includes visualizations as well as sonifications of astrophysical datasets.

“The dataset becomes like a musical score,” Ballora said.

Ballora — a self-described Deadhead — became involved in the project after he heard Hart and Smoot were working on a movie that would feature sounds of the Big Bang. “I thought, ‘I have to do that,’” Ballora said.

With that and other projects, choosing which instrument goes with which data is subjective. So, Ballora said he gets to ask, What would be a good instrument for a pulsar or an aurora borealis? It’s the same way an artist might choose which shade of blue conveys the nighttime sky.

“That’s the fun of it: designing something that says, ‘Oh, that sounds shimmery.’ Or, the galaxy is like a wind chime in the sky, so let me design something that sounds like a wind chime.”

From there he generates the sound using an audio synthesis software program — mapping the information to sound characteristics like melody and rhythm. He often has to transpose the numbers that describe the electromagnetic frequencies — the pulsars and aurora borealis — to ensure they are suitable for the pitches and that the listener can hear the contours of the lines.

Ballora thinks that while the idea of listening to science — not just looking at it — may be seen as a novelty now, it will be something that grows more common with young people.

“It will be part of the culture of research and science — that you don’t just look at it, you also listen to it. Young people will start finding the things they can discover by listening,” Ballora said. “Because we all respond to music, we dance to it, we tap our feet to it, we get tunes stuck in our heads, and the capabilities of the ear will be put into play. The strengths of the ear are different than the strengths of the eye. The ear detects some things better than the eye does, so discoveries will come about as a natural consequence of people being able to listen to datasets.”

In keeping with that, Ballora’s next projects received seed grants from the National Academies Keck Futures Initiative and the Gulf Research Program.

For one, he is working with marine biologist, underwater acoustician and cellist Heather Spence of Michelle’s Earth Foundation on a project focused on presenting the ocean’s dynamics through sound. Called “Layers of Meaning: How the Ocean’s Natural Acoustics and the Music of its Datasets Can Reveal Hidden Connections,” the project will include educational outreach.

For the other project, “Sonifications of Oxygen and Temperature Data in the Ocean: Creating a ‘Data Stethoscope’ to Detect the Ocean’s Vital Signs,” a “data stethoscope” will be created to translate into sound measurements of temperature, oxygen and biomass collected from the ocean’s depths. Ballora is working with Karen Wishner, professor of oceanography at University of Rhode Island.