[ineffable]

Video

[ineffable]

Computer application, digital map
Dimensions variable
2005

Email is now so pervasive and so essential to our lives that it’s difficult to remember how we communicated prior to its emergence. The ramifications of the nearly wholesale replacement of our physical voice with a detached written text continues to be felt everywhere email is used, whether connected to work or play, intimate exchange or legal agreements. How do we sound to those reading our emails, and how does the email from others sound to us? How do these voices shift when the recipient or the subject matter changes, and why?

As system designers, we are interested in the myriad of ways computer technology permeates and shapes our everyday lives. Email, easily the most widely used and standardized form of new media communication, has changed us in ways we are only beginning to examine, particularly in its complex effect on our relationship to language.

[ineffable] is a computer application which interprets emails between two correspondents and creates audio and visual ‘maps’ of language from the words used. The program reveals how our repeated phrases and stock answers encode the kinds of sounds we make, exposing the varied ‘voices’ we employ within the digital realm. The aggregate of our message’s phrases and the timeline in which they are sent and received constitute a potential map of interpersonal experiences (the external world) as well as how the user relates to the context of digital communication. The program uses all of these factors to generate distinctive portraits of our day to day communication. It functions as an experimental system which reconstitutes the “sound in one’s head” created when reading and writing as a synaesthetic experience. The viewer encounters a set of dynamic sonifications and visualizations from each set of correspondences in a split stereo sound environment.

HOW [ineffable] WORKS:

The application analyzes multiple factors from a cache of email messages. In addition to using the words, it also examines the chronology (including frequency and gaps between messages) as well as the length of each communication. [ineffable] then creates a pair of digitally animated maps representing each half of the conversation, switching between a moving line map and a ‘voice organism’ color collage generated by the system. The moving line map illustrates sound structures encountered by the system. It shows the frequency of the phonemes created by the email in the line’s thickness and color. The program’s voice organism visualizes time and the change in voice through time. It increases and decreases in complexity and transparency depending on the dates in between correspondences. Consistency of the ‘sound voice’ results in a consistent visual mapping, as derived from the general phonemic sounds found by the system. Changes in the ‘sound voice’ change the organism. The program also locates ‘sound signatures’ within the texts and generates a representative word from the aggregate of a user’s most frequently used sounds. For example, if a writer uses many “mmm”s and “ers” the word generated could be “murmur”. These are displayed under the visualization and change over time.

To generate the audio component, several tracks are created in response to the email data. First, phoneme sounds are assigned to an “instrument” created in the programming API Jsyn. Phonemes sampled from human voices are also played back in the rhythm of the email’s syllabic structure.

Exhibitions

  • Sonic Interventions, Amsterdam, Netherlands, 2005
  • Performance Studies International, “Becoming Uncomfortable”, Brown University, Providence, Rhode Island, 2005
  • SIGGRAPH 2004 Art Gallery: Synaesthesia, Los Angeles, California, 2004

Press

Additional Credits

This project is a collaboration between New York artist-scientists Mary Flanagan and Andrew Gerngross. Gerngross is a full time writer with a background in film-making and software engineering, among other talents.