What Does A Photograph Sound Like? Digital Image Sonification As Synesthetic AudioVisual Digital Humanities

Abstract

Abstract: Computers have the capacity to transpose the pixels,shapes,and other features of visualmaterial into sound. This act of data correlation between the visual and the audial produces anew artifact,a sonic composition created from the visual source. The new artifact,however,correlates precisely to data in the original,thus allowing for fresh ways of perceiving its form,content,and context. Seeming to distort the visual object into an aural one paradoxically allowsan observer to observe the visual evidence anew,with more accuracy. A kind of generative,synesthetic criticism becomes possible by cutting across typical boundaries between the visualand the audio,the optic and the aural. Listening to as well as looking at visual artifacts by wayof digital transpositions of data enables better close readings,more compelling interpretations, and deeper contextual understandings. Building on my earlier scholarship into image glitching,remixing,and sonification,this essay investigates a photograph of Joan Baez performing at theGreek Amphitheater in Berkeley,California,during the early 1960s. The image comes from myproject on the Berkeley Folk Music Festival and the history of the folk music revival on the WestCoast of the United States. Here,the use of digital image sonification becomes particularlyintriguing. While we cannot magically recover the music being made in the photograph,we can more closely attend to the ghosts of sound within the silent snapshot. Digital image sonification does not recover the music itself, but it does help to amplify issues of gender, power, embodiment, spectacle, performance, hierarchy, and performance in my perceptions of Baezmaking music in the photograph. Using the ear as well as the eye to scan the image for itsmultiple levels of meaning leads to unsuspected perceptions,which then support morerevealing analysis. In
digital image sonification,a cyborgian dance of data,signal,image,sound,history,and human perception emerges,activating visual materials for renewed scrutiny. In doing so,this mode of AudioVisual DH activates the scholarly imagination in promising newways.

Key words: associative audiovisuality, digital humanities, digital image audibility, Joan Baez

Michael J. Kramer Trans. Zhu Zitong. What Does A Photograph Sound Like? Digital Image Sonification As Synesthetic AudioVisual Digital Humanities[J]. Digital Humanities Research, 2022, 2(4): 63-73.

References

ARNOLD T, LAUREN T, 2019. Distant viewing: analyzing large visual corpora [ J ]. Digital scholarship in the humanities, 34

(Supplement1):1-14.

BAEZ J,2009. And a voice to sing with:a memoir[M]. New York:Simon and Schuster.

BARTHES R,1981. Camera lucida:reflections on photography[M]. RICHARD H Trans. New York:Farrar,Straus,and Giroux.

BENJAMIN W,1968. The work of art in the age of mechanical reproduction[M]. New York:Harcourt Brace Jovanovich,217-252.

BOLDER J D,RICHARD G,2000. Remediation:understanding new media[M]. Cambridge,MA:MIT Press.

BRECHT B,1964. Alienation effects in Chinese acting[M]. New York:Macmillan,91-99.

BULL M,LES B,2004. The auditory cultures reader[M]. New York:Bloomsbury,2004.

CAGE J,1961. Silence:lectures and writings[M]. Middle town, CT:Wesleyan University Press.

CAMPT T M,2017. Listening to images[M]. Durham:Duke University Press.

CANTWELL R,1997. When we were good:the folk revival[M]. Cambridge,MA:Harvard University Press.

CARPENTER E,MARSHALL M,1960. Acoustic space. In explorations in communications[M]. MA:Beacon Press,64-69.

CHUN W H K,KEENAN T,2005. New media,old media:a history and theory reader[M]. New York:Routledge.

CLEMENT T,2015. When texts of study are audio files:digital tools for sound studies in DH[M] / / SCHREIBMAN S,SIEMENS R,

UNSWORTH J. A new companion to digital humanities. Hoboken,NJ:Wiley-Blackwell,348-357.

COHEN R D,2002. The folk music revival and American society,1940-1970[M]. MA:University of Massachusetts Press.

CORBIN A,1998. Sound and meaning in the nineteenth-century French Countryside[M]. MARTIN T,Trans. New York:Columbia University Press.

DONALDSON C L,2014. “I hear America singing”:folk music and national identity[M]. Philadelphia:Temple University Press.

EINIX L. 2014. Archaeoacoustics:the archaeology of sound[M]. Myakka City,FL:OTS Foundation.

ERLMANN V,2014. Reason and resonance:a history of modern aurality[M]. New York:Zone Books.

ERNST W,2013. Digital memory and the archive[M]. Minneapolis:University of Minnesota.

ERNST W,2016. Sonic time machines:explicit sound,sirenic voices,and implicit sonicity [ M]. Amsterdam:Amsterdam University

Press.

FERGUSON K L,2019. Volumetric cinema [ C] / / Debates in the digital humanities 2019. Minneapolis: University of Minnesota,

335-249.

FILENE B,2000. Romancing the folk:public memory and American roots music[M]. Chapel Hill:The University of North Carolina

Press,2000.

FOO B,2014. Two trains:sonification of income inequality on the NYC subway[EB/ OL]. https:// datadrivendj. com/ tracks/ subway/.

GIBBS F,TREVOR O,2013. The hermeneutics of data and historical writing[M] / / DOUGHTERTY J,NAWROTZKI K. In writing history in the digital age. Ann Arbor:University of Michigan Press,159-170.

GITELMAN L,2006. Always already new:media,history,and the data of culture[M]. Cambridge,MA:The MIT Press.

GITELMAN L,2013. “Raw data” is an oxymoron[M]. Cambridge,MA:MIT Press.

GITELMAN L,2014. Paper Knowledge:toward a media history of documents[M]. Durham:Duke University Press.

HADJU D,2001. Positively 4th Street the lives and times of Joan Baez,Bob Dylan,Mimi Baez Farina,and Richard Farina[M]. New

York:Farrar Straus & Giroux.

JOQUE J,2011. Listening to the Dow[Z/ OL]. http://vimeo. com/ 23965023.

KRAMER M J,2018a. A foreign sound to your ear:digital image sonification for historical interpretation[M] / / LINGOLD C M,MUELLER D,TRETTIEN W. Digital Sound Studies. Duke University Press,178-214.

KRAMER M J,2018b. Glitching history:using image deformance to rethink agency and authenticity in the 1960s American folk music revival[J/ OL]. Current research in digital history,1. https:/ / crdh. rrchnm. org / essays/ v01-08-glitching-history / .
LAFRANCE A,2016. Hearing the lost sounds of antiquity[ EB/ OL]. (2016 -02 -19). https:/ / www. theatlantic. com/ technology / archive / 2016 / 02 / byzantine-angel-wings/ 470076 / .
LAUMEISTER M,2015. Realtime bitcoin transaction visualizer (formerly listen to bitcoin)[EB/ OL]. http:/ / www. bitlisten. com.
LIU A,2012. The state of the digital humanities:a report and a critique[J]. Arts and humanities in higher education,11(1):8-41.
McGANN J,SAMUELS L,1999. Deformance and interpretation[J]. New literary history,30:25-56.
MORETTI F,2007. Graphs,maps,trees:abstract models for literary history[M]. New York:Verso.
NOVAK D,MATT S. 2015. Keywords in Sound[M]. Durham:Duke University Press Books.
NOWVISKIE B,2016. Speculative collections[EB/ OL]. (2016-10-27). http:/ / nowviskie. org / 2016 / speculative-collections/ .
ONG W,1982. Orality and literacy[M]. New York:Routledge.
RATH R,2003. How early America sounded[M]. NY:Cornell University Press.
SAMPLE M,2012. Notes towards a deformed humanities[EB/OL]. (2012 -05 -12). http:/ / www. samplereality. com/ 2012 / 05 / 02/notes-towards-a-deformed-humanities.
SAYERS J. 2018. The Routledge companion to media studies and digital humanities[M]. New York:Routledge.
SCHEDEL M,2014. Sounds of science:the mystique of sonification[J/ OL]. Sounding Out!. https:/ / soundstudiesblog. com/ 2014/10/09/ sounds-of-science-the-mystique-of-sonification/.
SCHOFIELD T,DAVID K,AMARAL T,et al. ,2015. Archival liveness:designing with collections before and during cataloguing and
digitization[J/OL]. Digital humanities quarterly,9(3). http:/ / www. digitalhumanities. org /dhq/vol/9/3/000227/ 000227. html.
SHKLOVSKY V,2017. Art as device(1917 / 1919)[M]. New York:Bloomsbury.
SMITH M M,2001. Listening to nineteenth-century America[M]. Chapel Hill:The University of North Carolina Press.
SMITH M M. 2004. Hearing history:a reader[M]. Athens:University of Georgia Press.
SONTAG S,1977. On photography[M]. New York:Farrar,Straus and Giroux.
STERNE J,2003. The audible past:cultural origins of sound studies[M]. Durham,NC:Duke University Press.
STERNE J,2012. The sound studies reader[M]. New York:Routledge.
SUPPER A,2014. Sublime frequencies:the construction of sublime listening experiences in the sonification of scientific data[J]. Social studies of science, 44(1):34-58.
SUPPER A,2015. Sounds convincing:modes of listening and sonic skills in knowledge making[J]. Interdisciplinary science reviews,40(2):24-44.
SUPPER A,2016. Lobbying for the ear,listening with the whole body:the (anti-) visual culture of sonification[ J]. Sound studies,2(1):69-80.
THOMPSON E,2002. The soundscape of modernity:architectural acoustics and the culture of listening in America,1900-1933[M].Cambridge,MA:The MIT Press.
THOMPSON E,2013. Author's statement. The roaring twenties—an interactive exploration of the historical soundscape of New York City[J/OL]. Vectors,4(1). http:/ / vectors. usc. edu / projects/ index. php? project =98&thread =AuthorsStatement.
WALD E,2015. Dylan goes electric!:Newport,Seeger,Dylan,and the night that split the Sixties[M]. New York:Dey Street Books.
WALL J,2014. Transforming the object of our study:the early modern sermon and the Virtual Paul's cross project[J]. Journal of digital humanities,3(1).
WARD M,ADRIAN S W,2019. The archive after theory[C] / / Debates in the digital humanities 2019. Minneapolis:University of Minnesota Press,200-205.
WHITE R,2010. What is spatial history? [EB/ OL]. (2010-02-01). https:/ / web. stanford. edu / group / spatialhistory / media / images/publication / what%20is%20spatial%20history%20pub%20020110. pdf.

[1]	Matthew Wilkens Yang Xiaoyan. Analysis of the Digital Humanities Culture Analytics Field and Application Prospect of Large Language Models—Interview with Professor Matthew Wilkens [J]. Digital Humanities Research, 2023, 3(2): 3-14.
[2]	Bernadette Biedermann Trans. Gao Yan. Virtual Museums as An Extended Museum Experience:Challenges and Impacts for Museology,Digital Humanities,Museums and Visitors —in Times of (Coronavirus) Crisis [J]. Digital Humanities Research, 2023, 3(2): 15-30.
[3]	Tang Chen Zhang Ping. Reflection on the Method of Gathering of Scattered Literature in the Digital Humanity Era [J]. Digital Humanities Research, 2023, 3(2): 31-38.
[4]	Zhou Ruiqi Guan Fanyi. The Digital Turn in Early Modern Japanese Studies [J]. Digital Humanities Research, 2023, 3(2): 116-128.
[5]	Chien Chin-Sung Liao Hsiung-Ming　Wang Yong　Zhang Shujun Tang Chen　Yan Cheng　Xie Dinghong. An On-Site Study and Its GIS Presentation of the Grand Canal in Painting Album of the Great Canal Journeys by Qian Gu and Visual Travelogue of a Journey Through the Waterways by Zhang Fu of Ming Dynasty [J]. Digital Humanities Research, 2023, 3(2): 50-115.
[6]	Yang Zekun, Li Shaojian, Kai Limai. Panorama and Prospect:Digital Humanities and National Strategy for Digitalization of Culture—Overview of the 2022 China Digital Humanities Conference [J]. Digital Humanities Research, 2023, 3(1): 3-17.
[7]	Li Lei. Establishment of C. P. C Organization History Database [J]. Digital Humanities Research, 2023, 3(1): 38-49.
[8]	Chien Chin-Sung Liao Hsiung-Ming Wang Yong Zhang Shujun Tang Chen Yan Cheng Xie Dinghong. An On-Site Study and Its GIS Presentation of the Grand Canal in Painting Album of the Great Canal Journeys by Qian Gu and Visual Travelogue of A Journey Through the Waterways by Zhang Fu of Ming Dynasty [J]. Digital Humanities Research, 2023, 3(1): 50-60.
[9]	Gonçalo M. d. Silva, Ana C. Glória, Ângela S. Salgueiro, Bruno Almeida, Daniel Monteiro, Marco R. d. Freitas, Nuno Freire, Trans. Liu Yusi. ROSSIO Infrastructure: A Digital Humanities Platform to Explore the Portuguese Cultural Heritage [J]. Digital Humanities Research, 2023, 3(1): 95-112.
[10]	. [J]. Digital Humanities Research, 2022, 2(4): 3-13.
[11]	Feng Huiling. The New Liberal Arts and New Digital Humanities Education [J]. Digital Humanities Research, 2022, 2(4): 14-21.
[12]	Wang Tai-jui. Towards 2030 Cross-generational Research of the Metaverse: The Strategic Proposal for Digital Humanities of Chinese Opera Education [J]. Digital Humanities Research, 2022, 2(4): 38-62.
[13]	Simon Hengchen, Ruben Ros, Jani Marjane, Mikko Tolonen, Trans. Fang Huakang. A Data-driven Approach to Studying Changing Vocabularies in Historical Newspaper Collections [J]. Digital Humanities Research, 2022, 2(4): 74-92.
[14]	. Towards Language Sensitivity and Diversity in the Digital Humanities [J]. Digital Humanities Research, 2022, 2(3): 3-20.
[15]	Liu Jian. Digital Humanities Construction in the Development of Smart Museums: The Practice of Shanghai Museum as an Example [J]. Digital Humanities Research, 2022, 2(3): 39-49.