照片的声音是什么样的？——作为联觉试听数字人文方法的数字图像可听化

摘要/Abstract

摘要： 计算机具有将像素、形状和其他视觉材料特征转化为声音的能力。通过视觉与听觉之间的数据关联行为,一个以视觉为来源的声波组合得以生成,这是一种新的人工制品(artifact)。它精确地与原作数据相关联,从而提供了一种新的方式感知原作的形式、内容和背景。虽然它看起来似乎是将视觉对象扭转为一个听觉对象,但矛盾的是,它使观察者能够以更精确的方式重新观察视觉证据。通过跨越影像与音频、视觉与听觉之间的典型界线,一种具生产性的、联觉(synesthetic)的批判成为可能。通过对数据进行数字转置,听取和观察视觉作品,可以更好地进行近读,实现更具说服力的解释及更深入的理解背景。基于作者早期关于图像故障、可听化处理的学术工作,文章对美国著名歌手琼·贝兹1960年代初在加州伯克利希腊露天剧场演出的一张照片进行了研究。这张照片出自作者对伯克利民谣音乐节和美国西海岸民谣音乐复兴历史的研究项目。在此,数字图像可听化处理的使用尤为有趣。尽管不能魔术般地还原出照片里的音乐,但是我们可以更切近地关注照片中的“声音幽灵”。数字图像的可听化技术虽然无法还原音乐本身,但是在观察照片中贝兹如何创作音乐时,它确实有助于放大性别、权力、具化、场面、表演、等级制度和表现等问题。通过耳朵和眼睛来对图像进行多重意义的扫描,可以产生一些意想不到的感知,从而支持更具启发性的分析。由数据、信号、图像、声音、历史和人类感知构成的“赛博格舞蹈”在数字图像可听化中出现了,这激发了对视觉材料的新审视。照此,视听数字人文的联觉模式为激活学术想象力提供了一条充满希望的新途径。

关键词: 联觉视听, 数字人文, 数字图像可听化, 琼·贝兹

Abstract: Computers have the capacity to transpose the pixels,shapes,and other features of visualmaterial into sound. This act of data correlation between the visual and the audial produces anew artifact,a sonic composition created from the visual source. The new artifact,however,correlates precisely to data in the original,thus allowing for fresh ways of perceiving its form,content,and context. Seeming to distort the visual object into an aural one paradoxically allowsan observer to observe the visual evidence anew,with more accuracy. A kind of generative,synesthetic criticism becomes possible by cutting across typical boundaries between the visualand the audio,the optic and the aural. Listening to as well as looking at visual artifacts by wayof digital transpositions of data enables better close readings,more compelling interpretations, and deeper contextual understandings. Building on my earlier scholarship into image glitching,remixing,and sonification,this essay investigates a photograph of Joan Baez performing at theGreek Amphitheater in Berkeley,California,during the early 1960s. The image comes from myproject on the Berkeley Folk Music Festival and the history of the folk music revival on the WestCoast of the United States. Here,the use of digital image sonification becomes particularlyintriguing. While we cannot magically recover the music being made in the photograph,we can more closely attend to the ghosts of sound within the silent snapshot. Digital image sonification does not recover the music itself, but it does help to amplify issues of gender, power, embodiment, spectacle, performance, hierarchy, and performance in my perceptions of Baezmaking music in the photograph. Using the ear as well as the eye to scan the image for itsmultiple levels of meaning leads to unsuspected perceptions,which then support morerevealing analysis. In
digital image sonification,a cyborgian dance of data,signal,image,sound,history,and human perception emerges,activating visual materials for renewed scrutiny. In doing so,this mode of AudioVisual DH activates the scholarly imagination in promising newways.

Key words: associative audiovisuality, digital humanities, digital image audibility, Joan Baez

迈克尔·J.克莱默朱子彤/译. 照片的声音是什么样的？——作为联觉试听数字人文方法的数字图像可听化[J]. 数字人文研究, 2022, 2(4): 63-73.

Michael J. Kramer Trans. Zhu Zitong. What Does A Photograph Sound Like? Digital Image Sonification As Synesthetic AudioVisual Digital Humanities[J]. Digital Humanities Research, 2022, 2(4): 63-73.

参考文献

ARNOLD T, LAUREN T, 2019. Distant viewing: analyzing large visual corpora [ J ]. Digital scholarship in the humanities, 34

(Supplement1):1-14.

BAEZ J,2009. And a voice to sing with:a memoir[M]. New York:Simon and Schuster.

BARTHES R,1981. Camera lucida:reflections on photography[M]. RICHARD H Trans. New York:Farrar,Straus,and Giroux.

BENJAMIN W,1968. The work of art in the age of mechanical reproduction[M]. New York:Harcourt Brace Jovanovich,217-252.

BOLDER J D,RICHARD G,2000. Remediation:understanding new media[M]. Cambridge,MA:MIT Press.

BRECHT B,1964. Alienation effects in Chinese acting[M]. New York:Macmillan,91-99.

BULL M,LES B,2004. The auditory cultures reader[M]. New York:Bloomsbury,2004.

CAGE J,1961. Silence:lectures and writings[M]. Middle town, CT:Wesleyan University Press.

CAMPT T M,2017. Listening to images[M]. Durham:Duke University Press.

CANTWELL R,1997. When we were good:the folk revival[M]. Cambridge,MA:Harvard University Press.

CARPENTER E,MARSHALL M,1960. Acoustic space. In explorations in communications[M]. MA:Beacon Press,64-69.

CHUN W H K,KEENAN T,2005. New media,old media:a history and theory reader[M]. New York:Routledge.

CLEMENT T,2015. When texts of study are audio files:digital tools for sound studies in DH[M] / / SCHREIBMAN S,SIEMENS R,

UNSWORTH J. A new companion to digital humanities. Hoboken,NJ:Wiley-Blackwell,348-357.

COHEN R D,2002. The folk music revival and American society,1940-1970[M]. MA:University of Massachusetts Press.

CORBIN A,1998. Sound and meaning in the nineteenth-century French Countryside[M]. MARTIN T,Trans. New York:Columbia University Press.

DONALDSON C L,2014. “I hear America singing”:folk music and national identity[M]. Philadelphia:Temple University Press.

EINIX L. 2014. Archaeoacoustics:the archaeology of sound[M]. Myakka City,FL:OTS Foundation.

ERLMANN V,2014. Reason and resonance:a history of modern aurality[M]. New York:Zone Books.

ERNST W,2013. Digital memory and the archive[M]. Minneapolis:University of Minnesota.

ERNST W,2016. Sonic time machines:explicit sound,sirenic voices,and implicit sonicity [ M]. Amsterdam:Amsterdam University

Press.

FERGUSON K L,2019. Volumetric cinema [ C] / / Debates in the digital humanities 2019. Minneapolis: University of Minnesota,

335-249.

FILENE B,2000. Romancing the folk:public memory and American roots music[M]. Chapel Hill:The University of North Carolina

Press,2000.

FOO B,2014. Two trains:sonification of income inequality on the NYC subway[EB/ OL]. https:// datadrivendj. com/ tracks/ subway/.

GIBBS F,TREVOR O,2013. The hermeneutics of data and historical writing[M] / / DOUGHTERTY J,NAWROTZKI K. In writing history in the digital age. Ann Arbor:University of Michigan Press,159-170.

GITELMAN L,2006. Always already new:media,history,and the data of culture[M]. Cambridge,MA:The MIT Press.

GITELMAN L,2013. “Raw data” is an oxymoron[M]. Cambridge,MA:MIT Press.

GITELMAN L,2014. Paper Knowledge:toward a media history of documents[M]. Durham:Duke University Press.

HADJU D,2001. Positively 4th Street the lives and times of Joan Baez,Bob Dylan,Mimi Baez Farina,and Richard Farina[M]. New

York:Farrar Straus & Giroux.

JOQUE J,2011. Listening to the Dow[Z/ OL]. http://vimeo. com/ 23965023.

KRAMER M J,2018a. A foreign sound to your ear:digital image sonification for historical interpretation[M] / / LINGOLD C M,MUELLER D,TRETTIEN W. Digital Sound Studies. Duke University Press,178-214.

KRAMER M J,2018b. Glitching history:using image deformance to rethink agency and authenticity in the 1960s American folk music revival[J/ OL]. Current research in digital history,1. https:/ / crdh. rrchnm. org / essays/ v01-08-glitching-history / .
LAFRANCE A,2016. Hearing the lost sounds of antiquity[ EB/ OL]. (2016 -02 -19). https:/ / www. theatlantic. com/ technology / archive / 2016 / 02 / byzantine-angel-wings/ 470076 / .
LAUMEISTER M,2015. Realtime bitcoin transaction visualizer (formerly listen to bitcoin)[EB/ OL]. http:/ / www. bitlisten. com.
LIU A,2012. The state of the digital humanities:a report and a critique[J]. Arts and humanities in higher education,11(1):8-41.
McGANN J,SAMUELS L,1999. Deformance and interpretation[J]. New literary history,30:25-56.
MORETTI F,2007. Graphs,maps,trees:abstract models for literary history[M]. New York:Verso.
NOVAK D,MATT S. 2015. Keywords in Sound[M]. Durham:Duke University Press Books.
NOWVISKIE B,2016. Speculative collections[EB/ OL]. (2016-10-27). http:/ / nowviskie. org / 2016 / speculative-collections/ .
ONG W,1982. Orality and literacy[M]. New York:Routledge.
RATH R,2003. How early America sounded[M]. NY:Cornell University Press.
SAMPLE M,2012. Notes towards a deformed humanities[EB/OL]. (2012 -05 -12). http:/ / www. samplereality. com/ 2012 / 05 / 02/notes-towards-a-deformed-humanities.
SAYERS J. 2018. The Routledge companion to media studies and digital humanities[M]. New York:Routledge.
SCHEDEL M,2014. Sounds of science:the mystique of sonification[J/ OL]. Sounding Out!. https:/ / soundstudiesblog. com/ 2014/10/09/ sounds-of-science-the-mystique-of-sonification/.
SCHOFIELD T,DAVID K,AMARAL T,et al. ,2015. Archival liveness:designing with collections before and during cataloguing and
digitization[J/OL]. Digital humanities quarterly,9(3). http:/ / www. digitalhumanities. org /dhq/vol/9/3/000227/ 000227. html.
SHKLOVSKY V,2017. Art as device(1917 / 1919)[M]. New York:Bloomsbury.
SMITH M M,2001. Listening to nineteenth-century America[M]. Chapel Hill:The University of North Carolina Press.
SMITH M M. 2004. Hearing history:a reader[M]. Athens:University of Georgia Press.
SONTAG S,1977. On photography[M]. New York:Farrar,Straus and Giroux.
STERNE J,2003. The audible past:cultural origins of sound studies[M]. Durham,NC:Duke University Press.
STERNE J,2012. The sound studies reader[M]. New York:Routledge.
SUPPER A,2014. Sublime frequencies:the construction of sublime listening experiences in the sonification of scientific data[J]. Social studies of science, 44(1):34-58.
SUPPER A,2015. Sounds convincing:modes of listening and sonic skills in knowledge making[J]. Interdisciplinary science reviews,40(2):24-44.
SUPPER A,2016. Lobbying for the ear,listening with the whole body:the (anti-) visual culture of sonification[ J]. Sound studies,2(1):69-80.
THOMPSON E,2002. The soundscape of modernity:architectural acoustics and the culture of listening in America,1900-1933[M].Cambridge,MA:The MIT Press.
THOMPSON E,2013. Author's statement. The roaring twenties—an interactive exploration of the historical soundscape of New York City[J/OL]. Vectors,4(1). http:/ / vectors. usc. edu / projects/ index. php? project =98&thread =AuthorsStatement.
WALD E,2015. Dylan goes electric!:Newport,Seeger,Dylan,and the night that split the Sixties[M]. New York:Dey Street Books.
WALL J,2014. Transforming the object of our study:the early modern sermon and the Virtual Paul's cross project[J]. Journal of digital humanities,3(1).
WARD M,ADRIAN S W,2019. The archive after theory[C] / / Debates in the digital humanities 2019. Minneapolis:University of Minnesota Press,200-205.
WHITE R,2010. What is spatial history? [EB/ OL]. (2010-02-01). https:/ / web. stanford. edu / group / spatialhistory / media / images/publication / what%20is%20spatial%20history%20pub%20020110. pdf.

[1]	马修·威尔肯斯杨晓燕. 数字人文文化分析领域及大语言模型应用前景解析———马修·威尔肯斯博士访谈 [J]. 数字人文研究, 2023, 3(2): 3-14.
[2]	伯纳黛特·伯德曼高妍/ 译. 虚拟博物馆:数字人文与博物馆学的交叉研究对象 [J]. 数字人文研究, 2023, 3(2): 15-30.
[3]	唐宸张萍. 数字人文时代古典文学文献辑佚路径的实验与思考[J]. 数字人文研究, 2023, 3(2): 31-38.
[4]	周睿琪管凡谊. 近世日本研究的数字化转向[J]. 数字人文研究, 2023, 3(2): 116-128.
[5]	简锦松廖泫铭王勇张淑君唐宸严程谢定纮. 明钱榖《纪行图册》、张复《水程图》之大运河现地研究与 GIS 呈现(之二)[J]. 数字人文研究, 2023, 3(2): 50-115.
[6]	杨泽坤, 李少建, 凯丽麦. 全景与远景:数字人文与国家文化数字化战略———2022 年中国数字人文年会综述[J]. 数字人文研究, 2023, 3(1): 3-17.
[7]	乌尔苏拉·波利卡-德格尔, 郭佳楠/ 译. 数字人文基础设施与全球学术体系的“中心—边缘”模式 [J]. 数字人文研究, 2023, 3(1): 24-37.
[8]	李磊. 面向历史研究的中共组织史专题数据库构建探索[J]. 数字人文研究, 2023, 3(1): 38-49.
[9]	简锦松廖泫铭王勇张淑君唐宸严程谢定纮. 明钱榖《纪行图册》、张复《水程图》之大运河现地研究与 GIS 呈现(之一) ∗ [J]. 数字人文研究, 2023, 3(1): 50-60.
[10]	通萨洛·M. d. ·西利亚, 阿娜·C. ·格洛里亚, 安热拉·S. ·莎尔格萝, 布鲁诺·艾梅达, 达涅尔·蒙特罗, 马科·R. d. ·弗雷塔斯, 努诺·弗雷雷, 刘语丝/ 译. 葡萄牙国家社科人文研究基础设施“ROSSIO”规划与实践[J]. 数字人文研究, 2023, 3(1): 95-112.
[11]	刘炜, 刘越男, 王晓光, 王涛, 徐永明. 建构中国自主数字人文知识体系的使命与路径[J]. 数字人文研究, 2022, 2(4): 3-13.
[12]	冯惠玲. 新文科与数字人文教育之新[J]. 数字人文研究, 2022, 2(4): 14-21.
[13]	王台瑞. 迈向2030元宇宙世代数字人文之戏曲教育数字化战略建议[J]. 数字人文研究, 2022, 2(4): 38-62.
[14]	西蒙·恒晨, 鲁本·罗斯, 亚尼·马尔亚宁, 米科·托洛宁, 方华康/ 译. 基于数据驱动方法的历史报纸词汇变化研究[J]. 数字人文研究, 2022, 2(4): 74-92.
[15]	刘健. 智慧博物馆发展中的数字人文建设———以上海博物馆的实践为例[J]. 数字人文研究, 2022, 2(3): 39-49.