Crossmodal Working Memory Load: Perceptual and Conceptual Contributions of Image Characteristics

摘要

What can crossmodal associations reveal about working memory capacity for complex objects? Forming these associations starts before baby first shakes a rattle and continues past Grandma's first cellphone purchase. To examine the contributions of perceptual and conceptual characteristics of images to WM for novel visual-auditory associations, we assessed crossmodal change-detection accuracy (Gilman & Ware, 2009) for associations between animal sounds and four image types: grayscale shapes, color shapes, grayscale drawings (Rossion & Pourtois, 2004), and color photographs. Following Alvarez & Cavanagh (2004), image information load was measured by comparing time costs for adding more images of the same type to a search array. Participantsa?? average search times (N=22) were all under 20ms/item, comparable only to the two fastest image classesa??colored squares and lettersa??tested by Alvarez and Cavanagh (2004). Meaningfulness of the image types was assessed using word association counts, with each image prompting on average four associations; representational images garnered .7 more associations than shapes, and full-color images received approximately .2 more associations than grayscale ones. Similar results were obtained for associational variety. Crossmodal change-detection accuracy was higher for associations between color images (representational or abstract) and unrelated animal sounds than for those using grayscale images. However, response bias was lower for crossmodal associations with representational images (color photos or grayscale drawings) than for those with color or grayscale shapes. This difference parallels recognition memory findings of lower bias for more-distinct visual (Sekuler Kahana, 2007) and auditory (Visscher, Kaplan, Kahana, & Sekuler, 2007) stimulia??stimuli designed to avoid conceptual associations. Visual search cost for the present image types varied linearly with participants' measured working memory capacity for crossmodal associations (r2=.971, F(1,2)=66.99, p=.015); the proportion of variance explained is comparable to that found in the visual precedent. These crossmodal WM dynamics are compatible with information-theoretic WM models, showing sensitivity to unimodal perceptual and conceptual characteristics affecting the discriminability of stimuli to be associated.