PLoS ONE plos plosone PLOS ONE 1932-6203 Public Library of Science San Francisco, CA USA 10.1371/journal.pone.0317183 PONE-D-24-30219 Research Article Biology and life sciencesOrganismsEukaryotaAnimalsVertebratesAmniotesMammalsPrimatesMonkeys Biology and life sciencesZoologyAnimalsVertebratesAmniotesMammalsPrimatesMonkeys Biology and life sciencesNeuroscienceCognitive scienceCognitive psychologyLearning Biology and life sciencesPsychologyCognitive psychologyLearning Social sciencesPsychologyCognitive psychologyLearning Biology and life sciencesNeuroscienceLearning and memoryLearning Biology and life sciencesAgricultureAnimal managementAnimal performance Research and analysis methodsAnimal studiesExperimental organism systemsAnimal modelsRhesus monkeys Biology and life sciencesOrganismsEukaryotaAnimalsVertebratesAmniotesMammalsPrimatesMonkeysOld World monkeysMacaqueRhesus monkeys Biology and life sciencesZoologyAnimalsVertebratesAmniotesMammalsPrimatesMonkeysOld World monkeysMacaqueRhesus monkeys Biology and life sciencesNeuroscienceCognitive scienceCognitive psychologyLearningLearning curves Biology and life sciencesPsychologyCognitive psychologyLearningLearning curves Social sciencesPsychologyCognitive psychologyLearningLearning curves Biology and life sciencesNeuroscienceLearning and memoryLearningLearning curves Biology and life sciencesNeuroscienceCognitive scienceCognitive neuroscienceReaction time Biology and life sciencesNeuroscienceCognitive neuroscienceReaction time Biology and life sciencesNeuroscienceCognitive scienceCognitive psychologyPerceptionSensory perceptionVision Biology and life sciencesPsychologyCognitive psychologyPerceptionSensory perceptionVision Social sciencesPsychologyCognitive psychologyPerceptionSensory perceptionVision Biology and life sciencesNeuroscienceSensory perceptionVision Biology and life sciencesPsychologyBehaviorAnimal behaviorAnimal communicationVocalization Social sciencesPsychologyBehaviorAnimal behaviorAnimal communicationVocalization Biology and life sciencesZoologyAnimal behaviorAnimal communicationVocalization Monkeys can identify pictures from words Cross-modal association in monkeys Cabrera-Ruiz Elizabeth Data curation Formal analysis Investigation Methodology Software Visualization Writing – review & editing 1 2 Alva Marlen Data curation Formal analysis Software Visualization Writing – review & editing 1 https://orcid.org/0000-0002-2476-5214 Treviño Mario Formal analysis Software Validation Writing – review & editing 3 Mata-Herrera Miguel Data curation Formal analysis Investigation 1 Vergara José Formal analysis Validation Writing – review & editing 4 https://orcid.org/0009-0008-6689-7304 Figueroa Tonatiuh Data curation Software 1 Perez-Orive Javier Validation Writing – review & editing 2 https://orcid.org/0000-0002-6608-3684 Lemus Luis Conceptualization Formal analysis Funding acquisition Methodology Software Validation Visualization Writing – original draft 1 * Department of Cognitive Neuroscience, Instituto de Fisiología Celular, Universidad Nacional Autónoma de México, Mexico City, México Basic Neurosciences, Instituto Nacional de Rehabilitacion, “Luis Guillermo Ibarra Ibarra” Mexico City, México Laboratorio de Plasticidad Cortical y Aprendizaje Perceptual, Instituto de Neurociencias, Universidad de Guadalajara, Guadalajara, Jalisco, México Department of Neuroscience, Baylor College of Medicine, Houston, TX, United States of America Vatakis Argiro Editor Panteion University of Social and Political Sciences, GREECE

NO authors have competing interests.

 * E-mail: lemus@ifc.unam.mx 12 2 2025 2025 20 2 e0317183 20 7 2024 23 12 2024 2025 Cabrera-Ruiz et al This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Humans learn and incorporate cross-modal associations between auditory and visual objects (e.g., between a spoken word and a picture) into language. However, whether nonhuman primates can learn cross-modal associations between words and pictures remains uncertain. We trained two rhesus macaques in a delayed cross-modal match-to-sample task to determine whether they could learn associations between sounds and pictures of different types. In each trial, the monkeys listened to a brief sound (e.g., a monkey vocalization or a human word), and retained information about the sound to match it with one of 2–4 pictures presented on a touchscreen after a 3-second delay. We found that the monkeys learned and performed proficiently in over a dozen associations. In addition, to test their ability to generalize, we exposed them to sounds uttered by different individuals. We found that their hit rate remained high but more variable, suggesting that they perceived the new sounds as equivalent, though not identical. We conclude that rhesus monkeys can learn cross-modal associations between objects of different types, retain information in working memory, and generalize the learned associations to new objects. These findings position rhesus monkeys as an ideal model for future research on the brain pathways of cross-modal associations between auditory and visual objects.

 Consejo Nacional de Humanidades Ciencias y Tecnologias 256767 https://orcid.org/0000-0002-6608-3684 Lemus Luis Consejo Nacional de Humanidades Ciencias y Tecnologías 245771 Cabrera-Ruiz Elizabeth http://dx.doi.org/10.13039/501100006087 Dirección General de Asuntos del Personal Académico, Universidad Nacional Autónoma de México PAPIIT- IN229223 https://orcid.org/0000-0002-6608-3684 Lemus Luis Secretaría de Educación, Ciencia, Tecnología e Innovación de la Ciudad de México SECTEI/103/2022 Vergara José LL received funding from the Consejo Nacional de Humanidades, Ciencias y Tecnologías (CONAHCYT; Grant Number: 256767; https://conahcyt.mx/) and the Programa de Apoyo a Proyectos de Investigación e Innovación Tecnológica (PAPIIT; Grant Number: IN229323; https://dgapa.unam.mx/index.php/impulso-a-la-investigacion/papiit). JV was supported by the Secretaría de Educación, Ciencia, Tecnología e Innovación de la Ciudad de México (Grant Number: SECTEI/103/2022; https://www.sectei.cdmx.gob.mx/). Elizabeth Cabrera Ruiz conducted this study to fulfill the requirements of the Programa de Doctorado en Ciencias Biomédicas at Universidad Nacional Autónoma de México and received a doctoral scholarship from the Consejo Nacional de Humanidades, Ciencias y Tecnologías (Scholarship Number: 245771; https://conahcyt.mx/). The data presented in this work form part of her doctoral dissertation. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Data Availability All relevant data for this study is fully available without restrictions from the Figshare repository. The dataset can be accessed at the following DOI: https://doi.org/10.6084/m9.figshare.27111460. Introduction

Humans form cross-modal associations (CMAs) between sounds and images, which play a vital role in integrating semantic representations within language [1]. Supporting this, fMRI studies have shown that the temporal lobe of the human brain is actively involved in CMAs [2, 3] between words and visual objects [4]. It is believed that CMAs between phonological "templates"—developed in human infants by listening to caretakers—and observed objects are essential for creating semantic representations and aiding the production of a child’s first words [5–8]. Similarly, auditory templates have been proposed as a mechanism for vocal production in birds [9–13] and marmoset monkeys [14]. Recent studies, such as those by Carouso-Peck and Goldstein [15, 16], have also shown that visual signals during social interactions can also influence vocal production in birds. However, only a few ethological studies have suggested the existence of CMAs between vocal sounds and visual cues for semantic communication [17]. For instance, research has observed that vervet monkeys respond to calls signaling the presence of predators by looking upwards, downwards, or climbing into trees [18].

Neurophysiological recordings in monkeys have shown that the prefrontal cortex (PFC)—a brain area homologous to that in humans—utilizes working memory (WM) circuits [19] to perform CMAs between voices and faces [20–32], receiving inputs from various sensory regions [33–36]. CMAs have also been observed in the auditory and visual areas of the temporal lobe [37–45]. Notably, trained macaques have demonstrated the ability to perform cross-modal discriminations between visual and tactile objects [46, 47], and between stimuli that could be considered non-ethologically relevant (NER), such as between pitch and color [48] and between amodal information (i.e., information that does not belong to a particular modality) [49] such as numerosity [50] and flutter frequencies [51–54]. However, it remains to be explored whether non-human primates can establish CMAs between NER stimuli that are important for human language, like words—which monkeys can discriminate phonetically [55, 56], and pictures.

Therefore, to assess whether monkeys can form CMAs between NER stimuli, we trained two rhesus macaques in a delayed crossmodal match-to-sample task (DCMMS). We specifically designed the task to temporally separate auditory and visual stimuli, thus engaging WM circuits to retain one modality in mind while awaiting the corresponding cross-modal stimulus. Unlike prior studies, this task required the monkeys to retain auditory information during a 3-second WM period and then use this information to select a matching visual from a set of 2–4 pictures displayed simultaneously on a screen after the delay.

Our results show that rhesus monkeys can accurately identify sounds produced by various emitters and match them with images despite the temporal gap, highlighting the crucial role of WM circuits not only for storing information but also for actively evaluating the equivalence between stimuli of different sensory modalities. This finding suggests substantial similarities with human cognitive processing in analogous tasks [57, 58] and paves the way for future neurophysiological studies focused on identifying the specific brain pathways and mechanisms involved in these cross-modal processes.

 Materials and methods Ethics statement

Animal welfare was a priority throughout the study, conducted in strict accordance with the recommendations of the Official Mexican Norm for the Care and Use of Laboratory Animals (NOM-062-ZOO-1999). The protocol was approved by UNAM’s IACUC (i.e., Comité Institucional para el Cuidado y Uso de Animales de Laboratorio; CICUAL; Protocol number: LLS200-22). Descriptions comply with the ARRIVE recommended guidelines [59]. The portrayal of one of the authors of this manuscript was used in the experiments and has given written informed consent to publish this case details.

 Subjects

Two adult rhesus monkeys (Macaca mulatta), a 10-year-old female (monkey G, 7 kg) and a 12-year-old male (monkey M, 12 kg) participated in the experiments. The animals had no previous training in any other task and were not subjected to any surgery or head restraint for this behavioral study. We adhered to the 3R principles (Replacement, Reduction, Refinement) [60]; therefore, we achieved statistical significance for the study in the number of trials each monkey performed rather than in the number of animals employed. The monkeys were housed in cages in a temperature-controlled room (22°C) with filtered air and day/night light cycles. They had free access to a balanced diet of dry food (pellets) supplemented with nuts, fresh fruits, and vegetables. Regular weight monitoring and veterinary check-ups ensured their health and well-being. The monkeys also had access to an enriched environment with toys, a recreation area for climbing and socializing with other monkeys four days a week, and opportunities for grooming through mesh sliding doors. In addition, cartoons and wildlife videos of content unrelated to the experiments, were presented on TV for no more than four hours a day. However, the face and voice of one of the researchers with whom the monkeys interacted were used during the experiments. To motivate participation in the experiments, the monkeys followed a water restriction protocol for 12–15 hours before experimental sessions (Monday to Friday, with water intake of 20–30 ml/kg achieved during the experimental sessions and ad libitum on weekends). After the 2–3-hour experimental sessions, they received 150g rations of fruits and vegetables.

 Experimental setup

The monkeys were trained to leave their cages and sit in a primate chair (Crist Instrument, INC.) for transfer to a soundproof booth adjacent to the vivarium for the experiments. The chair faced a touchscreen (ELO 2201L LED Display E107766, HD wide-aspect ratio 22in LCD) positioned 30 cm in front. A spring lever below the touchscreen (ENV-610M, Med Associates) allowed the monkeys to initiate the trials. Two speakers were mounted above the touchscreen: a Yamaha MSP5 Studio (40 W, 0.050–40 kHz) and a Logitech speaker (12 W, 0.01–20 kHz). These speakers delivered the sounds and background noise at 45- and 55 dB SPL, respectively. The monkeys received liquid rewards through a stainless-steel mouthpiece attached to the chair (Reward delivery system 5-RLD-E2-C Gravity feed dispenser, Crist Instrument INC.).

 Acoustic stimuli

The experiment utilized a variety of sounds, including laboratory recordings of words and monkey vocalizations, as well as free online sounds of cow vocalizations (https://freesound.org/). The sounds were edited to a duration of 500ms, resampled to 44.1 kHz (with cutoff frequencies of 0.1–20 kHz), and finally normalized (RMS) with Adobe Audition® 6.0 software. The phonetic labels of words in Spanish in the text and figure legends were created using the Automatic Phonetic Transcription tool by Xavier López Morras (http://aucel.com/pln/transbase.html).

 Visual stimuli

The visual stimuli consisted of a red oval, grayscale cartoons of cows and monkeys, and pictures of human and monkey faces, as well as a cow circumscribed in ovals with a resolution of 200px/sq inch. Animal pictures used in the experiment were downloaded from free online sites and customized. However, the pictures shown in figures and supplementary information are similar but not identical to the original images used in the study; they were created for illustrative purposes only using an online AI image generator (https://www.fotor.com/ai-art-generator).

 Delayed crossmodal match-to-sample task

We trained two rhesus macaques in a DCMMS task to assess their ability to establish CMAs between sounds and images temporally decoupled. Each trial began with a 1° white cross appearing in the center of the touchscreen. In response to the cross, the monkeys had to press and hold down a lever so that a 0.5-second reference sound could be delivered. After hearing the sound, the animals had to wait during a 3-second delay until 2–4 four pictures were presented simultaneously at random positions but equidistant on an 4° radius from the center of the touchscreen. The monkeys were then allowed to release the lever and select, within a 3-second response window, the picture that matched the sound (S1 Video). Correct selections were rewarded with a drop of liquid.

After the monkeys learned the task (see the monkeys’ training section below), they were able to perform at different CMAs. Each CMA was established by associating a sound with a picture representing the same category of external stimulus (e.g., both corresponding to a human). For example, a CMA of the type ‘human’ consisted of the association between the word [si] and a human face. This way, CMAs of different types were created (e.g., monkey, cow, human, and color). In some cases, the monkeys associated a single sound with several pictures of the same type; for example, four monkey faces were associated with one ‘coo’, resulting in four ‘monkey’ CMAs (S1 Table). Each CMA at S1 Table was established by the monkeys after many sessions of practice (see the following methods’ sections). However, in an experimental condition which we designated as the ‘perceptual invariance experiment’, we explored the monkeys’ ability to recognize sounds uttered by different individuals the monkeys did not hear before the experiment. For example, a ‘monkey’ CMA substitution set was comprised of ten different coos (i.e., auditory versions uttered by different individuals) delivered randomly in different trials, but all those trials presented the same monkey picture as a match. Finally, all experimental sessions consisted of blocks of ~300 trials of intermixed CMAs. The Hit rate (HR) corresponds to the proportion of correct responses (i.e., audio-visual match) in a session; false alarms (FA) indicate the proportion of incorrect responses. Reaction times (RT) are the times to release the lever in response to the appearance of the pictures on the touchscreen. Motor times are the intervals between the lever release and the touching of the screen. The task was programmed using LabVIEW 2014 (64-bit SP1, National Instruments®). The artwork in the task description was created using a free online platform (https://www.fotor.com/ai-art generator).

 Monkeys training

To enhance the monkeys’ speed and efficiency in learning the DCMMS task, we tailored stimuli, durations, and rewards according to their ongoing performance. Initially, the animals were trained to produce the motor responses necessary for the task, such as pressing and releasing a lever and consistently activating the touchscreen. Rewards were given for holding down the lever when the cross appeared at the center of the touchscreen and for releasing the lever and touching the screen upon its disappearance. After the subjects completed more than 90% of the trials in consecutive sessions, we introduced a gray filled circle on the touchscreen that appeared at random positions, requiring the monkey to touch it to receive a reward. Within one or two weeks, the animals consistently reacted to the cross within a 500 ms window of appearance, maintained the lever pressed for 5–7 seconds, and released it upon the disappearance of the cross to touch the visual target.

In the subsequent training phase, the monkeys were required to respond to a tone (i.e., a 0.5-second 440Hz; 55 dB SPL) randomly emitted from speakers on either side of the screen. The goal was to indicate the direction of the sound by touching a right or left circle on the screen, which appeared simultaneously with the tone and then after a gradually increasing delay (from 1–3 seconds). Here, the objective was for the monkeys to associate the auditory and visual locations. However, after more than 35,000 trials (i.e., ~ 117 sessions), the performance remained at chance level. Consequently, we adopted a new approach that involved helping the monkeys to directly associate audio cues with specific images.

We replaced one circle with a cartoon image of a cow and added a 0.5-second broadband noise, so each trial featured either the tone or the noise. Rewards were given for correctly associating the cow cartoon with the broadband noise and the gray circle with the 440 Hz tone. From then on, sounds were delivered exclusively from a central speaker above the screen, and pictures appeared at different positions but were consistently separated by 180° (of visual angle) from each other. With this new training method, it took only a few sessions for Monkey G to begin performing above chance in associating the broadband noise with the cow cartoon (S1 Fig, upper leftmost panel). With many practice sessions, performance improved above chance, prompting us to gradually introduce new sounds and images to establish various CMAs. The initial CMAs involved only two different pictures on the touchscreen, while more complex associations involved the simultaneous presentation of three or four pictures.

 Learning measurements

Although the primary goal of our experiments was not to explore the learning process of macaques, we noted behavioral improvements throughout the sessions we aimed to document. To quantify this, we fitted learning curves to the performance at each CMA across sessions, thereby assessing the monkeys’ learning progress. For this analysis, we applied the Rescorla-Wagner model, a well-established framework in associative learning [61], which explains learning as the formation of associations between conditioned and unconditioned stimuli. The process of deriving the learning curves required solving the following ordinary differential equation: dVdt=αβ(λ−V)

This equation describes the progression of associative strength (V) in response to trained conditioned stimuli, dependent on the number of training trials (t). The model provided the parameters for this equation: αβ, which is the product of the salience of the conditioned stimuli and the strength of the unconditioned stimuli (assumed constant during training, though modifications are possible [62]), and λ, representing the maximum possible associative strength towards the unconditioned stimulus. From the learning curves derived from this model, we extracted three additional parameters. Y0 measured initial performance, representing the starting point of the curve along the Y-axis. Parameter γ, indicating statistical learning onset, was determined as the first session in which performance reliably exceeded chance, defined as surpassing two standard deviations from the mean probability of a correct response under a binomial distribution (where p = chance level, and n = average number of trials per session). Finally, the derivatives of these learning curves, coupled with predefined thresholds, allowed us to determine the ‘trend-to-optimal’ experimental session for each CMA (δ), marking the session where changes in performance from one session to the next did not exceed a designated minimal rate of improvement of y’ = 0.01, indicating an approach towards a learning plateau.

 Statistical analysis

We focused most of our analyses on data collected post-training, after the monkeys’ performance reached an asymptotic level, with their choices consistently exceeding the chance level. We used various statistical tests to compare RTs across different conditions. These tests included Spearman rank correlations to test the relationship between reaction time distributions and the number of pictures on the touchscreen and a Kruskal-Wallis test for differences between CMAs. If the Kruskal-Wallis’s test indicated a significant difference, we followed up with Mann-Whitney tests to compare conditions such as trials having 2 or 3 pictures when the same sound was presented. Finally, Bonferroni post hoc tests were used for multiple comparisons. The monkeys’ chance performance threshold depended on the number of pictures displayed; for monkey M, the chance was 0.5 since it was performed in two picture sets only, while for monkey G it was 0.25 at four picture sets. Analyses were performed using MATLAB R2022 (MathWorks).

 Results

To investigate the ability of two rhesus monkeys to form CMAs between auditory and visual stimuli, we engaged them in a DCMMS task. Each trial commenced with the monkeys hearing a reference sound, followed by a 3-second delay, after which 2–4 pictures were displayed on the screen. Their task was to identify on the touchscreen the picture that corresponded to the sound (Fig 1A). The monkeys mastered fourteen CMAs after associating six distinct sounds—including broadband noise, animal vocalizations like a coo and a moo, and words such as [‘tsan. gi], [si], and [‘ro. xo]—with fourteen images (S1 Table). The trials varied, presenting either 2–4 pictures for Monkey G, while consistently presenting 2 pictures for Monkey M. Illustrative examples of four CMAs are depicted in Fig 1B. Fig 1C and 1D shows the monkeys’ hit rate (HR) and false alarm rate (FA) across these CMAs (S2 Table). For example, when a coo sound was used as the reference, Monkey G correctly matched it with the monkey face 87.43% of the time, while its most frequent incorrect choice was the cow face, selected 5.94% of the time (Fig 1C, open boxplots). Overall, Monkey G exhibited a HR of 85.12% ± 9.11 (mean ± SD), and Monkey M achieved a HR of 87.07% ± 5.71. Statistical analysis showed no bias in their selection of specific positions on the touchscreen (one-way ANOVA with multiple pairwise comparisons; Tukey’s HSD, p < 0.05) (S2 Fig). These outcomes indicate that both monkeys proficiently learned to discriminate each sound, against 2 to 4 pictures.

10.1371/journal.pone.0317183.g001 Delayed crossmodal match-to-sample task.

(A) Task Events. A trial begins with the monkey pressing a lever in response to a cross appearing in the center of the touchscreen. This is followed by a 0.5-second reference sound, succeeded by a 3-second delay. After the delay, 2–4 pictures are simultaneously presented on the touchscreen. The monkey must then release the lever and touch the picture that matches the sample sound to receive a reward. LP indicates lever press. (B) Examples of Crossmodal Associations. Each column displays a CMA between a sound, represented visually by its sonogram and spectrogram and a picture. The sounds, marked in black, include two Spanish words (in IPA notation) and vocalizations of a monkey and a cow. (C) HR (close boxplots) and FAs (open boxplots) during the presentations of the CMAs shown in B. The dashed line indicates the performance at chance level (i.e., 25% for sounds discriminated against four pictures). The reference sound is labeled in red at the top of the graph. (D) Same as in C, but for Monkey M. The dashed line is set at the 50% chance level (i.e., two pictures on the screen). The pictures are similar but not identical to the original images used in the study and are therefore for illustrative purposes only.

 Rhesus monkeys can learn cross-modal associations between stimuli of different types

The monkeys successfully established each CMA after several sessions of engaging in the DCMMS task, during which we initially presented two pictures simultaneously; only one of which corresponded to the played sound. To investigate the learning dynamics, we measured four learning parameters derived from fitting simple associative learning curves to the performance data across sessions. These parameters included the HR in the first session (y0), and the sessions marking statistical learning (γ), increasing learning (δ), and asymptote of learning (λ), respectively (refer to Methods for detailed descriptions). The left panel in Fig 2A illustrates Monkey G’s performance for the CMA between the coo sound and the monkey cartoon across sessions. Initially, the performance before learning was at chance level (~300 trials; see methods section on monkeys’ training and learning measurements), aligning with the intersection of the learning curve (black line) and the Y-axis, termed Y0. Subsequently, the γ performance level was reached after eight sessions from Y0 (~2700 trials); this level is defined as the session when the HR was above chance, marked by the intersection between the left edge of the gray box and the learning curve. A consistent increase in HR continued until the 15th session, reaching δ performance (right edge of the gray box), and by approximately the 40th session, the performance stabilized at the λ level, where changes in performance from one session to the next were insignificant. Similarly, middle and right panels in Fig 2A show two CMAs learned in trials ending with 3 and 4 pictures, respectively.

10.1371/journal.pone.0317183.g002 Learning CMAs in monkeys.

(A) Monkey G’s learning progress for three CMAs across sessions with trials presenting 2, 3, or 4 pictures simultaneously on the screen. The black line represents the average performance across sessions, while the blue line maps the first derivative of performance over training sessions (y’ values), illustrating the rate of change at each session. The initial HR (Y0) was near chance level (indicated by the black line at the ordinates), followed by γ (the left edge of the gray box), where the HR statistically exceeded chance. The learning parameter δ, marks a period when HR increased consistently above chance, culminating in a performance plateau at the session denoted by the asymptote of learning λ. (B) Sessions before δ for each CMAs. (C) Represents the average performance of Monkey G across all CMAs over the sessions. (D) Same as in C, but for Monkey M. The pictures are similar but not identical to the original images used in the study and are therefore for illustrative purposes only.

S1 Fig shows performance evolving at each CMA across sessions in monkey G. In addition, the number of sessions needed for reaching sustained performance (i.e., the δ parameter) decreased in most new CMAs as the monkeys learned the aim of the task (Fig 2B). However, for the ‘color’ CMA formed by the word [ro. xo] (Spanish for ’red’) and the red oval, Monkey G spent ~14 sessions to reach δ at the conditions where four pictures appeared on the screen. We interpret this increase in learning sessions as the result of introducing those stimuli for the first time in trials that presented four pictures on the screen. Finally, Fig 2C and 2D present the mean HR for all CMAs across sessions for both monkeys. We interpret the reduction in γ and δ as the monkeys solving the cognitive control of the motor behavior required for the task (procedure memory), e.g., pressing and releasing the lever and interacting correctly with the touchscreen, so that once this was done, the animals could focus only on learning the CMA associations.

 The RTs increased as a function of the selected picture and number of pictures on the touchscreen

To explore how different sounds and pictures influenced the monkeys’ ability to find a cross-modal match, we analyzed the RTs during hits across various CMAs. Fig 3A displays Monkey G’s RTs and motor times (MT) distributions across four CMAs. Notable differences are observed between the RT distributions, which pertain to the decision-making period (i.e., the time taken to decide which picture on the touchscreen matches the sound before releasing the lever). In contrast, the MT distributions, which relate to the stereotyped arm movement toward the chosen picture, showed no differences.

10.1371/journal.pone.0317183.g003 Crossmodal associations influenced the monkeys’ reaction times.

(A) Cumulative probabilities of reaction and motor times across four CMAs. (B) Left panel, pie charts displaying hit rates in sets presenting three CMAs. In all trials, the reference sound was consistently a "coo," but the match in each session was one of the four monkey pictures. Hits are depicted in colors, while false alarms (FAs), occurred when the monkey chose a non-matching picture, are shown in gray or white. Right panel, reaction time (RT) distributions of hits are illustrated with the same color coding as in the left panel. Inset, FA distributions produced in trials where one of the four monkey pictures was presented as a match, but a picture of a ‘human’ or a ‘cow’ was selected. (C) Same format as B but for ’cow’ CMAs. (D) The standard deviations (STDs) of the RT distributions increased as a function of their means during hits, false alarms (FAs), and in trials with two, three, or four pictures on the screen. (E) Plot of the monkeys’ HRs as a function of the mean RTs of hit distributions in D.

To assess whether acoustic or visual information primarily influenced the monkeys’ RT distributions, we analyzed RTs to different pictures associated with a single sound (S3 Table), in trials presenting 3 pictures simultaneously. For instance, Fig 3B shows Monkey G’s RT distributions (right panel) during correct responses to various pictures of the type ‘monkey’ (left panel) associated with a single ‘coo’. Fig 3C shows the same for pictures of the type ‘cow’ associated with a single ‘moo’ sound. The RT distributions differed significantly in both instances (p < 0.001, Kruskal-Wallis’s test), indicating that since the sounds were constant, the differences in RTs must have stemmed from variations among the pictures. This trend continued across all CMAs where different pictures were associated with the same sound (p < 0.001 for all comparisons, post hoc Mann-Whitney U tests with Bonferroni correction); pairwise comparisons between all pictures with each sound revealed significant differences in RT distributions (p < 0.01 for 71.43% of coo comparisons, 76.19% for moo comparisons, and 82.14% for [si] comparisons; Mann-Whitney U tests with Bonferroni correction). A similar effect is observed for FAs as shown in Fig 3B (insets), where the differences in RTs resulted from incorrect matchings (p < 0.001, Kruskal-Wallis’s test).

Furthermore, Fig 3D shows that both the mean and the standard deviation (STD) of the RT distributions increased with the number of pictures displayed on the screen (2–4 pictures), indicating that locating the crossmodal match took longer as the number of distractor pictures increased. This tendency aligns with Weber’s Law and studies in time processing [63]. Here, we interpret that the variation in STDs suggests that the faster RTs likely occurred when the matching picture was found first among the pool of pictures on the screen, and longer RTs when the match was found last. Notably, these variations in RTs did not impact the accuracy across different CMAs (Fig 3E). These findings imply that RT was more heavily influenced by the amount of visual information processed than by differences in sounds.

 The monkeys recognized sounds uttered by different speakers

We explored whether the monkeys could recognize sounds of the same type they learned but uttered by different individuals they did not hear before (Fig 4A, S4 Table). Fig 4B show how Monkey G performed above the 25% chance level in 98.33% of cases (paired-sample t-test, p < 0.05). Notably, the RTs during correct responses grouped into the four CMAs (i.e., pictures) used at this experiment rather than in the number of new sounds (Fig 4C). Fig 4D shows that Monkey M presented a similar effect in 3 CMAs, performing above 50% chance in trials of only two pictures on the touchscreen (i.e., 72.22% of the versions; paired-sample t-test, p < 0.05) and distributing RTs by picture category (Fig 4E), further supporting the notion of auditory invariance. In other words, regardless of variations in sounds, the animals could recognize them. Altogether, our findings suggest that monkeys can perform CMAs based on the ability to perceive equivalences within different sounds of the same type.

10.1371/journal.pone.0317183.g004 Monkeys recognized sounds uttered by different individuals.

(A) Spectrograms from various speakers depicting the Spanish word [’ro. xo] (red). The spectrogram of the learned sound is on the left. (B) Hit rate of monkey G in all sounds’ versions. Closed boxes on the left represent HR in the learned sounds (L). Open boxes, different versions’ HR. Closed boxes on the right of each group correspond to the HR in versions comprised of double repetitions of some sounds including L. (C) Cumulative density functions of the RTs in the learned sounds (bold lines) of monkey G and their versions. Notice how the distributions group by the picture category rather than by sounds. (D) Same as in B, but for monkey M. (E) Same as C, but for monkey M. The pictures are similar but not identical to the original images used in the study and are therefore for illustrative purposes only.

 Discussion

To investigate if rhesus monkeys can associate sounds with images regardless of their ethological relevance, we engaged two of these primates in a DCMMS task. To solve the task, monkeys had to retain in WM either an auditory replay or a crossmodal equivalent of the sounds (i.e., a face) and compare the memory against different pictures to find the match. Evaluation of their performance across various tests yielded two main outcomes: 1) the monkeys adeptly formed associations between sounds (e.g., animal vocalizations, words) and pictures (e.g., faces, cartoons), demonstrating human-like word-object associations that form the basis of language (Figs 1 and 2, S1 Table), and 2) these associations generalized even when the vocalizations and words they learned were uttered by different voices (Fig 4, S4 Table). Subsequent sections will detail these findings and explore the potential mechanisms to establish CMAs.

Rhesus macaques create crossmodal associations between sounds and images of different types

Previous studies demonstrated that monkeys could perform crossmodal discriminations of supramodal information such as numerosity and flutter frequencies [50–54] and learn and group numerous sounds into categories irrelevant to their ethology [55, 56]. However, establishing cross-modal associations between NER categories in monkeys has proved to be challenging [37, 46–48]. In training two rhesus monkeys in the DCMMS task, we initially encountered hurdles as the monkeys tended to disregard sounds [64, 65]. To counter this, training began with sound detection and progressively moved to crossmodal associations. We obtained different learning parameters from the monkeys’ performances in each CMA across sessions (Fig 2).

During the initial training phase, the monkeys learned to interact with the task’s apparatus (i.e., pressing the lever and touching the screen), achieving controlled motor responses within one or two weeks. Learning the first CMA (i.e., a broadband noise paired with a cow cartoon) required many sessions. Subsequent CMAs achieved statistical performance in just a few sessions; however, the animals excelled at the task after many practice sessions. We found no clear evidence that learning CMAs that included possible ethologically relevant stimuli like human and monkey faces, or coos [20–31] were facilitated more than other CMAs to which they had no previous exposure. In other words, the animals learned all CMAs at similar rates, providing behavioral data that could be highly informative regarding the brain responses underlying CMAs. Future neurophysiological evidence could build on these behavioral findings.

Three of our results aligned with the idea that CMAs could be created from templates [5–12]: 1) monkeys learn each new CMA faster; 2) mastering a CMA requires a prolonged period, akin to learning to speak in humans; 3) the animals’ performance remained consistently high when the same vocalizations or words were presented with different voices, suggesting that the acoustic variations activated auditory templates, similar to how formants in words trigger acoustic recognition in monkeys [55]. Similarly, our results suggest that visual templates could create perceptual equivalence among different faces of the same type (Fig 3B). This is the strongest evidence to date that supports the possibility that monkeys can connect auditory and visual templates as humans do.

The formation of supramodal circuits linking vocalizations with other motor behaviors [12, 13] has suggested that the integration process in NHPs might similarly involve motor and spatial associations across sensory modalities [66, 67]. In our task, such associations were unnecessary since the animals had to match a sound with the corresponding picture, which was presented at different locations every trial. Moreover, studies exploring the convergence of crossmodal information in WM [22–27, 48] indicate that while motor or spatial associations may facilitate initial learning, more abstract associations such as numerosity or flutter [50, 54], extending beyond immediate and innate categories, can be developed through direct CMAs. Therefore, the monkeys performing our task could have created direct connections between auditory and visual templates.

 Working memory mechanisms for crossmodal matching

In contrast to other tasks [21, 37], our monkeys had to retain information about sounds over a 3-second delay and use it to compare with different pictures until they found a match, similar to previous work on the intra- and cross-modal discrimination of flutter [51–54]. Given that the animals performed above chance in all CMAs, and strategies such as selecting a particular picture or location cannot explain their performance (S2 Fig), we conclude that the most parsimonious explanation was the cross-modal matching of sounds and pictures. In other words, monkeys must have retained information about the sounds in WM to find the cross-modal match presented 3 seconds later. A candidate brain region for the type of WM involved in our task is the PFC [19], which participates in the retaining of parametric and nonparametric information of different sensory modalities compared intra- or cross-modally [2, 3, 20–31]. Notably, the PFC is also responsible for intramodal associations of stimuli separated in space and time [50]. Therefore, it is probably capable of translating information cross-modally; in our task, this could involve possibly invoking visual representations after hearing sounds, thus retaining visual information in WM for later comparisons with the pictures, rather than keeping the reference sound in working memory until the pictures appear.

On the other hand, it is well documented that PFC activity in the context of CMAs is activated by ethologically relevant stimuli such as conspecific faces and voices in monkeys not engaged in their active recognitions [26, 42]. This suggests that ethologically relevant circuits could be established there since birth [26, 31]. Therefore, active cross-modal discrimination and the learning of CMAs between non-ethological stimuli may occur in other areas of the temporal lobe, known to represent and integrate auditory and visual objects [37–45], showing activations to superimposed audiovisual stimuli [37], perhaps to facilitate the recognition of individuals within their social group [26]. However, only future neurophysiological experiments in monkeys trained in the DCMMS task would reveal not only how and where in the brain non-ethological auditory and visual categories are learned, stored, and associated cross-modally, but also whether auditory or visual images invoked by sounds are retained in WM during the resolution of the task.

 Supporting information Monkey G performing the DCMMS task.

(MP4)

 Monkeys’ learning parameters and hit rate.

(PDF)

 Overall hit rate (mean ± STD) in four CMAs.

(PDF)

 The proportion (mean ± STD) of pictures selected.

Selections of pictures during hits and FAs in the condition when one sound was associated with different pictures of the same type.

(PDF)

 Hit rate (mean ± STD) in different versions of the learned sounds.

(PDF)

 Learning of CMAs.

(PDF)

 Hit rate and reaction times at different picture locations.

To analyze biases toward selecting a P at any angle from the center of the touchscreen, we performed a one-way ANOVA, False Discovery Rate corrected for multiple pairwise comparisons. Monkey M showed no location bias (p-values > 0.034). Monkey G, however, exhibited a significant effect for the monkey face position (F [15, 160.67] = 1.97; p = 0.014) and the cow face (F [15, 150.619] = 2.51; p = 0.001), but not for the human (p = 0.988). Post-hoc analysis (Tukey’s HSD) revealed these differences occurred in angles < 90° within each screen quadrant. In other words, while there were biases in selecting pictures at angles, there was no consistent preference for a specific quadrant. Based on these findings, the behavioral results presented here correspond to subsequent experiments presenting pictures only in four quadrants.

(PDF)

We extend our gratitude to Vani Rajendran for valuable feedback; Francisco Pérez, Gerardo Coello, and Ana María Escalante from the Computing Department of the IFC; Aurey Galván and Manuel Ortínez of the IFC workshop; and Claudia Rivera for veterinary assistance. Additionally, we thank Centenario 107 for their hospitality.

 References Bowerman M, Choi S. Shaping meanings for language: universal and language–specific in the acquisition of spatial semantic categories. In: Bowerman M, Levinson S, editors. Language acquisition and conceptual development. Cambridge, UK; 2001. p. 475–511. Beauchamp MS, Lee KE, Argall BD, Martin A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron. 2004;41(5):809–23. doi: 10.1016/s0896-6273(04)00070-4 15003179 Noesselt T, Rieger JW, Schoenfeld MA, Kanowski M, Hinrichs H, Heinze HJ, et al. Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices. Journal of Neuroscience. 2007 Oct 17;27(42):11431–41. doi: 10.1523/JNEUROSCI.2252-07.2007 17942738 Mesulam MM, Wieneke C, Hurley R, Rademaker A, Thompson CK, Weintraub S, Rogalski EJ. Words and objects at the tip of the left temporal lobe in primary progressive aphasia. Brain. 2013 Feb;136(Pt 2):601–18. doi: 10.1093/brain/aws336 Epub 2013 Jan 29. 23361063; PMCID: PMC3572925. Vihman M, Croft W. Phonological development: Toward a “radical” templatic phonology. Linguistics. 2007 Jul 20;45(4):683–725. Coffey JR, Shafto CL, Geren JC, Snedeker J. The effects of maternal input on language in the absence of genetic confounds: Vocabulary development in internationally adopted children. Child Dev. 2022 Jan 1;93(1):237–53. doi: 10.1111/cdev.13688 34882780 Bloom L, Tinker E. The intentionality model and language acquisition: engagement, effort, and the essential tension in development. Monogr Soc Res Child Dev. 2001;66(4):1–91. 11799833 Locke JL. Movement patterns in spoken language. Science. 2000 Apr 21;288(5465):449–51. doi: 10.1126/science.288.5465.449 10798981 Goldstein MH, King AP, West MJ. Social interaction shapes babbling: testing parallels between birdsong and speech. Proc Natl Acad Sci U S A. 2003 Jun 24;100(13):8030–5. doi: 10.1073/pnas.1332441100 12808137 Mooney R. Neurobiology of song learning. Curr Opin Neurobiol. 2009 Dec;19(6):654. doi: 10.1016/j.conb.2009.10.004 19892546 Margoliash D. Evaluating theories of bird song learning: implications for future directions. J Comp Physiol A Neuroethol Sens Neural Behav Physiol. 2002 Dec 1;188(11–12):851–66. doi: 10.1007/s00359-002-0351-5 12471486 Chen Y, Matheson LE, Sakata JT. Mechanisms underlying the social enhancement of vocal learning in songbirds. Proc Natl Acad Sci U S A. 2016 Jun 14;113(24):6641–6. doi: 10.1073/pnas.1522306113 27247385 Hisey E, Kearney MG, Mooney R. A common neural circuit mechanism for internally guided and externally reinforced forms of motor learning. Nat Neurosci. 2018 Apr 1;21(4):589–97. doi: 10.1038/s41593-018-0092-6 29483664 Takahashi DY, Fenley AR, Teramoto Y, Narayanan DZ, Borjon JI, Holmes P, et al. Language Development. The developmental dynamics of marmoset monkey vocal production. Science. 2015 Aug 14;349(6249):734–8. doi: 10.1126/science.aab1058 26273055 Carouso-Peck S, Goldstein MH. Female Social Feedback Reveals Non-imitative Mechanisms of Vocal Learning in Zebra Finches. Curr Biol. 2019 Feb 18;29(4):631–636.e3. doi: 10.1016/j.cub.2018.12.026 30713105 Takahashi DY. Vocal Learning: Shaping by Social Reinforcement. Curr Biol. 2019 Feb 18;29(4): R125–7. doi: 10.1016/j.cub.2019.01.001 30779900 Ratcliffe VF, Taylor AM, Reby D. Cross-modal correspondences in non-human mammal communication. Multisens Res. 2016;74(5657):49–91. doi: 10.1163/22134808-00002509 27311291 Seyfarth RM, Cheney DL, Marler P. Vervet monkey alarm calls: Semantic communication in a free-ranging primate. Anim Behav. 1980;28(4):1070–94. Romo R, Brody CD, Hernández a, Lemus L. Neuronal correlates of parametric working memory in the prefrontal cortex. Nature. 1999;399(June):470–3. doi: 10.1038/20939 10365959 Plakke B, Hwang J, Romanski LM. Inactivation of Primate Prefrontal Cortex Impairs Auditory and Audiovisual Working Memory. 2015;35(26):9666–75. Diehl MM, Plakke BA, Albuquerque R, Romanski LM. Representation of expression and identity by ventral prefrontal neurons. Neuroscience. 2022;496(2022):243–60. doi: 10.1016/j.neuroscience.2022.05.033 35654293 Hwang J, Romanski LM. Prefrontal neuronal responses during audiovisual mnemonic processing. J Neurosci. 2015 Jan 21;35(3):960–71. doi: 10.1523/JNEUROSCI.1328-14.2015 25609614 Sugihara T, Diltz MD, Averbeck BB, Romanski LM. Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. J Neurosci. 2006;26(43):11138–47. doi: 10.1523/JNEUROSCI.3550-06.2006 17065454 Romanski LM, Sharma KK. Multisensory interactions of face and vocal information during perception and memory in ventrolateral prefrontal cortex. Philos Trans R Soc Lond B Biol Sci. 2023 Sep 25;378(1886). doi: 10.1098/rstb.2022.0343 37545305 Romanski LM. Representation and integration of auditory and visual stimuli in the primate ventral lateral prefrontal cortex. Cereb Cortex. 2007 Sep;17. doi: 10.1093/cercor/bhm099 17634387 Adachi I, Hampton RR. Rhesus monkeys see who they hear: Spontaneous cross-modal memory for familiar conspecifics. PLoS One. 2011;6(8). doi: 10.1371/journal.pone.0023345 21887244 Diehl MM, Romanski LM. Responses of Prefrontal Multisensory Neurons to Mismatching Faces and Vocalizations. 2014;34(34):11233–43. Romanski LM, Averbeck BB, Diltz M. Neural representation of vocalizations in the primate ventrolateral prefrontal cortex. J Neurophysiol. 2005 Feb;93(2):734–47. doi: 10.1152/jn.00675.2004 15371495 Romanski LM, Goldman-Rakic PS. An auditory domain in primate prefrontal cortex. Nat Neurosci. 2002;5(1):15–6. doi: 10.1038/nn781 11753413 Cohen YE, Theunissen F, Russ BE, Gill P. Acoustic features of rhesus vocalizations and their representation in the ventrolateral prefrontal cortex. J Neurophysiol. 2007;97(2):1470–84. doi: 10.1152/jn.00769.2006 17135477 Gifford GW, MacLean K a, Hauser MD, Cohen YE. The neurophysiology of functionally meaningful categories: macaque ventrolateral prefrontal cortex plays a critical role in spontaneous categorization of species-specific vocalizations. J Cogn Neurosci. 2005;17(9):1471–82. doi: 10.1162/0898929054985464 16197700 Huang Y, Brosch M. Neuronal activity in primate prefrontal cortex related to goal-directed behavior during auditory working memory tasks. Brain Res. 2016 Jun 1;1640(Pt B):314–27. doi: 10.1016/j.brainres.2016.02.010 26874071 Eacott MJ, Gaffan D. Inferotemporal-frontal Disconnection: The Uncinate Fascicle and Visual Associative Learning in Monkeys. Eur J Neurosci. 1992;4(12):1320–32. doi: 10.1111/j.1460-9568.1992.tb00157.x 12106395 Romanski LM, Bates JF, Goldman-Rakic PS. Auditory belt and parabelt projections to the prefrontal cortex in the rhesus monkey. Journal of Comparative Neurology. 1999;403(April 1998):141–57. doi: 10.1002/(sici)1096-9861(19990111)403:2<141::aid-cne1>3.0.co;2-v 9886040 Romanski LM, Tian B, Fritz J, Mishkin M, Goldman-Rakic PS, Rauschecker JP. Dual streams of auditory afferents target multiple domains in the primate prefrontal cortex. Nat Neurosci. 1999; 2:1131–6. doi: 10.1038/16056 10570492 Gaffan D, Harrison S. Auditory-visual associations, hemispheric specialization and temporal-frontal interaction in the rhesus monkey. Brain. 1991; 114:2133–44. doi: 10.1093/brain/114.5.2133 1933238 Chandrasekaran C, Lemus L, Ghazanfar AA. Dynamic faces speed up the onset of auditory cortical spiking responses during vocal detection. Proc Natl Acad Sci U S A. 2013;110: E4668–77. doi: 10.1073/pnas.1312518110 24218574 Ghazanfar AA, Chandrasekaran C, Logothetis NK. Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys. J Neurosci. 2008;28(17):4457–69. doi: 10.1523/JNEUROSCI.0541-08.2008 18434524 Foxe JJ, Schroeder CE. The case for feedforward multisensory convergence during early cortical processing. Neuroreport. 2005 Apr 4;16(5):419–23. doi: 10.1097/00001756-200504040-00001 15770144 Ghazanfar AA, Schroeder CE. Is neocortex essentially multisensory? 2006;10(6). doi: 10.1016/j.tics.2006.04.008 16713325 Huang Y, Brosch M. Behavior-related visual activations in the auditory cortex of nonhuman primates. Prog Neurobiol. 2024 Sep 1;240. doi: 10.1016/j.pneurobio.2024.102637 38879074 Perrodin C, Kayser C, Logothetis NK, Petkov CI. Auditory and visual modulation of temporal lobe neurons in voice-sensitive and association cortices. Journal of Neuroscience. 2014;34(7):2524–37. doi: 10.1523/JNEUROSCI.2805-13.2014 24523543 Chandrasekaran C, Ghazanfar AA. Different neural frequency bands integrate faces and voices differently in the superior temporal sulcus. J Neurophysiol. 2009 Feb;101(2):773–88. doi: 10.1152/jn.90843.2008 19036867 Dahl CD, Logothetis NK, Kayser C. Modulation of visual responses in the superior temporal sulcus by audio-visual congruency. Front Integr Neurosci. 2010 Apr;4(APRIL 2010). doi: 10.3389/fnint.2010.00010 20428507 Tyree TJ, Metke M, Miller CT. Cross-modal representation of identity in the primate hippocampus. Science. 2023;382(6669):417–23. doi: 10.1126/science.adf0460 37883535 Zhou YD, Fuster JM. Visuo-tactile cross-modal associations in cortical somatosensory cells. Proc Natl Acad Sci U S A. 2000;97(17):9777–82. doi: 10.1073/pnas.97.17.9777 10944237 Weiskrantz L, Cowey A. Cross modal matching in the rhesus monkey using a single pair of stimuli. Neuropsychologia. 1975;13(3):257–61. doi: 10.1016/0028-3932(75)90001-9 808744 Fuster JM, Bodner M, Kroger JK. Cross-modal and cross-temporal association in neurons of frontal cortex. Nature. 2000;405(6784):347–51. doi: 10.1038/35012613 10830963 Stein BE, Meredith MA. The merging of the senses. Cambridge, Massachusetts: MIT Press; 1993. doi: 10.1162/jocn.1993.5.3.373 Nieder A. Supramodal numerosity selectivity of neurons in primate prefrontal and posterior parietal cortices. Proceedings of the National Academy of Sciences. 2012;109(29):11860–5. doi: 10.1073/pnas.1204580109 22761312 Lemus L, Hernández A, Luna R, Zainos A, Romo R. Do sensory cortices process more than one sensory modality during perceptual judgments? Neuron. 2010; 67:335–48. doi: 10.1016/j.neuron.2010.06.015 20670839 Lemus L, Hernandez a, Romo R. Neural encoding of auditory discrimination in ventral premotor cortex. Proc Natl Acad Sci U S A. 2009; 106:14640–5. doi: 10.1073/pnas.0907505106 19667191 Lemus L, Hernández A, Romo R. Neural codes for perceptual discrimination of acoustic flutter in the primate auditory cortex. Proc Natl Acad Sci U S A. 2009;106(23):9471–6. doi: 10.1073/pnas.0904066106 19458263 Vergara J, Rivera N, Rossi-Pool R, Romo R. A Neural Parametric Code for Storing Information of More than One Sensory Modality in Working Memory. Neuron. 2016;89(1):54–62. doi: 10.1016/j.neuron.2015.11.026 26711117 Melchor J, Vergara J, Figueroa T, Morán I, Lemus L. Formant-based recognition of words and other naturalistic sounds in rhesus monkeys. Front Neurosci. 2021; 15:1–10. doi: 10.3389/fnins.2021.728686 34776842 Morán I, Perez-Orive J, Melchor J, Figueroa T, Lemus L. Auditory decisions in the supplementary motor area. Prog Neurobiol. 2021;202:1–11. doi: 10.1016/j.pneurobio.2021.102053 33957182 Chandrasekaran C, Lemus L, Trubanova A, Gondan M, Ghazanfar A a. Monkeys and humans share a common computation for face/voice integration. PLoS Comput Biol. 2011;7(9). Majerus S, Cowan N, Péters F, Van Calster L, Phillips C, Schrouff J. Cross-Modal Decoding of Neural Patterns Associated with Working Memory: Evidence for Attention-Based Accounts of Working Memory. Cerebral Cortex. 2016 Jan 1;26(1):166–79. doi: 10.1093/cercor/bhu189 25146374 du Sert NP, Hurst V, Ahluwalia A, Alam S, Avey MT, Baker M, et al. The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research. PLoS Biol. 2020 Jul 1;18(7). Russell WMS, Burch RL. The Principles of Humane Experimental Technique. London: The Universities Federation for Animal Welfare; 1959. Rescorla RA, Wagner AR. A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and non-reinforcement. In: Black A, Prokasy W, editors. In Classical conditioning II. New York: Appleton-Century Crofts; 1972. p. 64–99. Treviño M. Associative learning through acquired salience. Front Behav Neurosci. 2016 Jan 11; 9:168673. doi: 10.3389/fnbeh.2015.00353 26793078 Merchant H, Zarco W, Prado L. Do we have a common mechanism for measuring time in the hundreds of millisecond range? Evidence from multiple-interval timing tasks. J Neurophysiol. 2008 Feb;99(2):939–49. doi: 10.1152/jn.01225.2007 18094101 Ng CW, Plakke B, Poremba A. Primate auditory recognition memory performance varies with sound type. Hear Res. 2009 Oct 1;256(1–2):64–74. doi: 10.1016/j.heares.2009.06.014 19567264 Scott BH, Mishkin M, Yin P. Monkeys have a limited form of short-term memory in audition. Proc Natl Acad Sci U S A. 2012 Jul 24;109(30):12237–41. doi: 10.1073/pnas.1209685109 22778411 Shushruth S, Zylberberg A, Shadlen MN. Sequential sampling from memory underlies action selection during abstract decision-making. Curr Biol. 2022 May 9;32(9):1949–1960.e5. doi: 10.1016/j.cub.2022.03.014 35354066 Bennur S, Gold JI. Distinct representations of a perceptual decision and the associated oculomotor plan in the monkey lateral intraparietal area. J Neurosci. 2011 Jan 19;31(3):913–21. doi: 10.1523/JNEUROSCI.4417-10.2011 21248116 10.1371/journal.pone.0317183.r001 Decision Letter 0 Vatakis Argiro Academic Editor 2025 Argiro Vatakis This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 0

19 Aug 2024

PONE-D-24-30219

Crossmodal association between auditory and visual objects in rhesus monkeys

PLOS ONE

Dear Dr. Lemus,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please read carefully the comments posed by the reviewers and that will also address any specific concerns from my side.

Please submit your revised manuscript by Oct 03 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Argiro Vatakis

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match. 

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

3. When completing the data availability statement of the submission form, you indicated that you will make your data available on acceptance. We strongly recommend all authors decide on a data sharing plan before acceptance, as the process can be lengthy and hold up publication timelines. Please note that, though access restrictions are acceptable now, your entire data will need to be made freely accessible if your manuscript is accepted for publication. This policy applies to all data except where public deposition would breach compliance with the protocol approved by your research ethics board. If you are unable to adhere to our open data policy, please kindly revise your statement to explain your reasoning and we will seek the editor's input on an exemption. Please be assured that, once you have provided your new statement, the assessment of your exemption will not hold up the peer review process.

4. We note that Figures 1-4 ans S3 includes an image of a participant in the study. 

As per the PLOS ONE policy (http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research) on papers that include identifying, or potentially identifying, information, the individual(s) or parent(s)/guardian(s) must be informed of the terms of the PLOS open-access (CC-BY) license and provide specific permission for publication of these details under the terms of this license. Please download the Consent Form for Publication in a PLOS Journal (http://journals.plos.org/plosone/s/file?id=8ce6/plos-consent-form-english.pdf). The signed consent form should not be submitted with the manuscript, but should be securely filed in the individual's case notes. Please amend the methods section and ethics statement of the manuscript to explicitly state that the patient/participant has provided consent for publication: “The individual in this manuscript has given written informed consent (as outlined in PLOS consent form) to publish these case details”. 

If you are unable to obtain consent from the subject of the photograph, you will need to remove the figure and any other textual identifying information or case descriptions for this individual.

5. Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript. If you need to cite a retracted article, indicate the article’s retracted status in the References list and also include a citation and full reference for the retraction notice.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The manuscript of Cabrera-Ruiz is focused on macaques‘ abilities to learn audio-visual crossmodal associations in a delayed match-to-sample task.

Although the text is generally very well written, in my opinion the main focus/emphasis of the paper is not stated clearly enough. As far as I understand, one of the main achievements of the study is the new procedure for training macaques to perform the audio-visual cross modal task. However instead of emphasizing that achievement and discussing it properly, the authors extensively comment on (arbitrary) proposed models of cross-modal associations, which cannot be tested without application of electrophysiological methods.

Major comments:

Introduction: More information (including existing models) about cross-modal association in macaques should be provided.

Line 42: The authors mention that the subjects are relatively old, which makes me think that they have experience in many other tasks. It would be good to comment whether and in what way the previous tasks, which the subjects were trained in, could affect their ability to learn the crossmodal associations.

Line 49: What would be the influence of the environmental enrichment to the performance of the audio-visual task?

Line 94-102 – Monkey Training: In my opinion this is one of the most important methodological part of the paper. More details should be provided regarding the training procedure.

Line 146 / Fig 2C, insets: It would be interesting not just to present the FA rates, but also to show which are the most common mistakes (e.g. are monkey faces confused more often with human faces in comparison to the red oval).

Lines 188-195: The authors identify 3 key findings, the first 2 of which directly follow the presented results. The third finding claims that “the monkeys’ success in forming crossmodal associations required […] working memory”, which is logical, however it is unclear how it could be derived from the results.

Lines 208-212: In my opinion this is the most important contribution of the study and I think that it should be discussed more extensively.

Lines 232-270: The two presented models are very interesting, however in the context of this behavioral experiment they are completely arbitrary. I am not sure whether they should be discussed so extensively without any electrophysiological data which can support either of them.

Minor comments:

Line 2: the sentence starting with “To find out” is ambiguous. To find out what?

Line 98: probably “cue to” instead of “clue for”.

Reviewer #2: Review for “Crossmodal association between auditory and visual objects in rhesus monkeys”

PONE-D-24-30219

This paper assessed whether rhesus macaques can learn crossmodal associations, how they are perceiving the visual sets versus the auditory sets in those associations, and what models of visual or auditory memory might explain these patterns. They successfully trained two macaques on a set of crossmodal associations (CMAs), which is no trivial feat. Performance on the task is well presented and clearly above chance. Upon further assessment, macaques appear to treat different individual sounds of the same type as perceptually equivalent, whereas patterns of reaction times for different images suggests this might not be the case for the visual image sets. This has interesting implications for how macaques may form these types of representations, and the authors put forth some possible models for how these representations might be stored in memory to explain these patterns.

I believe this experiment is interesting and relevant to ongoing work in the field. The data is compelling, and the experimental design effectively assesses their questions. There is room for improvement in how the paper is framed, as well as in the clarity of some of the explanations and data presentation, but I believe the work itself is sound, and will be useful groundwork for future studies in the field. I recommend this paper for publication, pending some revisions based on my suggestions below.

General Suggestions

1) Framing of memory systems in the intro. Your intro only talks about assessing whether macaques can learn CMAs. This makes sense since that’s a major point of your study, but this paper isn’t really about the CMA learning itself. I spent most of the paper thinking about the learning and acquisition of CMAs, since that’s how it’s framed here, but you don’t really make any points about acquisition at all. So much of your discussion is dedicated to how those CMAs are represented and maintained in the brain, and using macaques as a model system for assessing these different possible neural models. You need to lay the groundwork for that here. Are there existing competing hypotheses about how CMAs are represented in humans or macaques that you could discuss here? At the very least you need to mention that CMAs can be held in working memory, or that memory is relevant to this process at all. Even some basic background about audio versus visual working memory and how they might be relevant here would help readers keep that framing in mind and not be blindsided by the discussion.

2) How you’re referring to CMAs and sets of CMAs. It took me a while to figure out how exactly your different, specific CMAs were grouped (i.e. one single coo was trained to be associated with a few different monkey images, and each of those individual associations was one CMA). It would be useful to clarify this early, and possibly give name to the different CMA type groups (i.e. CMA sets or something like that), because knowing the associations were one coo to multiple images changes how a reader might interpret the results. I know you added more coos later on, but I spent the majority of the paper thinking each CMA was made up of completely different individual sounds and images. Even the S3 table didn’t clarify this for me, as I read the list as being grouped into categories (i.e. “coos”) with the individual pictures presented, and I interpreted the letters next to each picture as specifying the name of the specific sound used with that picture, rather than as the identifying name of the picture itself.

3) CMA acquisition. Do you have acquisition data for every time a new CMA is added? It sounds like once monkey M initially learned a CMA type (the coo, for example), they didn’t take much training to learn to associate a new image with that coo (I believe you added in new individual CMA images gradually, correct? If that’s not the case then I’m misinterpreting the methods and they should be clarified). If they are in fact generalizing that single “coo” learning to new images, that is pretty compelling evidence for a learned label (“coo” means “choose monkey”, for example) rather than many separate individually learned associations. That seems like really relevant information for how these CMAs are acquired and represented, and would definitely be worth adding. You could plot the number of presentations they took to learn each new CMA image, and if they’ve really learned a category label, I would expect those numbers to drop drastically for CMAs acquired later.

4) Other crossmodal association literature. There is a fair amount of work looking at crossmodal associations in humans and animals, though it tends to be framed more towards identity recognition, that would be worth looking into and including. Here’s a useful review:

Perrodin, C., Kayser, C., Abel, T. J., Logothetis, N. K. & Petkov, C. I. Who is That? Brain Networks and Mechanisms for Identifying Individuals. Trends in Cognitive Sciences 19, 783–796 (2015).

5) Arbitrary learned associations, or existing associations between animals and their calls? Because I was thinking so much about learning, I had a specific running question throughout the paper, which is whether the macaques are actually learning arbitrary associations, or coming in with existing, ecologically relevant associations of certain sounds with certain animals. I think this would be easy to fix with some additional support. If you think they’re arbitrary (and I think I agree with you), you just need to provide evidence. For example, if you have reason to believe the macaques don’t perceive faces on screen as though they’re actual macaques/humans/etc, or that you saw similar patterns even with your cartoon stimuli and the color red, or that they’ve never seen or heard cows before, or something along those lines.

Specific Suggestions

Abstract:

Line 1: I would be careful here. Your topic of interest is symbolic crossmodal associations, but even with this cool task, you can’t really make claims that your macaques have learned something symbolic, especially since it’s a unidirectional association and you never test the reverse (i.e. see image choose sound, which would be a bigger methods challenge). Your first line is fine, because it’s true that it is uncertain, but in the second line you need to make it clear that you’re testing for a step below symbolism, just any crossmodal associations at all, and not making any claims about whether they’re represented symbolically in macaques.

(see these references for more info on symbolic representations:

Palmer, F. R., & Palmer, F. R. (1981). Semantics. Cambridge University Press.

Deacon, T. W. (1997). The symbolic species: The co-evolution of language and the brain (1st ed). W.W. Norton.)

Line 6: You say performance remained constant, but I think that undercuts you a bit. They did well, it wasn’t just constant, it was consistently high.

Line 9: You say “semantic and conceptual thinking at the single-neuron level,” which doesn’t really say specifically what you mean and also is never mentioned anywhere else in the paper. I would shift this to refer to what you actually conclude they could be a model for, which is, from what I can tell, the neural pathways by which semantic information stored in memory, and later applied.

Intro:

Line 18: Can you add a reference from human literature to support this claim? It’s not controversial, but there is SO much research on how words are acquired in children, it seems odd to leave it out.

Line 28: See first general suggestion about framing.

Materials and Methods:

Line 77: Stims were presented in an equidistant circle, but did the circle rotate? Was configuration within the circle randomized? How was the actual position and configuration decided?

Line 84: Specify that this is just a cue to denote the start of the trial, I was confused for a moment thinking you were presenting the actual picture before the sound was played.

Line 88: This would be a good place to clarify the specifics of the CMA sets.

Line 100: Here for example, when you say they learned new CMAs, were they all the same type? As in they started out differentiating coos and human words, and the additional CMAs were new images corresponding to coos and words? Or were they totally new CMA types, as in the addition of moos?

Results:

Line 119: In S2 Fig B, there are 3 clear dips in performance that I imagine correspond to the addition of a new CMA type/additional foil stimulus, is that right? You should mark where those additions occur on the graph. Unless I’m misinterpreting and this is plotting performance once all 4 CMA types have been added, in which case that should be clarified.

Line 156: They seem to have learned a category set. Coo means touch monkey. So the thing that would slow them down is potentially just how long it takes to recognize that the individual monkeys are monkeys, leading to the reaction time differences (Fig. 2 A and C). When you gave them new images with the sounds, did they generalize to new monkey pictures? This is where the acquisition data would be really useful.

In Fig 2D, it seems the takeaway is that when average RTs are longer their variance is higher, is that correct? So there’s more variance in RTs when they’re taking longer to decide. I wasn’t sure what this graph added until I got to the conclusions, because it seems this pattern is the evidence you’re citing for the visual imagery model. I don’t fully see why this is evidence for that model, so I could use more clarity on that front overall. As it stands I’m not sure that I see the importance of this figure.

Line 172: I would clarify that perceptual invariance is not actually an inability to perceive a difference between the sounds. The critical thing is that they are treating these sounds as equivalent (the way we treat a word as the same no matter whose voice is saying it, or a musical note as the same even if it’s in a different octave), but that they would still likely be able to differentiate the individual sounds if need be. That is an important distinction.

Possible follow-up thought, I wonder where macaques draw these category lines. Would they view humans saying different words as equivalent, or would different words each have their own set? What about monkey coos vs screams, presumably those would fall into different categories, but could they learn to lump them together into categories of “monkey” and “human” instead?

Discussion:

Line 191: A point where you could defend that you think these associations are arbitrary.

Line 193: This memory conclusion is surprising, would be less jarring if more theoretical background on this is added to the intro.

Line 220: Do you report this data anywhere?

Line 224-230: This explanation helped me understand why you included figure 2, I would try to bring some of this clarity up above.

Line 232: Interesting hypotheses being put forth but again comes out of nowhere, add better framing for it in the intro.

Line 261: How so? What makes the patten you see a better match for the visual model? Don’t both models require some amount of processing and decision making once the visual stimuli are presented, resulting in differences in RT based on the specifics of the image? Arguably converting an image to a sound to match against your representation of a sound would in fact require more processing at image presentation than matching an image directly to your existing image representation, so wouldn’t the auditory model potentially result in longer and more varied processing once the choice images are presented? Maybe I’m missing something, but it would be useful for you to explain to the readers why you think this model is a better match to your data.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Ivo D. Popivanov

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

 10.1371/journal.pone.0317183.r002 Author response to Decision Letter 0 Submission Version 1

9 Nov 2024

Dr. Argiro Vatakis

Academic Editor, PLOS ONE

Dear Prof. Vatakis,

We are pleased to submit a revised version of our manuscript, "Crossmodal Association between Auditory and Visual Objects in Rhesus Monkeys" (PONE-D-24-30219), which addresses all the concerns and suggestions raised by the reviewers, as well as editorial requests. We have modified the manuscript, figures, and figure captions to meet the publication criteria of PLOS ONE.

We believe the updated manuscript, enhanced by the reviewers' input, effectively demonstrates its contribution to the understanding of learning mechanisms in cross-modal associations and their implications in sensory processing and working memory. The revisions feature an introduction reframed to emphasize working memory and cross-modal associations in macaques, together with comprehensive methodological details on training the macaques and analyses of their performance during various learning stages. Additionally, we updated the title to "Monkeys Can Identify Pictures from Words," capturing a crucial insight from the reviewers regarding the learning of cross-modal associations between non-ethological stimuli. To address concerns about the monkeys' learning processes, we included the former S2 Fig as a new Fig 2. We have also incorporated supplementary information and new panels within the original figures to better address the reviewers’ comments and clarify the findings. We also withdrew the previous Fig 4 from the manuscript since it depicted possible models for solving cross-modal associations in the brain, which were not tested in our experiments. Nevertheless, all the results presented in the initial draft are maintained, enriched only by deeper interpretations suggested by the reviewers. We confirm the completeness and correctness of the reference list, with a few references added to the introduction to align with suggestions on the study framing.

We also confirm that all the figures in the manuscript (Figs 1-4, S1, S2 and S3) were created entirely in our lab. Some of them appear as a preprint in bioRxiv.org (https://doi.org/10.1101/2024.01.05.574397), but none of them have been previously copyrighted. The original images used in the experiments were downloaded from free online sites and the original version of the manuscript depicted them. However, we replaced all those depictions from all figures to avoid any possible copyright conflicts, and substituted them with pictures similar but not identical, created using an AI image generator (https://www.fotor.com/ai-art-generator). Fotor.com explicitly states that: "For the AI image generator, the AI-generated images can be used for both personal and commercial purposes. You can share them on social media platforms, use them for marketing campaigns, or sell them. You are the copyright owner of your creations and will be responsible for any output that you generate using AIGC Related Service." (https://support.fotor.com/hc/en-us/articles/17767970123417-Are-the-AI-generated-images-commercially-available-Do-I-have-ownership-of-them). Moreover, as requested by PLOS ONE, we included the following sentence in figure captions: “The pictures are similar but not identical to the original images used in the study and are therefore for illustrative purposes only.” We also included the following description at the Methods section (lines 109-113): “Animal pictures used in the experiment were downloaded from free online sites and customized. However, the pictures shown in figures and supplementary information are similar but not identical to the original images used in the study; they were created for illustrative purposes only using an online AI image generator (https://www.fotor.com/ai-art-generator).” However, we included the picture of one of the researchers in the study. Therefore, we submitted the signed consent format as provided by PLOS ONE, and report at the Ethics statement (lines 65-66) the following: “The portrayal of one of the authors of this manuscript was used in the experiments and has given written informed consent to publish this case details.”

Regarding the sounds used in our experiments, we confirm that the cow vocalizations were downloaded from Freesound.org, and under the following License: Creative Commons 0 (CC0; No copyright). However, all the human and monkey sounds were recorded in our lab and have not been published elsewhere. We clarify this at lines 99-101 of the methods as follows: “The experiment utilized a variety of sounds, including laboratory recordings of words and monkey vocalizations, as well as free online sounds of cow vocalizations (https://freesound.org/).”

Finally, we were notified that the Funding Information, Financial Disclosure and Data Availability statements need to be clarified. Therefore, we kindly ask to include the following statements:

Funding Information

LL received grant support from the Consejo Nacional de Humanidades Ciencias y Tecnologías (CONAHCYT; Number: 256767; https://conahcyt.mx/), and the Programa de Apoyo a Proyectos de Investigación e Innovación Tecnológica (PAPIIT; Number: IN229323; https://dgapa.unam.mx/index.php/impulso-a-la-investigacion/papiit). JV received support by Secretaría de Educación, Ciencia, Tecnología e Innovación de la Ciudad de México (SECTEI/103/2022; https://www.sectei.cdmx.gob.mx/). Elizabeth Cabrera Ruiz conducted this study to fulfill the requirements of Programa de Doctorado en Ciencias Biomédicas of Universidad Nacional Autónoma de México and received a doctoral scholarship from Consejo Nacional de Humanidades Ciencias y Tecnologías (Scholarship number: 245771; https://conahcyt.mx/). The data in this work are part of her doctoral dissertation.

Financial Disclosure

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Data Availability Statement

The data is fully available without restrictions from Figshare.com https://figshare.com/s/998043a02c8b15315632

Competing Interests

No authors have competing interests

Author contributions

Conceptualization: LL. Data Curation: ECR, MMH, TF. Formal Analysis: ECR, MA, MT, MMH, JV & LL. Funding Acquisition: LL, ECR, JV. Investigation: ECR, MMH. Methodology: ECR & LL. Software: ECR, MA, TF, MT, & LL. Validation: JPO, MT, JV & LL. Visualization: ECR, MA & LL. Writing original draft: LL. Review & draft editing: ECR, MA, JPO, MT & JV.

Responses to the reviewers (highlighted in blue)

Reviewer #1:

The manuscript of Cabrera-Ruiz is focused on macaques‘ abilities to learn audio-visual crossmodal associations in a delayed match-to-sample task.

Although the text is generally very well written, in my opinion the main focus/emphasis of the paper is not stated clearly enough. As far as I understand, one of the main achievements of the study is the new procedure for training macaques to perform the audio-visual cross modal task. However instead of emphasizing that achievement and discussing it properly, the authors extensively comment on (arbitrary) proposed models of cross-modal associations, which cannot be tested without application of electrophysiological methods.

The revised manuscript now emphasizes the achievement of training macaques in cross-modal discrimination. For example, line 8 of the Abstract now reads: "We found that the monkeys learned and performed proficiently in over a dozen associations." We also included full descriptions of the training protocol as a "Monkeys training" subsection at the Methods (lines 146-177), and a "Learning measurements" subsection about the analytical methods for evaluating the learning process (lines 178-203). In addition, we included a new section "Rhesus Monkeys Can Learn Cross-Modal Associations Between Stimuli of Different Types" at the Results (lines 251-294), and included S2 Fig in the text as Fig 2 (lines 277-282) with better descriptions of the learning of CMAs across training sessions, and a new panel at Fig 2B (also suggested by Reviewer 2), showing the number of sessions to performing each CMA above chance. Finally, we created a new S1 Fig showing the performance for each CMA across sessions.

We also agree with the reviewers about the problem of presenting models about the neuronal mechanisms of CMAs without evidence. Therefore, we decided to remove Figure Four from the document, limiting ourselves exclusively to adding information about cross-modal association models in the introduction. We now refer to the acoustic template model as a mechanism for association between modalities and to the alternative model of learning through social interactions. These models are important for the association between modalities as a basis for associations between stimuli that are not necessarily ethological but that are used in humans for language (which was what we tried to test in our study). The first paragraph of the introduction now reads as follows:

“Humans form cross-modal associations (CMAs) between sounds and images, which play a vital role in integrating semantic representations within language [1]. Supporting this, fMRI studies have shown that the temporal lobe of the human brain is actively involved in CMAs [2,3] between words and visual objects [4]. It is believed that CMAs between phonological "templates"—developed in human infants by listening to caretakers—and observed objects are essential for creating semantic representations and aiding the production of a child's first words [5–8]. Similarly, auditory templates have been proposed as a mechanism for vocal production in birds [9–13] and marmoset monkeys [14]. Recent studies, such as those by Carouso-Peck and Goldstein [15,16], have also shown that visual signals during social interactions can also influence vocal production in birds. However, only a few ethological studies have suggested the existence of CMAs between vocal sounds and visual cues for semantic communication [17]. For instance, research has observed that vervet monkeys respond to calls signaling the presence of predators by looking upwards, downwards, or climbing into trees [18].”

Major comments:

Introduction: More information (including existing models) about cross-modal association in macaques should be provided.

We included important references to cross-modal association processes in monkeys. Importantly, we also framed the introduction within the scope of working memory since it plays a key role in multisensory association processes (as noted by reviewer 2). The second paragraph of the introduction now reads as follows:

“Neurophysiological recordings in monkeys have shown that the prefrontal cortex (PFC) — a brain area homologous to that in humans — utilizes working memory (WM) circuits [19] to perform CMAs between voices and faces [20–32], receiving inputs from various sensory regions [33–36]. CMAs have also been observed in the auditory and visual areas of the temporal lobe [37–45]. Notably, trained macaques have demonstrated the ability to perform cross-modal discriminations between visual and tactile objects [46,47], and between stimuli that could be considered non-ethologically relevant (NER), such as between pitch and color [48] and between amodal information (i.e., information that does not belong to a particular modality) [49] such as numerosity [50] and flutter frequencies [51–54]. However, it remains to be explored whether non-human primates can establish CMAs between NER stimuli that are important for human language, like words —which monkeys can discriminate phonetically [55,56], and pictures.”

Line 42: The authors mention that the subjects are relatively old, which makes me think that they have experience in many other tasks. It would be good to comment whether and in what way the previous tasks, which the subjects were trained in, could affect their ability to learn the crossmodal associations.

The monkeys had not participated in any experiments previously nor had they been trained in any task. We now clarify this in the Methods on line 69, which now reads as follows:

“The animals had no previous training in any other task and were not subjected to any surgery or head restraint for this behavioral study.”

Line 49: What would be the influence of the environmental enrichment to the performance of the audio-visual task?

Environmental enrichment was implemented as part of the animals' housing, e.g., plastic boxes with snacks inside to promote manual dexterity. Additionally, the monkeys spent time in a shared space with room to climb and socialize with other monkeys, which apart from favoring the display of social behaviors characteristic of their species, probably played a role in the 'monkey' CMA. Similarly, the animals were partially exposed to the faces of the researchers (i.e., they wore masks), and only the picture of the authors was used as a stimulus. Therefore, we believe that it is unlikely it helped the monkeys significantly in creating the associations for the task. The lines 77-82 address enrichment as follows:

“The monkeys also had access to an enriched environment with toys, a recreation area for climbing and socializing with other monkeys four days a week, and opportunities for grooming through mesh sliding doors. In addition, cartoons and wildlife videos of content unrelated to the experiments, were presented on TV for no more than four hours a day. However, the face and voice of one of the researchers with whom the monkeys interacted were used during the experiments”. In this regard, line 393 of the discussion reads: “We found no clear evidence that learning CMAs that included possible ethologically relevant stimuli like human and monkey faces, or coos [20–31] were facilitated more than other CMAs to which they had no previous exposure.”

Line 94-102 – Monkey Training: In my opinion this is one of the most important methodological part of the paper. More details should be provided regarding the training procedure.

We expanded the Methods section ("Monkeys Training," lines 146-177) to include a detailed description of the training protocol in the Methods section ("Learning Measurements," lines 178-203), where we describe analytical methods of the monkeys’ learning process. We incorporated S2 Fig as Fig 2 so the monkeys' learning of CMAs can now be observed—a critical finding in our study. We also included S1 Fig to show the performance at all CMAs throughout learning and added sections to the Results and the Discussion specifically addressing the learning of CMAs. Specifically, sections "Rhesus Monkeys Can Learn Cross-modal Associations Between Stimuli of Different Types" (lines 251-294), and "Rhesus Macaques Create Crossmodal Associations Between Sounds and Images of Different Types" (lines 377-417), respectively.

Line 146 / Fig 2C, insets: It would be interesting not just to present the FA rates, but also to show which are the most common mistakes (e.g. are monkey faces confused more often with human faces in comparison to the red oval).

Addressing this question, we included pie charts in Figure 3 (formerly Figure 2), showing the pictures selected during false alarms. We also included better descriptions of the results (lines 316-331) and a new S3 Table with the percentage of choices during hits and false alarms.

Lines 188-195: The authors identify 3 key findings, the first 2 of which directly follow the presented results. The third finding claims that “the monkeys’ success in forming crossmodal associations required […] working memory”, which is logical, however it is unclear how it could be derived from the results.

Regarding working memory, our task required it due to the 3s delay between auditory stimuli and picture options. We now provide better descriptions in the abstract, introduction, methods, results, and discussion. The abstract (lines 6-8) now reads: “In each trial, the monkeys listened to a brief sound (e.g., a monkey vocalization or a human word), and retained information about the sound to match it with one of 2–4 pictures presented on a touchscreen after a 3-second delay.” The Introduction (lines 45-50) now reads: “We specifically designed the task to temporally separate auditory and visual stimuli, thus engaging WM circuits to retain one modality in mind while awaiting the corresponding cross-modal stimulus.

Submitted filename: Response to Reviewers.docx

 10.1371/journal.pone.0317183.r003 Decision Letter 1 Vatakis Argiro Academic Editor 2025 Argiro Vatakis This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Submission Version 1

23 Dec 2024

Monkeys can identify pictures from words

PONE-D-24-30219R1

Dear Dr. Lemus,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Argiro Vatakis

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: In my view the manuscript is substantially improved with respect to the previous version. I don't have any additional comments and I would be glad to see that this well performed study will be published in this journal! Good luck with your further research on this topic!

Reviewer #2: All comments have been addressed, I believe this manuscript should be accepted for publication to PLOS One.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Ivo D. Popivanov

Reviewer #2: No

**********

 10.1371/journal.pone.0317183.r004 Acceptance letter Vatakis Argiro Academic Editor 2025 Argiro Vatakis This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

10 Jan 2025

PONE-D-24-30219R1

PLOS ONE

Dear Dr. Lemus,

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team.

At this stage, our production department will prepare your paper for publication. This includes ensuring the following:

* All references, tables, and figures are properly cited

* All relevant supporting information is included in the manuscript submission,

* There are no issues that prevent the paper from being properly typeset

If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps.

Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

If we can help with anything else, please email us at customercare@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Argiro Vatakis

Academic Editor

PLOS ONE