The present work reports on a set of perceptual psychological experiments aimed to portray the underlying meta-structure of affective communication, exploiting a crossmodal database of verbal and non-verbal (gaze, facial expressions, and gestures) American English video clips, for a comparative analysis of the subjective perception of emotional states dynamically presented through the visual and auditory channels, considered either singularly or in combination. The participants to the experiments are 90 Italian and 90 American English subjects. The results reveal that the amount of emotional information conveyed by the audio and visual components separately is much the same as that conveyed through the combined channels, suggesting a nonlinear processing of the emotional information and a cognitive load effect when it arrives from both the channels. In addition, and differently from previous results observed in experiments involving Italian stimuli, both the Italian and the American subjects seem to exploit more the visual than the auditory information, debating the previously suggested “language expertise hypothesis” (see [12, 13]) that favours the identification of emotional expressions in the audio when subjects are asked to label emotional stimuli in their own language because of their expertise in language-specific prosodic and paralinguistic information. It is possible that at the base of the encoding of the emotional information there is a more language specific process that is strictly related to the language and the way suprasegmental information are encoded and expressed in it.