Introduction

Demonstrative is an indispensable component in linguistic grammar, which is used to “attribute deictic features to noun phrases” (Arista and Moreno, 2004, p. 70). As a member of the demonstratives in English grammar, that has been studied by virtue of the spatial proximity in Pragmatics, Semantics, Cognitive Linguistics, and Systemic Functional Linguistics. It was proved that that was distinguishable from this in light of the orientation of the physical distance or localism (Lyons, 1977), the speaker’s location (Levinson, 1983; Saeed, 2016), the egocentric orientation (Gibbons, 2012; Bohnemeyer, 2021), and the spatio-temporal condition (Halliday and Hasan, 1976; Diessel, 2013; Martin and Rose, 2003/2007). As a discursive resource, demonstrative that can be used to identify an element co-textually by retrieving the preceding or following content, or in context of the situation, which provides the information outside the text to interpret it.

Before multimodal discourse analysis was developed in the field of Linguistics, the linguistic demonstrative that had been discussed together with visual pointing (Bühler, 1934; Kaplan, 1989) and the gestural use of the deictic expressions (Fillmore, 1997; Huang, 2014) based on the co-contextualizing relation. However, the visual deictic had not been taken as an independent but equal system with the linguistic deixis. After O’Halloran (1996) put forward Multimodality, the visual image has been discerned as an independent semiotic resource, which has the equal stance of making meaning with language (Page, 2010). Many scholars have systematically discussed the deixis or deictic in visual images, for instance, the deictic gesture (Bateman, Wildfeuer, and Hiippala, 2017), the pictorial elements, and the character’s gaze (Painter, Martin and Unsworth, 2012), pointing (Kita, 2003), the arrows (Acartürk, Coskun and Emil, 2021) and the reappearance (Tseng, 2013). All these researches have been done under the heading of visual deictics on the ground that they perform a similar deictic function to linguistic deictics. The studies on visual deictics mainly focus on the one-to-one interpretation of a concrete entity, and very few of them are on the abstract or even compressed referents.

The study aims to address three issues: (1) how the visual deictic eliminates the deictic indeterminacy of that in implying the proximity in the modern TV series; (2) how explicit and implicit that sheds light on the deictic divergency of the visual proximity from the linguistic one when identifying an element; (3) how a visual deictic forces demonstrative that to transform into this in the modern TV series. The research questions are expected to unravel the relation between the semiotic world and reality through the visual and verbal deictic relation. To instantiate the intersemiotic interpretation of demonstrative that, the data are drawn from the modern TV series and analyzed to show the brain processing, the perception, and the memory along the verbal and visual dimensions by means of ELAN 6.3 which is utilized to annotate the audiovisual recordings as “a free, multimodal annotation tool” (Lausberg and Sloetjes, 2009, p. 841).

Literature review

In linguistic discourses, demonstrative that has the potential to demonstrate a distal entity in the spatial dimension by virtue of the predominant symbolic nature of language (Stöckl, 2004, p. 26), which constitutes a contrastive pair with this. These two terms that and this were argued to be exclusive from each other according to their spatial proximity. They are used to “find an identity, ‘near’ with this or ‘far’ with that” (Martin and Rose, 2007, p. 162) in linguistic discourses. As a neutral or unmarked item, demonstrative that performs two functions: demonstrating the identity of an element and forming a contrastive pair with this on the spatial scale in linguistic discourses. The use of that is commonly speaker-oriented, and the referent is the one that is not near the speaker. Demonstrative that is typically interpreted by the retrieval of the preceding discursive item, which, in a sense, breaks the forward movement of the communication. The backward narrative retrieval diverges from “the sequential unfolding of events in real-time” (Thibault, 2004, p. 109), which creates a specific discursive effect.

In the visual semiotic system, the arrow is a prototypical visual deictic which directly points towards a visual referent. The orientation of the visual deictic arrow is flexible since the visual elements can be non-sequential or “quite random” (Norris, 2004, p. 9) in the spatio-temporal dimension of a visual image. The non-sequential arrow embodies the dynamic nature of the visual deictic orientation with the starting point towards the endpoint where the visual referent is located. The deictic function of the arrow responds to the predominant iconic nature of a visual image (Stöckl, 2004, p. 26).

In multimodal discourses, demonstrative that is often accompanied by a visual deictic to track the identity of the presumed element. Owing to its dominant iconic nature, the visual deictic plays the vital role of identifying the referent when no verbal expression identifies the element that that demonstrates, or of eliminating the ambiguity of the referent when more than one element has the potential of becoming the referent. In a word, a visual deictic can synergize with a verbal deictic that to identify an element in multimodal discourses, which pushes the communication forward on the timescale in reality. The visual deictic function can be realized by the reappearance of a visual element, either explicit reappearance or implicit one (Tseng, 2008, 2013), the arrow, and even the visual actions such as pointing and the movement of the eyeballs, to track the identity of an entity or a participant in the visual images. Take the reappearance of a visual element as an example. It functions as a visual deictic to facilitate the audience to follow the forward movement of the communication or the narration without looking backward to track the referent that that demonstrated (see Fig. 1).

Fig. 1
figure 1

Opposite directions of demonstrative that and visual reappearance (deictic).

Full size image

Methodology

To illustrate the intersemiotic interpretation of demonstrative that, the study adopts the qualitative method to instantiate how the visual deictic eliminates the deictic indeterminacy of that when identifying an element in multimodal discourses.

The data are drawn from modern TV series, including the Animated Series Milo Murphy’s Law and the TV series The Big Bang Theory and Gray’s Anatomy in which the deictic function is performed by the visual and linguistic elements. The relevant data are annotated by the annotation tool ELAN 6.3. ELAN (EUDICO Linguistic Annotator) was developed to annotate the video and audio materials with an unlimited number of textual annotations. The words, the gestures, and the other elements can be annotated by creating different layers or tiers. ELAN 6.3 features the modification of the start and/or end time of all the annotations on the selected tiers, which meets the demand of annotating different elements related to the intersemiotic interpretation of demonstrative that in the modern TV series.

The analyses will be implemented in the following procedures. Firstly, it is illustrated that demonstrative that may not be adequately interpreted in the verbal expressions, but requires a visual deictic to correct or modify the referent that that demonstrates. The visual deictic can indicate the cause-effect relation between/among different activities based on the deictic function. Secondly, it is demonstrated that demonstrative that can be classified into explicit that and implicit that according to the proximal-distal relation. This classification can be used to explicate the deictic divergency of the visual proximity from the linguistic one in multimodal discourse analysis. Finally, it is exemplified that a visual deictic may break off the spatial and temporal limitations to force demonstrative that to transform into this by the reappearance in the modern TV series. All illustrate that it is justified to interpret demonstrative that by reference to other semiotic deictic resources in multimodal discourse analysis. The results are expected to provide an academic lens to unfold the multi-dimensional but dynamic world we are living in through the intersemiotic interpretation of that.

Visual complementarity to the interpretation of demonstrative that

As argued, a visual deictic may eliminate the deictic indeterminacy of that by correcting or modifying the referent that that demonstrates on the ground of its predominant iconic nature. Take (1) for example. It is a dialogue between a young employee and an elder one who works in the theater in the Animated Series Milo Murphy’s Law (Season 1, Episode 4). The elder employee tries to explain to the young employee under what conditions those lights above the stage of the theater may become loose.

  1. (1)

    Young: Ah, those lights up there, they seem a little loose to me.

Elder: Ah, they’ll be all right. Unless someone accidentally leans on that backdrop too hard, hitting that fireman’s ax, causing it to fall on the floor, startling that rat who runs into that lamp, causing it to fall over, knocking into those tie-offs, causing that rope to come loose.

Young: Wait a minute. Whoa, whoa! How could that lamp hit those ropes?

Elder: No, no, no, not that lamp. That lamp and those ropes.

Young: Oh, I see.

In the verbal dialog, a demonstrative adjective that is used to demonstrate backdrop, fireman’s ax, rat, lamp, rope, lamp, lamp, and lamp respectively so as to specify the cause-effect relation of the possible accident. As the perspective shifts back and forth between the elder and young employees, it is difficult to identify the referents that that demonstrated just by reference to these verbal expressions. Since the perspective is speaker-oriented, the elder employee is definitely clear about which backdrop, fireman’s ax, rat, lamp, and rope that that demonstrates when he explains under what conditions those lights above the stage of the theater may become loose. He takes for granted that the young employee who stands aside shares his perspective as the “linguistically mediated social meaning-making is always to some extent explicitly meaningful for the participants in the exchange” (Thibault, 2004, p. 181). When the young employee raises a question, the perspective shifts to him. He takes for granted that the elder worker uses that to refer to the lamp far away from them. However, it is proved to be logically wrong since it fails to explain the cause-effect relation between that lamp and those ropes in the hypothetical accident. Then the perspective shifts back to the elder employee when he corrects and reconfirms the referents that that demonstrates. As for the audience on the opposite side of the screen, it is impossible for them to interpret the referents that that demonstrated by reference to the verbal expressions. It is difficult for them to figure out the cause-effect of the possible accident since they do not share either of the elder and young employees’ perspectives. In other words, these two speakers do not take the audience into account in the verbal dialog (see Fig. 2).

Fig. 2
figure 2

Shifts of perspective in the interpretations of that.

Full size image

The frequent shifts of the perspective increase the difficulty of identifying the referents that that demonstrates especially when no other linguistic expressions are uttered to identify the entities. The audience cannot decode cause-effect of the possible accident just by means of the linguistic deictic that, which makes the verbal expressions meaningless. As a specific genre, the animated series makes a profit from the audience. If the audience abandons them owing to the deictic ambiguity, the animated series inevitably loses its commercial and even artistic values.

To avoid such an awkward situation, the visual deictics (the red arrows, the yellow curve line on the upper-left corner of the screen, and the yellow crosses) are used to eliminate the deictic ambiguity of that so as to elaborate cause-effect of the hypothetical event. Both the visual deictics and demonstrative that have been annotated by ELAN 6.3 (see Fig. 3). For example, as the elder employee utters No, no, no, not that lamp, a yellow curve line appears on the upper-left corner of the screen to rule out the lamp marked by the red arrow that the young employee mentions. The yellow curve line implies the denial of the previous cause-effect relation inferred from the dynamic images as well. Though the referents that that demonstrated are not adequately identified in the linguistic expressions, the visual deictics fill the deictic vacuum by visually signaling the referents. The synergy between the visual deictics and that pushes the conversation forward on the timescale in the TV world.

Fig. 3
figure 3

Visual and verbal deictics in Milo Murphy’s Law (Season 1, Episode 4) annotated by ELAN 6.3.

Full size image

Besides, these visual deictics do not exist in the real world but are created by modern TV series producers who invite the audience to share the elder and young employees’ perspectives (see Fig. 4). In a sense, the shifts of the perspective between the elder and young employees can be visualized and perceived by the audience by means of the visual deictics so as to eliminate the deictic indeterminacy of demonstrative that in the verbal expressions of the animated series.

Fig. 4
figure 4

Shifts of perspective visualized by means of visual deictics.

Full size image

As far as the dynamic is concerned, the visual deictics in synergy with demonstrative that push the communication forward on the timescale in the TV world. Firstly, in the verbal dialog, the entities that that demonstrates change as the communication continues, just as “in talk, we mobilize language as sounded speech” (Iedema, 2003, p. 39). Secondly, each demonstrative that is accompanied by a dynamic visual deictic which bears the inherent dynamic orientation to realize “some type of convergence” (Lim-Fei, 2004a, p. 239). The visual deictics eliminate the deictic ambiguity of demonstrative that, and push the conversation forward from the audience’s perspective without retrieving the previous content or inferring the information in context of situation outside the TV world. Finally, these visual deictics visualize the speakers’ brain processing, which can only be made in the animated series, the films, the TV series, and the things alike. In the daily communication, the arrows may be replaced by some deictic gestures, for example, the pointing finger for “locating entities or actions in space vis-à-vis a reference point” (McNeill, 2006, p. 60). Such dynamic intersemiotic interpretations of demonstrative that are not available to written discourse but only to multimodal discourse. The above analyses illustrate the complementarity of the linguistic and visual semiotics in the dynamic co-contextualizing meaning-making process of multimodal discourses (Martin and Rose, 2003/2007).

Explicit and implicit demonstrative that

Demonstrative that has the potential of demonstrating a distal entity in the spatial dimension, and forms a contrastive pair with this. These two terms are argued to be exclusive from each other in light of their spatial proximity. They are used to “find an identity, ‘near’ with this or ‘far’ with that” (Martin and Rose, 2007, p. 162) in linguistic discourses. However, proximity is more complicated in multimodal discourses since more than one semiotic resource makes the deictic meanings. It happens that the proximity inferred from the visual deictic is contradictory to that that implies in the semiotic world.

Take Sheldon’s monologue in The Big Bang Theory (Season 10, Episode 9) for example. The background of the story is that Bert won the genius grants supported by MacArthur Fellows Program. Sheldon is jealous of him and cannot accept it. He is persuaded to read Bert’s paper to find out why Bert won the prize. When he reads, he murmurs as follows:

  1. (2)

    Sheldon: Ye…why?!

(He continues to read on the same page)

Sheldon: Eh, …why?!

(He turns to the next page)

Sheldon: Oh, that’s …why.

When Sheldon finds the answer to his own question why in Bert’s paper, he uses that to demonstrate its identity in the monologue. On the one hand, regarding the linguistic temporal dimension, the present-time predicator is used in that’s …why. However, it was argued that “that tends to be associated with a past-time referent” (Halliday and Hasan, 1976, p. 60). Thus, that is not the appropriate demonstrative pronoun to indicate the linguistic temporal proximity. On the other hand, according to the close physical distance to the answer on the page held in his hand in the visual image, Sheldon is supposed to use this instead of that to demonstrate the proximal answer to his question why. The visual-spatial distance is contradictory to the linguistic one implied by the verbal deictic that in this case.

However, it is common to use that in such a way in the daily communication. The reasonable explanation is that demonstrative that is not inherent to demonstrate a distal entity on the ground that it is the unmarked member in English deixis (Lyons 1977; Halliday and Matthiessen 2004/2014). Its unmarked or neutral nature determines the deictic indeterminacy of demonstrating an element in terms of the distal-proximal proximities in the communication. I propose that demonstrative that should be classified into explicit demonstrative that and implicit demonstrative that according to the potential of forming a contrastive pair with this: the former performs the function of demonstrating a distal element that constitutes the contrastive proximity with a proximal one demonstrated by this in the spatial or/and temporal dimension; the latter plays the role of identifying any entity but does not form a contrastive pair with this in time and/or space. The use of implicit demonstrative that does not elicit the distal meaning from the speaker’s or the listener’s perspective in the communication, or make any distal impression on the audience who do not participate in the communication but observe it. Implicit that implies a neutral distance between the speaker, the listener, and the audience, which can be used to explicate the divergence between the visual and linguistic proximities in the spatial and/or temporal dimensions.

In (2), Sheldon uses that to demonstrate the answer to his own questions why when he reads Bert’s paper held in his hand. In this example, that does not perform the function of demonstrating a distal entity in contrast with any proximal one in either the spatial or temporal dimension from the speaker Sheldon’s perspective or of expressing any specific feeling towards the content that he reads. It just functions as a neutral demonstrative pronoun, which does not evoke any proximal-distal sense among the speaker, the listeners in the scene, and even the audience on the opposite of the screen. As the demonstrative pronoun that in (2) is ascribed to be implicit that, it can explicate the divergency of the visual proximity from the linguistic one in the scenario.

Besides, implicit that in (2) embodies the dynamics of the text and of the external world. On the one hand, implicit that is a part of the dynamic linguistic monologue on the ground that “meanings flow, as texts unfold” (Martin and Rose, 2003, p. 188, 2007, p. 201). On the other hand, it synergizes with the movement of the eyeballs to indicate that Sheldon is reading Bert’s paper in the dynamic external world. The relevant visual and verbal elements have been annotated by ELAN 6.3 (see Fig. 5). The visual deictic function is realized by Sheldon’s gaze at Bert’s paper when he finds the answer to his why in (2). Different from the typical visual deictic arrows in (1), the gaze serves as “one kind of deictic vector, an additional option helping to link elements within a focus group” (Painter, Martin, and Unsworth, 2012, p. 116) in (2). When Sheldon reads the paper, his eyeballs move. When he finds the answer, his eyes fixed on some point on the page, which narrows down the scope of the referent (Sidnell and Enfield, 2017). The linguistic demonstrative pronoun that synergizes with his gaze to indicate that he finds the answer to his own question why. The synergy of the linguistic and visual deictics illustrates the contextualization of the linguistic and visual semiotics “on the timescale of particular discursive activities” (Thibault, 2004, p. 192).

Fig. 5
figure 5

Visual and verbal deictics in The Big Bang Theory (Season 10, Episode 9) annotated by ELAN 6.3.

Full size image

As far as explicit that is concerned, it can form a contrastive pair with the linguistic demonstrative this according to the spatio-temporal proximity when that is used to demonstrate a distal element in linguistic discourses. In multimodal discourse, demonstrative that can form a contrastive pair with a visual proximal deictic when that is used to demonstrate the distal entity, even though there is no this identifying the proximal entity. Take No, no, no, not that lamp. That lamp and those ropes in (1) for example. The first verbal that should be taken as explicit that to demonstrate the distal lamp since it forms a contrastive pair with the red arrow signaling the proximal lamp on the top right side of the visual image though there is no linguistic demonstrative adjective this to demonstrate the proximal lamp in the verbal expressions (see Fig. 3). The young employee notices it and takes for granted that the elder employee uses that to demonstrate the distal lamp. This is why the young employee mistakes the lamp and raises the question How could that lamp hit those ropes. The linguistic and visual semiotics complement each other to “form a complex whole” (Baldry and Thibault, 2006, p. 18) which cannot be realized by the mono-semiotic expressions.

It is significant to classify demonstrative that into explicit that and implicit that since they can solve the divergency between the visual and linguistic proximities in the spatial and/or temporal dimensions in multimodal discourse. The implicit demonstrative that is used to demonstrate a neutral and unmarked entity without any proximal entity, either visual or linguistic, to form a contrastive pair according to the spatio-temporal proximity. The use of implicit that does not cause any deictic ambiguity for the speaker, the addressee and even the audience who do not participate in the communication but observe it. Implicit that may be recontextualized as the visual deictic to reinforce the deictic function when no proximal deictic vector is depicted in the visual texture. On the contrary, the explicit demonstrative that can form a distal-proximal pair with a visual proximal deictic vector when no linguistic deictic this involved. The intersemiotic deictics reflect the complementarity between the linguistic and visual semiotics in encoding “a semiotic product or event” (Kress and van Leeuwen, 2001, p. 20). An atypical choice between explicit that and implicit that can achieve artistic effects, such as hilarity, horror, and mystery, in the TV series, the films, the animated series, commercials, etc.

Transformation of that into this

As argued, the demonstratives that and this seem to be exclusive from each other according to the proximity in the linguistic deixis system: “near and remote from the point of view of the speaker” (Halliday and Matthiessen, 2004, p. 558, 2014, p. 631). This viewpoint implies:

(1) there is no overlap between the demonstratives that and this from the speaker’s perspective;

(2) the speaker takes his or her own position as a reference point of proximity;

(3) that is used to demonstrate a distal element or content which is far from the speaker, whereas this a proximal one that is near the speaker.

However, the implications are faced with new challenges: (1) there is some overlap between the demonstratives that and this in light of the classification of explicit that and implicit that; (2) demonstrative this may be transferred to that when the proximal-distal relation between two elements is ignored by the speaker; (3) demonstrative that may be transformed into this due to the shift of the perspective or the involvement of other semiotic deictic vectors.

Firstly, the implicit demonstrative that as a neutral verbal deictic does not form the contrastive pair with this to demonstrate the distal and proximal elements, but performs the function of identifying the presumed content or entity. It implies that implicit that can be used to demonstrate any discursive element, either remote or near. For example, implicit that is used to demonstrate a proximal answer which is about one foot away from the speaker’s eyes in (2). Implicit that gives the speaker an additional choice of demonstrating a proximal element, and avoids the deictic ambiguity when identifying an entity. When it is recontextualized in the visual image, it is justified to design only one visual deictic instead of two to visually demonstrate an entity in multimodal discourse such as the films and the TV series.

Secondly, demonstrative this can be transferred to that if the speaker ignores the proximal-distal difference between two elements in the contextualizing relation. Take (1) for example. There are two lamps above the stage of the theater. One is near the elder and young employees and the other is far from them. Ideally, they should be identified by this and that according to the physical proximity. However, the elder employee does not take the other lamp farther away from himself above the stage of the theater into account. Therefore, when he mentions the lamp that may make the lights loose, he uses that to demonstrate it instead of this.

Thirdly, demonstrative that may be transformed into this due to the shift of perspective. It happens that when the perspective shifts, the proximal-distal difference changes. To be specific, for A, an entity is remote in the spatial or temporal dimension in a visual image and/or a linguistic expression. Thus, demonstrative that is supposed to be used to identify it from A’s perspective. But when it is B’s turn to talk about this entity, this may be appropriate to identify it since the entity is near B (see Fig. 6). In the TV series or the films, the audience changes their perspectives as the communicators do, and therefore “shift their deictic orientation” (Gibbons, 2012, p. 30). The shift of the perspective changes the proximity in the spatio-temporal dimension, which causes the transformation of that into this.

Fig. 6
figure 6

Proximal-distal changes of an element from A to B due to the shift of the perspective.

Full size image

Finally, demonstrative that may be transferred to this because of the involvement of other semiotic deictic vectors which evoke the distal-proximal change in the spatio-temporal dimension. Take (3) for example. Doctor Bailey receives psychotherapy after the shooting in Seattle Hospital in Gray’s Anatomy (Season 7, Episode 1). She is talking about her feelings towards the shooting with the psychiatrist in the scenario.

(3) Psychiatrist: On the, uh, day of the shooting, I understand that you had a particularly difficult day.

Bailey: On the day I gave birth to my son, my husband had a car crash, and he needed a craniotomy, and he almost died on Derek Shepherd’s table. That was a particularly difficult day. And at the end of that day, my son was born and my husband lived. But at the end of this day, (recalling the shooting she suffered from) uh, this day was the worst day of my life (recalling that she leaves her husband to her mother’s home with her son after the shooting).

There is a prominent linguistic phenomenon in the dialog between the psychiatrist and Doctor Bailey who is one of the victims of the shooting in Seattle Hospital. Doctor Bailey uses two that and two this to identify two days that changed her life. The demonstratives that and this can be interpreted in terms of the temporal proximity in the reality, the tense, and the discursive proximity to the speakers respectively.

Given the temporal proximity in reality, the birth of her son and the survival of her husband happened earlier than the shooting in Seattle Hospital when Bailey talks with the psychiatrist. Apparently, the former are farther away from the speaker Doctor Bailey than the latter on the timescale in reality. Thus, it is justified for Bailey to use that to demonstrate the day when she gave birth to her son and that her husband survived a car accident, and this to identify the day of the shooting. The uses of that and this in (3) adhere to the temporal proximity in reality.

According to the tense, Halliday and Hasan (1976) argued that “that tends to be associated with a past-time referent and this for one in the present or future” (1976, p. 60). Therefore, that is supposed to be used to demonstrate the day when Bailey gave birth to her son and that her husband survived a car accident in light of the simple past tense indicators gave, had, needed, died, and was. And it is true with the day of the shooting by virtue of the simple past tense indicator was. However, the speaker uses this to demonstrate the day of the shooting in (3), which is contradictory to the principle of making the choice between that and this by virtue of the tense.

Regarding the discursive proximity to the speakers, this can be distinguished from that on the ground that “‘what I have just mentioned’ is, textually speaking, ‘near me’ whereas ‘what you have just mentioned’ is not” (Halliday and Hasan, 1976, p. 60). This criterion is closely related to two communicators I and you, which is similar to the shift of the perspective when making the choice between that and this. In (3), for Bailey, the psychiatrist is you and the day of the shooting that he mentions is not near her in accordance with the discursive proximity. Thus, when Bailey answers his question, she is supposed to use that to demonstrate the day of the shooting instead of this. On the contrary, since she initiatively mentions the day when her son was born and that her husband was saved by Dr. Shepherd, she is supposed to use this to identify it to convey the meaning of being near me. The uses of that and this in (3) are totally not in line with the principle of the discursive proximity to the speaker.

These three interpretations of that vs. this in the verbal expressions of (3) are summarized in terms of acceptability, ground, and revision in Table 1.

Table 1 Three interpretations of that vs. this in the verbal expressions of (3).
Full size table

However, no matter whether these three interpretations are theoretically acceptable or not, the fact is that the uses of that vs. this in (3) are accepted by native English speakers. An appropriate explanation is that the difference between remote and near can be neutralized on the utterance level (Sennholz, 1985) by the visual deictic vectors in the meaning-making process. In other words, a visual deictic vector may force demonstrative that to transform into this in some cases. According to the second and third interpretations, the day of the shooting a month ago should be demonstrated by that on the ground of the tense and the discursive proximity to the speaker. But Bailey uses this to demonstrate it instead of that. This is because the visual images from Bailey’s memory fill the vacuum of the identity of the day, including some miserable pieces of what happened to her and her colleague on the day of the shooting and then her departure to her mother’s home with her son after the shooting. The memories are visualized on the screen but not verbalized in the linguistic expressions since “not all of the meanings conveyed visually are also conveyed verbally” (Kress and van Leeuwen, 2006, p. 53) (see Figs. 7 and 8). The deictic function is realized by the episode shift in the narrative (Galbraith, 1995). As the horrible and panic memories in and after the shooting are presented in the dynamic images with high fidelity and inserted in the forward movement of the communication, it is justified to transform the discourse deictic that into this to highlight that the fear still lingers in her mind. This reflects the advantage of multimodal discourse in which the dynamic images may present what has happened without the spatio-temporal restraint, and impact the linguistic deictic choice between that and this to identify an element.

Fig. 7
figure 7

Visual and verbal deictics in Gray’s Anatomy (Season 7, Episode 1) annotated by ELAN 6.3.

Full size image
Fig. 8
figure 8

Visual and verbal deictics in Gray’s Anatomy (Season 7, Episode 1) annotated by ELAN 6.3.

Full size image

Besides, what that and this demonstrate in (3) is the abstract entity day that can be directly construed by the predominate symbolic linguistic elements, but indirectly identified by the dominant iconic visual ones. To be specific, the day that that identifies is directly construed by the linguistic expressions as “language is a dominantly symbolic mode” (Stöckl, 2004, p. 26) in this example. On the contrary, the day that this demonstrates is indirectly realized by the dynamic images that visually show what happened on the day of the shooting. The visual deictic images imply that the shooting still affects the victim. With respect to the dynamic, demonstrative that embodies the flow of the meanings which inherits from the events (the birth of her son and the survival of her husband) verbalized in English. Demonstrative this is accompanied by the dynamic images of what happened on the day of the shooting without any verbal explanation or narration. Though the visual and linguistic semiotics often synergize with each other to indicate what happens in the co-contextualizing relation in simultaneity, the visual images of what happened on the day of the shooting appear after Bailey’s words at the end of this day in (3). The sequential organization foregrounds the verbal expressions and the visual images separately to underscore their equal stances of conveying the meanings. Basically, the dynamics of that and this are the same in nature since they originated from the dynamic reality. But what that demonstrates is realized by the linguistic semiotic. It forms a contrastive pair with this and the visual reappearance of what happened on the day of the shooting with high fidelity as the visual proximal deictic vector. In contrast, what this demonstrates is construed by the visual semiotic. But there is no visual distal deictic to identify that day, which is the specific semogenic phenomenon in multimodal discourses.

Conclusion

The study illustrates that when demonstrative that cannot adequately identify an element in the linguistic expressions, the visual deictic vectors, for example, the arrows, the curve line, the cross, the gaze, and other visual images, may eliminate its deictic indeterminacy of that in multimodal discourses. The explicit and implicit that can shed light on the divergence of the visual proximity from the linguistic one, especially when an identical element is demonstrated by different semiotic deictics. The study also exemplifies that the visual deictic may break off the spatial and temporal restraints and reappear in the forward movement of the communication, which forces demonstrative that to transform into this. The instances illustrate that the visual deictics are not the paralinguistic features of the linguistic demonstratives just as the visual images “are not merely embellishments to language” (Norris, 2004, p. x). The visual images “as a legitimate semiotic resource” (Lim-Fei, 2004b, p. 52) can affect the choice of linguistic demonstratives by virtue of the predominant iconic nature. The divergence of the visual deictics from the linguistic ones responds to different dominant semiotic natures of the visual images and language which have been discussed by other scholars such as Peirce (2005) and Stöckl (2004).

Now that “a knowledge of other semiotic modes can open new perspectives on language” (Kress and van Leeuwen, 2006, p. ix), the intersemiotic interpretation of demonstrative that provides an academic lens for the understanding of the multi-dimensional reality that we live in. This study does not exhaust all the visual and linguistic deictics, which leaves room for the intersemiotic studies on other deictics in other genres such as the advertisement, the news reports, the commercials, and so on in the future.