![[PDF]](/pb-assets/Icons/adobe-pdf-icon-1498649896243.png){kind=icon}
Abstract
Our understanding of reading—including reading multimodal texts—is always constrained or opened up by what we consider to be a text, what aspects of a reader’s embodied activity we focus on, and how we draw a boundary around a reading event. This article brings together five literacy researchers who respond to a human-scale graphic novel, comprised of over 300 large-scale paintings, recently exhibited in an art gallery and also published in print form. The researchers' responses reflect a variety of theoretical orientations, including postcolonial theory, critical theory, affect theories, new materialisms, social semiotics, and reading development theories. The author of the novel also reflects on his own creative processes and goals. These various responses, and the multiple modalities of the work itself, are intentionally juxtaposed in order to create productive tensions, contrasts, and open spaces for reconsidering how multimodal texts are read and experienced. Dimensions of reading as a meaning-making, affective, embodied experience are productively put into play with one another.
Readings and Experiences of Multimodality
Louise Rosenblatt, drawing on Bentley, discussed the impossibility of understanding organism and environment as separate from one another, building the construct of “transaction” in reading that remains significant to date:
We do not, however, take the organism and environment as if we could know about them separately in advance of our special inquiry, but we take their interaction itself as subject matter of study. We name this transaction to differentiate it from interaction. We inspect the thing-seen not as the operation of an organism upon an environment nor as the operation of an environment upon organism, but as itself an event. (Bentley, 1954, as cited in Rosenblatt, 1978, p. 17)
A perspective on meaning-making where the poem comes into being through a transactional “event” between all that the text brings to the table and all that the reader brings to the table—where meanings are shaped across the ecology of an event—was incredibly prescient for its time and is still difficult for us to lay hold of in reading research, several decades later. Of course, there are many social, cultural, and practical reasons why holding an individualist conception of the reader (with capacities, interpretations, and a history) and the text (with modalities, affordances, and linguistic traits) separately stays with us. As a way of knowing (for the reader) and understanding (for the researcher), the idea of the transaction as a fundamentally relational movement, where the nexus of reader/environment are inseparable, remains radical and challenging.
For research on multimodality, the question of the transaction as a relational movement, where the textbook page, the YouTube video, or the street sign become produced through the interactions of them as “texts” with the particular interpretations of the “reader” are vast and thorny issues to consider, especially in ways that are deeply relational, refusing to isolate text or reader. While Rosenblatt wanted to recover a strong sense of the reader’s purposes, history, and individual meaning-making in the transactional nexus, multimodal research on the other hand has often been focused on the text and its different modes, with an eye toward the “affordances” or possible transactions that may happen with imagined readers.
Moreover, our vision of an ecological perspective is not merely an abstraction or model but based on the actual actors that we put into a scene, what the temporal–spatial qualities of that scene are, what reading (and writing) come to be in that scene, and what other experiences are taking place. Consider Rosenblatt again:
A group of men and women, graduate students in English, were handed a text. They were told that they were to remain anonymous, and that they should start writing as soon as possible after beginning to read. They were not asked to introspect about what they were doing, but simply to jot down whatever came to them. (Rosenblatt, 1978, p. 6)
Presumably, the scene in this case was a classroom or academic lab space, a desktop, a quatrain by Robert Frost (“It Bids Pretty Fair”), a writing tablet, a pen, and a graduate student’s memory and experiences. The way in which we understand reading as a transaction, and what poem may be created in the event of reader/text, will depend not only on a general concept of “ecology” but also on the particular participants within a given ecological array and their relations to one another. In this case, our notion of the transaction is shaped on an ecology where individuals work alone, where bodies remain stationary, where materials other than the text are made sparse, where the text is a canonical poem, and where the first response to meaning-making is writing.
By contrast, consider the following scene, the participants within its ecological array and their relations to one another:
A crowd of people move through an art gallery to read and experience a new work, Th3 Anomaly: Crossing the Rubicon. This work is a very large-scale graphic novel that has been painted by author and artist David Landry, in an effort to bring “high art” to a “low art” form. The 321 paintings comprising the graphic novel are large and some life-sized; together they fill all of the walls of one large exhibit room and travel down a long hallway, to offer the sequence of the story in chapters comprised of giant 8-foot pages. In the hallway and exhibit room are displayed objects designed as models for the finished paintings: a ray gun, a dirigible balloon, a wide-brimmed fedora, and many others. Costumed actors and actresses, who also served as models for the paintings, roam throughout the space. Individuals in the crowd chat with one another, laugh, pull in close to the character dialogue and images on the mounted paintings, step back to ponder larger views of the work, walk backwards and forwards through the hallway, skip large sections, and interact with the author, who roams about. (Observation, January 30, 2016)
Of course, there are many obvious changes from the ecology designed by Rosenblatt in conceiving of reading as a transaction: The text is experienced socially, bodies move about, the text is surrounded by material objects and human characters from the storyline, the text involves a fanciful “low art” comic book story painted in “high art” style that moves back and forth in time and space, bringing together the historical characters of Nicola Tesla, Jules Verne, and others. Moreover, the moving bodies of those experiencing and reading this massive graphic novel are relatively small compared to the 8-ft “pages” displayed—the author of this ecology, David Landry (see Artist’s Statement in the Online Appendix), worked toward designing an “immersive” environment and often called the exhibit of Th3 Anomaly a “world.”
The preceding description is not only for the sake of contrast or illustration but is offered up as an introduction to the following “gallery” of pages, in which you are invited to join with us in reading and experiencing Th3 Anomaly. Our goal in these pages is to offer an expanded view of multimodality, which, we believe, ultimately, offers an expanded view of reading. One of our arguments in the piece—more or less explicit—is that readings of multimodality cannot be separated from our (embodied) experiences of multimodality. Hence, our subtitle of this piece, “Readings and Experiences of Multimodality” captures the “and-and-and” we have engaged in to offer up multiple readings associated with multiple embodied experiences. Prior to describing more of this multiplicity, Kevin will describe the history and process of the project in the section given subsequently.
Multiple Readings and Experiences of Th3 Anomaly: A Brief History
In January 2016, Kevin visited the exhibit of “Th3 Anomaly” in a warehouse district in Nashville, which is currently being transformed into a maker space and arts center. Having heard of the project of a life-size graphic novel, he became fascinated with the project and the work, spending hours in the gallery on different occasions, returning with family members, wearing a head-cam on one visit, and eventually returning to visit the exhibit during off hours with leadership from Literacy Research Association (LRA). David Landry, the artist and author of the work, was invited to exhibit the work at LRA as part of a session on multimodality, and we decided that three of the 8-foot panels or “pages” could be safely transported to the convention center.
The plan for this exhibit was that LRA conference goers as well as the speakers in our multimodal (research review) session would be able to experience something of the immersive qualities of the work at the conference. Of course, that plan involved a much constrained version of the work involving a display of less than 10% of the work, no exposure to physical models or characters, no sense of the whole, and other related limitations. Kevin shares this story not merely to discuss the constraints of this “experiment” for LRA but to make evident again that part of why we come to (mis)understand literacy as a particular kind of practice, given how we normalize the material and ecological qualities and relations of this practice. For example, some texts are much more mobile than others. For the most part, we come to know literacy by studying texts that function most readily as “immutable mobiles” (Latour, 1986, p. 10)—objects that travel readily, without distortion, and can be combined with other texts—or by reducing expansive, clumsy, and immersive texts into immutable mobiles. In the end, and despite our best efforts to disrupt typified literacy practice, much of how the writers in this piece came to know Th3 Anomaly was through a book version of the graphic novel that was sent to us ahead of the LRA conference. As such, we had an early tabletop experience of the work and then a later, reduced experience of the walk-through exhibit. Thus, we had no singular experience of the work, which supports a broader argument of the piece: There is no singular reading or experience of the work.
Speakers for the session (all coauthors of this piece) were selected for our markedly different approaches to multimodal texts and were prompted with the following questions in preparing their comments:
What types of connections are you making with the text/work as you interpret and/or experience from the tradition of multimodality you have synthesized?
At the heart of this session is the notion of transformation. As meanings shift, transcend, and accumulate, how do we make sense out of what it means to read?
What counts as text in this form of “reading”? How do we find ways to meaningfully contribute to material, print, digital, and popular culture?
What parts of the body or mind are engaged in this form of “reading”? What forms of thinking or feeling are most active and what forms are not? What seems gained and lost?
(How) does this life-size text challenge the assumptions made by the perspective you are reading/experiencing from?
These questions were merely constructed as opening prompts for further thinking and seemed to be more productive for some of us than others. Even in writing these questions, it was difficult to cast them in ways that would allow for the entire range of divergent responses. For example, while some of us were clearly more invested in a more conventional notion of “reading” as meaning-making, others were more invested in the idea of embodied experience.
After the researchers presented their responses at the research review session, along with Th3 Anomaly author David Landry, we each wrote brief response pieces as separate papers. In effort to juxtapose our voices and our multiple readings and experiences of the work, Kevin divided each of these written pieces into “bits” that could be potentially put into relation to the author’s work and to the embodied experiences of those at the exhibit. Kevin then selected three pages from Th3 Anomaly and composed an additional page of photographs from the exhibit event. Together, these four pages (included in this article) became a surface on which he could associate the “bits” from the respondents and the project author, as annotations, placing them in multiple juxtapositions. Kevin’s orientation with this type of “spatial writing” was to shape the “and-and-and” nature of multiple readings and experiences of the text. Moreover, he also believed that in the spaces between the annotations and in the spaces between the annotations and the images, important differences are created that may invite further investigation and questioning in the field. In essence, the attempt here is to recreate the multimodality of the work itself for our readers—each bit or annotation is like a panel, and the invitation to our readers is to read and experience “in” the panels, “between” the panels, and back away from the panels. The risk that we are hoping for in this piece is that the “unsaid” is just as important as “the said” for moving toward a rich theory of multimodality. Especially, at this moment in time, we believe that a rich history of ways of reading and the social semiotics of multimodality are coming into contact with critical theory and with newer embodied theories of affect and materiality. The contact zones of these movements create breaks or ruptures with one another, of which we believe will emerge productive difference.
A Note on Stances as (Internal) Perspectives and (Embodied) Prepositions
One of the advantages to a very large-scale text is that, relationally, it materializes what gets greatly reduced or perhaps internalized when we shrink down to the desktop in creating an ecology. The famous spectrum of reading “stances” from Rosenblatt involve, on the one hand, “efferent” readings (literally, where meanings are “carried away”) and “aesthetic” readings, concerned with formal textual features related to beauty. For our purposes, we are concerned not with this duality but with how Rosenblatt’s (and most others’) conception of stances itself (in the reading ecology) is internal to the reader. No “body” is taking a stance with respect to the text—the inert text is experienced by the inert body, held still except for the representations it is creating from memory or from the experience of perceiving the text.
Materializing the notion of stance, and stepping out of mentalistic constructions of it, may provide literacy scholars ways of seeing that which we have already been doing to texts and with texts as well as some newer ways of thinking about these relations. Consider each of the author’s positions with respect to “Th3 Anomaly” described not merely as mind-sets but also as embodied orientations. Presently, these relations are described as prepositions rather than perspectives so as to suggest a relationship with the text that is more fully embodied (Table 1). Rather than privileging one (prepositional) relationship over another, these stances are represented in ways that are relational with respect to one another and more fully grounded in experience.
|
Table 1. Prepositional Stances With Respect to Text.
From an embodied perspective, and in understanding our relations between cognitive or critical stances and embodied stances, we might imagine a range of concurrent and copresent relations to multimodal texts that would far surpass a spectrum of efferent and aesthetic. Such concurrent and copresent relations are brought to life in the following pages, with the hope that their copresence creates not merely the evidence of multiple readings and experiences but a living energy from their movements in shared space.
Finally, as a transitional thought before moving into the multiple relations of images and annotations: David Landry wrote in reflection to his own work, “I did not want to be a superhero—I wanted to draw superheroes.” In the end, perhaps that is our best hope for the kind of energy, we can create in new directions of multimodality—that reading experiences, in our renderings of them, will become more multiple, expansive, and powerful and that these drawings of superheroes that we sketch through research and theory will invite readers of our work to enter in—to want to be (Figure 1).
Jules Verne was French. He was born in 1828 and died in 1905 (aged 77). He was a novelist, poet, and playwright. Intertextually, the reality of lived experiences of characters intersects with fictional existence of characters within this graphic novel (Aziz).
I posit here that images are where facts, in visual form, are kept, transferred, conceptualized, reconstructed, and altered by an imperial gaze into the myths and metaphors of place and identity. Further, mainstream ways of seeing makes the dominant cultures view their own imperial power (Aziz).
Consider Deleuze and Guatarri’s (1987) example of affective qualities of everyday life that require such nonrepresentational orientations to knowing. They contrasted the qualities of feeling of “stepping off a train into the bustle of Central Station at five o’clock on a winter Monday morning” with “passing the threshold to home as twilight falls” at five o’clock in the afternoon while “around the neighborhood a nation of televisions flicker on to keep company with the night” (in Massumi, 2016, pp. 9–10). These qualities of “five o’clock” have intensity and feeling that resist representation and stability and that resist scales and structures, which can be easily parsed and portrayed in pieces (Ehret).
Three tenets of postcolonial theory that I based my analysis on are: imaginative geographies, the “gaze,” and “othering.” (Schwartz, 2003) (Aziz).
Being critical in reading this text is an imperative to appreciate the subliminal visual, metaphorical, and real messages that are embedded in its visual/ verbal representations. The play of power relationships that encompasses this brilliantly colorful and vivid world of a futuristic text, projects dominance of one kind of people from past to present catapulting them into a shaky future, and it therefore points toward colonial/postcolonial concerns and issues. Further, the main protagonists project an imperialistic viewpoint through their actions, narrative structures, and the plot (Aziz).
Wetherell (2014) insists that affects needs to be linked with meaning-making. She talks about affect and meaning making as entangled during production:
Human affect and emotion are distinctive because of their immediate entanglement with very particular human capacities for meaning-making. These entanglements organize the moment of embodied change and are crucial to the ways in which affect articulates and travels. They need to be center stage in any social theory of affect and emotion (para. 2).
Thinking about this quotation, I have gradually moved away from a design-centric notion of multimodality to far more affectively driven modal practices (Rowsell).
Readers look at panels, captions, and word balloons in order to understand what is happening, and the gutters, the blank spaces that separate all these features, foster leaps in logic, that is to say inferences, much in the same way that reader response theorists (e.g., Iser, 1978) posited that gaps in print text did (Botzakis).
In analyzing this novel, I came away with the understanding that the perception of the world as a playground for the privileged few is an ongoing rhetoric. As Massey (2009) posits, “There is still the perception among some people that geography is boys conquering the world by various strange means. TV programmes can give this image…but also because it’s such a disrespectful view of the planet—not only to the people already there, but also because it resonates with that imperialist view that we can have a right to go anywhere and treat the world as our playground” (p. 82; Aziz).
Creating Th3 Anomaly was nearly a 5-year project and required the help of all of my friends at abrasiveMedia. Working with such a diverse group of artists helped to challenge my own preconceived notions about diversity. And thankfully I continue to be challenged with each new interaction that I have regarding Th3 Anomaly (Landry).
Sensual signs, such as the madeleine, force thought into action, and bring to life the Quality of an experience or place. These signs therefore reveal more than an arbitrary association between thing and past experiences. They disclose the Quality, for Proust’s narrator, of Venice and of Combray (Ehret).
What signs represent is not what sends nonrepresentational theorists searching; it is the Quality of life of that signs are only a part of the making. And it is the quality of signs’ coming together in singular experiences that move bodies and that contribute to an overall Quality of the feeling of a singular something, like five o’clock (Ehret).
Fiction: Tesla/Verne: Main Superheroes as Glocal Citizens (Stephens, 2013): Tesla belongs to the United States and Verne belongs to France. Both cross-over (decohere/recohere) with the Rubicon’s power that connects past, present, and future but do not age or regress in age. Tesla has an eternal life and is resurrected after death by his own brother who heads the evil brotherhood organization. Tesla also sires a son that confirms his permanent presence in world. Tesla also does not lose control even knowing that Orius will kill him dramatically because of a vision he saw while crossing over. This novel seems to be a tribute to Tesla (Aziz).
Education research on comics has been similarly sparse until recently, and consequently the processes and actions involved in reading comics have been largely unresearched and undertheorized. Most famously, reading comics has been called an “immediate experience” (Warshow, 2001) or an “invisible art” (McCloud, 1993), but such phrases only point to the speed of comprehension and sell short the complexity bound in reading texts that utilize images and print in unique manners (Botzakis).
Schwartz (2003) contends that the colonial powers use, “Masterful gaze of detached authority in projecting a world through their eyes” (p. 155). Schwartz contends that the new medium of photography/images was employed by colonial powers in various ways to establish imperial control, extend imperial connections, and articulate imperial identity. This idea of a “scopic regime” of visual images is deeply involved in the process of colonial hegemony, affirming, and reconstituting the physical appropriation of space. (Aziz)
In hindsight, there are some things that I would have done differently, such as the black-facing of certain characters in the story. Rather than simply painting a range of colors into my own stories, I needed to seek out the words of other cultures and tell their stories as well. I love that about art. It inspires us to think critically and allows us to embrace our differences (Landry; Figure 2).
Massey’s (2009) reading of the “world-as-playground” can do much to dispel the notion of post-colonialism, and the irony of such a concept lends itself to a greater and continued emphasis upon the present-day workings of colonial ideology through the medium of a graphic novel (p. 82; Aziz).
In this way, Deleuze (1972/2000) argues, “the Search is oriented to the future, not to the past” (p. 4). The search is therefore an ongoing process of discovery, a narrative apprenticeship in signs, which, hieroglyphically, deny immediate understanding. Over time, unfolding the meaning of signs, in their multiplicity and multiple forms, lead to a revelation from the work of art. Reading the search becomes an apprenticeship in singular signs that together create a sense of unity, a feeling, and a revelation that is not contained in any one sign alone (Ehret).
During interviews with professionals, they have frequently alluded to the emotional nature of multimodal design work. The interviews that I have conducted in the past have convinced me that emotions trump logic, or, expressed differently that designs are not necessarily structured and built, but instead evolve out of thoughts, ideas, convictions, felt sensibilities, and feelings (Rowsell).
According to Said (1994), “…none of us is completely free from the struggle over geography. That struggle is complex and interesting because it is not only about soldiers and cannons but also about ideas, about forms, about images and imaginings” (p. 6; Aziz).
Although visual texts proliferate in everyday life, they are not always deemed apt or rigorous enough for academic study, a condition that calls into question the relevance of solely emphasizing (and assessing) past modes of reading (Botzakis, Burns, & Hall, 2014) (Botzakis).
The work of Serafini (2010) has been instrumental in creating a framework that drew on multiple perspectives from a diverse array of fields, including art theory, communication studies, media literacy, semiotics, visual grammar, visual literacy, and literary theory in order to more fully realize and explain the multimodal effects that occur when people read comics. He has also coined a new term to describe such visual texts, calling them “multimodal ensembles” (Serafini, 2014, p. 11), highlighting how they were constituted of the combined elements of “visual images, graphic designs, and written language” (p. 11; Botzakis).
Signs of love differ from worldly signs in that they evoke an unknown, yet potential, world: “The beloved expresses a possible world, unknown to us, implying, enveloping, imprisoning worlds that must be deciphered, that is, interpreted” (Deleuze, 1972/2000, p. 7). Falling in love is thus an apprenticeship in signs of love, a becoming sensitive to them, and pursuing the possible worlds and relations implied in, for instance, the beloved’s eyes (Ehret).
As Deleuze (1972/2000, p. 4) argued, “Learning is essentially concerned with signs. Signs are the object of a temporal apprenticeship, not of abstract knowledge. To learn is first of all to consider a substance, an object, a being as if it emitted signs to be deciphered, interpreted.” This is reading as experience, and experience as reading: to become an apprentice to signs. (Ehret)
Women in this novel are seen as secondary characters emotionally or physically dependent on men (as exemplified in the characters of Maureen: Verne’s mute daughter; Sarah: a pod for pregnancy, carrying Tesla’s child; and Orius: a hybrid & confused personality, part of whom loves Tesla). (Aziz).
The author Scott McCloud (2016) put it best when he wrote, “The heart of comics lies in the spaces between the panels–where the readers’ imagination makes the still pictures come alive!” (p. 1). The invisible spaces between the panels of a comic book are just as essential as the illustrated panels themselves. You, the reader, complete the story by filling in the blanks with your own past experiences and knowledge. Everyone has a different experience between the panels. That is the magic of sequential art, the story is not complete until you read it. (Landry)
To be specific, Landry’s illustrations are not profoundly about looking good/being aesthetically pleasing (Van Leeuwen, 2015) or about sounding smart and intellectual. They are about an emotional journey. The driving force of Landry’s graphic story seems less about following the story lock-step than it is about being enchanted by the visuals and kineikonic movements (Burns & Parker, 2003) that the illustrations invite. (Rowsell)
Worldly signs are everyday encounters with social conventions, conversations, customs, and decorum, which pose enigmas such as, was that a furtive glance or disdainful glare? (Ehret)
We also need to take into account Haraway’s (1988, 1991) conception of situated knowledge(s) where she argues that where the gaze is premised upon illustrates the difficulties in achieving geographical objectivity and passivity. Thus, informing us that subjectivity in how we view the world is impossible. This is projected in most of the images within the graphic novel. The visual gaze implied or otherwise direct the readers gaze in the various directions that the author/illustrator’s vision projects. Therefore, we as readers observe a smug, well-dressed Tesla as he proceeds to unravel all the mysteries of the past, present, and future of this world within the full spread of pages 29 and the final image on page 121 (Aziz).
“How big is red? As big as it comes and as small as it gets. As wide as a sunset, and as bitsy as a pixel. What size is pain?” (Massumi, 2016, p. 123). Contrasting with social semiotic theories of multimodality (e.g., Kress, 2010), nonrepresentational theory does not begin with a search for smallest units accreting into larger scale compositions. For example, nonrepresentational theory is interested in degrees of intensity, in capacities to move and be moved (Ehret).
Research on reading has isolated specific areas of importance, which in turn have been operationalized in ways that some find deleterious to instruction (e.g., Meier & Wood, 2004). Thus far, views on reading comics have resisted such atomization. Preliminary research into the conventions of comic reading has shown that understanding simple comics, such as four-panel comic strips might be akin to sentence-level comprehension of print texts (Cohn, 2013), but researchers looking at more substantive texts, such as graphic novels that span many pages (e.g., Jimenez & Meyer, 2016), have shown that when texts are longer and more substantive, those rules are abandoned (Botzakis; Figure 3).
David Landry’s graphic story, Th3 Anomaly: Crossing The Rubicon, with its bright, exuberant color palette and its arresting illustrations stirred and enchanted me. This enchantment became all the more real and felt when I saw the three-dimensional structures that he built for the larger installation of his story which he exhibited in Nashville some time before our expert panel convened at LRA (Rowsell).
My passion for sequential art began as soon as I was able to read. Even as a young kid, I was inspired by the idea of telling a narrative with a medium that uses visual art, literary elements, auditory triggers (through onomatopoeia), and welcomes the reader into the narrative process through the use of panels (Landry).
The primacy of print has long extended into the arena of literacy research. Some of the first, well-respected theoretical models and processes of reading (e.g., Gough, 1972; Samuels, 1994) incorporated graphic or visual components into their bases but in the end shifted the focus from imagery to that of language comprehension. Even a later model built around a dual coding system of verbal (logogen) and nonverbal (imagen) units ultimately represent and propagate language comprehension (Sadoski & Paivio, 2004). Thus, even though sequential art, the medium of comics, is communicated via words and images, it has been explored in largely monomodal manner in terms of literacy education (Botzakis).
Rather than imposing an evaluative, perhaps even punitive yardstick to meaning making and communicative practices that dictates what literacy and multimodal work counts, whose culture and class counts, a nonrepresentational and embodied view of multimodality opens up communication to difference and listens to alternative framings and contestations (Rowsell).
I conceived of a very ambitious project, a life-size graphic novel. But simply making a comic book bigger was not enough for me. My project needed to be a brand-new experience (Landry).
Blommaert and Backus (2012) have explored how semiotic repertoires are drawn upon in a variety of contexts over time to support young people’s meaning-making trajectories. These modal choices are filled out with meaning and are imbued with ways of knowing that are historically situated. Modes are culturally shaped and subject to cross-cultural ways of knowing and seeing. This presents a challenge to the text-based stance of multimodality. Instead, multimodal meaning-making can be linked to the movement of bodies in space, or, so I argue in this short piece (Rowsell).
I take the mediation between visual and verbal text to analyze the symbiosis of both (Barker, 1989; Dittmer, 2005; Eisner, 2008, Kress, 2010; Van Leeuwen, 2015). This text demands intertextual analysis, especially in the intermingling of the visual and verbal texts along with real lived experiences of the two main characters. In analyzing the text through multimodality, the meaning of the visual and the verbal text enmeshes and alters dramatically (Aziz).
How big is red? What size is pain? On the one hand, representational logic requires a process of back-formation that searches in the smallest textual features for signs of experience for the intentions of the text maker as a rational designer of signs. On the other hand, nonrepresentational logic meets the past as it emerges in its singular newness, the text-maker-materials-signs moving each other in more than human intra-action and bringing something unintended to life: redness, pain (Ehret).
Readers take different tactics as they read: Some look at images first and others the text. Some look at the page as a whole and then circle around to scan what they perceive as the flow of the page to be. Others start on the top left and work their ways to the right and down. In addition, their background knowledge of page design, art history, genre, and media literacy (Serafini, 2014) comes to be more important in understanding specific references, jokes, and other text features (Botzakis).
One of my favorite parts about my life-size graphic novel is watching the different ways that everyone approaches an immersive story (Landry).
Using redesign or transformation as an agentive act would create a more transformed, diverse, and just world where meanings shift and transcend but where there is a world not only ruled by the dominant thought and imaginings but where everyone has a presence and a voice (Harste, 2010; Janks, 2010) (Aziz).
Insights from the affect and new material turn in literacy studies have been instrumental in viewing matter as offering agency and as playing a part in the cultural of shaping of modes in everyday life (Pahl, 2014) (Rowsell).
The madeleine and lost time are one and the same. One does not represent the other. One affects the other in an irrational and desiring searching for something, something human, and something that evokes a condition of our humanity. This sort of response to a sign is not rational and cannot be understood through a parsing of modes and their orchestrations to understand how the text compels it—in the case of Th3 Anomaly, color, facial expression, and the design grammar of graphic novels (Ehret).
Scholars who have worked in sociocultural and multimodal approaches to meaning-making and literacy have shifted their understandings of modes from designed texts with structures and grammars to materialities that are in process, in a state of becoming laden with emotions and embodiments (Ehret, Hollett, & Jocius, 2016; Leander & Boldt, 2013; Lewis & Tierney, 2013) (Rowsell).
I came of age theoretically working with Kress and Street (2006) and combining an ethnographic orientation to research with a social semiotic interpretative lens that has always informed my work, but of late, this lens has not done enough theoretical work for me and I feel constrained by structures that eclipse key dimensions that are left out like feelings, embodiment, and affect (Rowsell).
I wanted to inject deeper multimodality into sequential art than had ever been done before. The traditional blank spaces between panels became 3-dimensional levels between paintings. The characters were ripped from the pages and became real tangible costumes that the reader can interact with. Every detail of the story would surround the reader in a fully immersive environment. An environment that encouraged using parts of the brain that are not typically attributed to reading (Landry; Figure 4).
How can we develop, and redevelop, concepts related to multimodality through a nonrepresentational perspective in order to know what Leander and Boldt (2013) called an “ongoing series of affective intensities that are different from the rational control of meanings and forms” (p. 26)? How do we come to know the irrational in literate activity, in making and experiencing multimodal texts? How do we come to know the quality of a multimodal (or any) text? How do signs, and texts, move bodies? (Ehret).
The reality of the characters’ lived experiences is a necessary component to intertextuality in analyzing this novel to study concepts of power, Nikola Tesla, a prominent figure whose lifespan was from July 10, 1856, to January 7, 1943, was a Serbian–American inventor, electrical engineer, mechanical engineer physicist, and futurist (Aziz).
In a Kressian Before Writing (1997) sense, I wondered about the motivations and inspirations behind the pictures that Landry drew and how he felt deep down as he made them. When Gunther Kress talks about meaning making in Before Writing (Rowsell & Pahl, 2013), his account is so tied in with being a parent of young children at the time and an intense period of observation of his children making things that are almost entirely driven by senses and their interests. The strong kernel of a feeling that I spoke of earlier told me that Landry draws from emotions and without a set structure in advance (Rowsell).
Thirty years ago, I was given my first comic book. It was nothing more than a cheap 3-in. reprint of a Spider-Man (TM) solicitation given away as free publicity. But it was the most fantastic thing I had ever seen! There was so much compelling action, the colors screamed off the pages, and the protagonist was a kid me. Only he wasn’t just like me; he kicked serious ass and got the girl! (Landry).
Contrasting a multiliteracies design perspective with a nonrepresentational perspective on Emily, a young writer, while she wrote a series of young adult gothic novels, Smith (2017) argued that a nonrepresentational perspective allowed her to see the young composers’ process “as affective, emergent, and experimental, and how everyday experiences with text, even when they occur on the phenomenological periphery, impact their doing of literacy moment to moment” (p. 139) (Ehret).
Critical observations of traditional American comics have shown its propensity toward violence, sexual exploitation, and inflexibility in a Caucasian male-dominated genre of literature. And yet there is so much potential in sequential art (Landry).
Normally, this is where the narrator says, “and I wanted to be like Spider-Man(TM).” But I did not want to be a superhero, I wanted to draw superheroes (Landry).
Nodelman (1990) posits that placing words and pictures into relationships with each other inevitably alters their meaning so that they become more than the sum of their parts (Aziz).
Signs of art are essences, for instance, the veritable unity of a novel’s signs are “the ‘effect;’ of the multiplicity and of its disconnected parts” (Deleuze, 1972/2000, p. 163) (Ehret).
Bogue (2003) summarized Deleuze’s argument: “Truth, then, is both fortuitous and inevitable, and its exploration proceeds through chance encounters with signs that select the truth to be explored. To search for truth is to interpret signs, but the act of explicating the sign, of unfolding its hidden sense, is inseparable from the sign’s own unfolding, its own self-development” (p. 34) (Ehret).
In addition to reading specific visual images, readers have to recognize how they operate both as text features and as visual conventions, keeping in mind that the images are constructed to convey more than visual messages. They convey modalities such as sound (when the sound effect used when a ray gun is fired), action (when jagged panels communicate violence or time travel), tone (when colors are shifted in different scenes), and motion (some images being close-ups and some more panoramic views). All of these modes mentioned can and have also been communicated via print text, although the mechanics operate in different manners and not in a one-to-one correspondence from text to image (Botzakis).
Is multimodality always about representation? Is it always intentional and preplanned? Must literacy or multimodality have an intended meaning to count as meaning making? What happens to multimodality when it is emotionally driven? (Rowsell).
Further, the absence of multiple thoughts leads to othering of the absent/invisible people, within this text, which would be non-Western voices (Aziz).
Hence, a deceptively simple way of knowing signs in a nonrepresentational theory of multimodality: What is signed is the nature of our searching. This is the quality of the search and of a nonrepresentational theory of multimodality (Ehret).
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental Material
Supplementary material is available for this article online.
References
|
Barker, M. (1989). Comics: Ideology, power and the critics. Manchester, England: Manchester University Press. Google Scholar | |
|
Blommaert, J., Backus, A. (2012). Superdiverse repertoires and the individual. In de Saint-Georges, I., Weber, J. (Eds.), Multilingualism and multimodality: Current challenges for educational studies (pp. 11–32). Rotterdam, the Netherlands: Sense. Google Scholar | |
|
Bogue, R. (2003). Deleuze on literature. New York, NY: Routledge. Google Scholar | |
|
Botzakis, S., Burns, L. D., Hall, L. A. (2014). Literacy reform and common core state standards: Recycling the autonomous model. Language Arts, 91, 223–235. Google Scholar | |
|
Burns, A., Parker, D. (2003). Analysing media texts. London, England: Continuum International. Google Scholar | |
|
Cohn, N. (2013). The visual language of comics: Introduction to the structure and cognition of sequential images. New York, NY: Bloomsbury. Google Scholar | |
|
Deleuze, G . (2000). Proust and signs: The complete text. Minneapolis: University of Minnesota Press. (Original work published 1972) Google Scholar | |
|
Deleuze, G., Guattari, F. (1987). A thousand plateaus: Capitalism and schizophrenia (Massumi, B. , Trans.). Minneapolis: University of Minnesota Press. Google Scholar | |
|
Dittmer, J. (2005). Captain America’s empire: Reflections on identity, popular culture, and post-9/11 geopolitics. Annals of the Association of American Geographers, 95, 626–643. Google Scholar | Crossref | |
|
Duncan, R., Smith, M. J. (2009). The power of comics: History, form and culture. New York, NY: Continuum. Google Scholar | |
|
Ehret, C., Hollett, T., Jocius, R. (2016). The matter of new media making: An intra-action analysis of adolescents making a digital book trailer. Journal of Literacy Research, 48, 346–377. Google Scholar | SAGE Journals | ISI | |
|
Eisner, E. (2008). Persistent tensions in arts-based research. In Cahnmann-Taylor, M., Siegesmund, R. (Eds.), Arts-based research in education: Foundations for practice (pp. 16–27). New York, NY: Routledge. Google Scholar | |
|
Gough, P. B. (1972). One second of reading. Visible Language, 6, 291–320. Google Scholar | |
|
Haraway, D . (1988). Situated knowledges: The science question in feminism and the privilege of partial perspective. Feminist Studies, 14, 575–599. Google Scholar | Crossref | ISI | |
|
Haraway, D . (1991). Simians, cyborgs and women: The reinvention of nature. New York, NY: Routledge. Google Scholar | |
|
Harste, J . (2010). Multimodality. In Albers, P., Sanders, J. (Eds.), Literacies, the arts & multimodality (pp. 27–43). NCTE. Google Scholar | |
|
Iser, W. (1978). The act of reading: A theory of aesthetic response. Baltimore, MD: The Johns Hopkins University Press. Google Scholar | |
|
Janks, H. (2010). Literacy and Power. New York, NY: Routledge. Google Scholar | |
|
Jimenez, L. M., Meyer, C. K. (2016). First impressions matter: Navigating graphic novels utilizing linguistic, visual, and spatial resources. Journal of Literacy Research, 48, 423–447. Google Scholar | SAGE Journals | ISI | |
|
Kress, G. (1997). Before writing: Rethinking the paths to literacy. New York, NY: Routledge Press. Google Scholar | |
|
Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. New York, NY: Routledge. Google Scholar | |
|
Kress, G., Street, B. (2006). Foreword. In Pahl, K., Rowsell, J. (Eds.), Travel notes from the new literacy studies: Instances of practice (pp. vii–x). Clevedon, England: Multilingual Matters. Google Scholar | Crossref | |
|
Latour, B. (1986). Visualisation and cognition: Drawing things together. Knowledge and Society: Studies in the Sociology of Culture and Present, 6, 1–40. Google Scholar | |
|
Leander, K. M., Boldt, G. (2013). Rereading “a pedagogy of multiliteracies”: Bodies, texts, and emergence. Journal of Literacy Research, 45, 22–46. Google Scholar | SAGE Journals | ISI | |
|
Lewis, C., Tierney, J. D. (2013). Mobilizing emotion in an urban classroom: Producing identities and transforming signs in a race-related discussion. Linguistics and Education, 24, 289–304. Google Scholar | Crossref | ISI | |
|
Lopes, P. (2009). Demanding respect: The evolution of the American comic book. Philadelphia, PA: Temple University Press. Google Scholar | |
|
Massey, D. (2009). I’m a geographer. Geographical, 81, 82. Google Scholar | |
|
Massumi, B. (2016). Such as it is: A short essay in extreme realism. Body & Society, 22, 115–127. Google Scholar | SAGE Journals | ISI | |
|
McCloud, S. (1993). Understanding comics: The invisible art. Northampton, MA: Tundra. Google Scholar | |
|
McCloud, S. (2016). Reinventing comics. New York, NY: HarperCollins. Google Scholar | |
|
Meier, D., Wood, G. (2004). Many children left behind: How the No Child Left Behind Act is damaging our children and our schools. Boston, MA: Beacon Press. Google Scholar | |
|
Nodelman, P. (1990). Words about pictures: The narrative art of children’s picture books. Athens, GA: University of Georgia Press. Google Scholar | |
|
Pahl, K. (2014). Materializing literacies in communities: The uses of literacy revisited. London, England: Bloomsbury Academic. Google Scholar | |
|
Rosenblatt, L. (1964). The poem as event. College English, 26, 123–128. Google Scholar | Crossref | ISI | |
|
Rosenblatt, L. M. (1978). The reader, the text, the poem. Carbondale: Southern Illinois University Press. Google Scholar | |
|
Rowsell, J., Pahl, K. (2013). After writing: Rethinking the paths to composition. In Bock, M., Pachler, N. (Eds.), Multimodality and social semiosis: Communication, meaning-making, and learning in the work of Gunther Kress (pp. 162–174). New York, NY: Routledge. Google Scholar | |
|
Ruddell, R. B., Unrau, N. J. (Eds.). (2004). Theoretical models and processes of reading (5th ed.). Newark, DE: International Reading Association. Google Scholar | |
|
Sadoski, M., Paivio, A. (2004). A dual coding theoretical model of reading. In Ruddell, R. B., Unrau, N. J. (Eds.), Theoretical models and processes of reading (5th ed., pp. 1329–1362). Newark, DE: International Reading Association. Google Scholar | Crossref | |
|
Said, E. W. (1978). Orientalism. Harmondsworth, England: Penguin. Google Scholar | |
|
Said, E. W. (1994). Culture and imperialism. London, England: Vintage. Google Scholar | |
|
Samuels, S. J. (1994). Toward a theory of automatic information processing in reading, revisited. In Ruddell, R. B., Ruddell, M. R., Singer, H. (Eds.), Theoretical models and processes of reading (4th ed., pp. 816–837). Newark, DE: International Reading Association. Google Scholar | |
|
Schwartz, J. M. (2003). Photographs from the edge of empire. In Blunt, A., Gruffudd, P., May, J., Ogborn, M., Pinder, D. (Eds.), Cultural geography in practice (pp. 154–171). London, England: Arnold. Google Scholar | |
|
Schwartz, J. M., Ryan, J. R. (2003). Introduction: Photography and the geographical imagination. In Schwartz, J. M., Ryan, J. R. (Eds.), Picturing place: Photography and the geographical imagination (pp. 1–18). London, England: I.B. Tauris. Google Scholar | |
|
Serafini, F. (2010). Reading multimodal texts: Perceptual, structural and ideological perspectives. Children’s Literature in Education, 41, 85–104. Google Scholar | Crossref | |
|
Serafini, F. (2014). Reading the visual: An introduction to teaching multimodal literacy. New York, NY: Teachers College Press. Google Scholar | |
|
Smith, A. R. (2017). Bare writing: Comparing multiliteracies theory and nonrepresentational theory approaches to a young writer writing. Reading Research Quarterly, 52, 125–140. Google Scholar | Crossref | ISI | |
|
Stephens, J. (2013). Subjectivity in Asian children’s literature and film: Global theories and implications. New York, NY: Routledge. Google Scholar | |
|
Van Leeuwen, T. (2015). Looking good: Aesthetics, multimodality and literacy studies. In Rowsell, J., Pahl, K. (Eds.), The Routledge handbook of literacy studies (pp. 426–439). New York, NY: Routledge. Google Scholar | |
|
Warshow, R. (2001). The immediate experience: Movies, comics, theatre and other aspects of popular culture (Expanded Edition ed.). Cambridge, MA: Harvard University Press. Google Scholar | |
|
Wetherell, M. (2014, 10 14). An interview with Margaret Wetherell and David Beer. Retrieved from http://www.theoryculturesociety.org/the-future-of-affect-theory-an-interview-with-margaret-wetherall/ Google Scholar |
Author Biographies
Kevin M. Leander is an associate professor of Language, Literacy and Culture at Peabody College of Vanderbilt University (USA). His research interests include the new literacy practices of youth, spatial approaches to understanding youth identity and learning, media and migration, nonrepresentational approaches to embodied learning, new materialism, and affect theory. Leander has published widely in venues such as Review of Research in Education, Ethos, Reading Research Quarterly, Journal of Literacy Research, and Cognition and Instruction.
Seemi Aziz is a visiting and adjunct assistant professor in Global Cultures, Literacy, and Literature in the Department of Teaching and Learning and Sociocultural Studies in the College of Education at the University of Arizona. Her research interests include children’s and young adult literature, issues of representation and religious freedom, and arts and visual cultural analysis in education.
Stergios Botzakis is an associate professor in the Theory and Practice in Teacher Education Department at The University of Tennessee. His research interests include middle and secondary education, adolescent literacies, popular culture, graphic novels, and media literacy. He has been published in the Journal of Adolescent & Adult Literacy, Review of Research in Education, English Journal, Language Arts, ALAN Review, and Teacher Education Quarterly among other venues.
Christian Ehret is an assistant professor at McGill University in the Department of Integrated Studies in Education. Professor Ehret investigates affective dimensions of writing, literacies, and digital culture through anthropological modes of inquiry across schools, communities, and children’s hospitals. He received a 2016 NCTE Promising Researcher award, and his research has been published in such journals as Cognition and Instruction, the Journal of Literacy Research, Research in the Teaching of English, Anthropology & Education Quarterly, and Learning, Media and Technology.
David Landry is an alumnus of the Academy of Art University San Francisco, David J. Landry has been a professional fine art illustrator for 25 years and is a Nashville based commercial scenic artist. In 2016, David presented the world’s first life-size graphic novel, Th3 Anomaly, at abrasiveMedia in Houston Station, which to this day is the largest single collection of paintings to be displayed in Nashville, TN. David has a penchant to meticulously craft into his works of art endless Easter eggs, most of which have yet to be discovered.
Jennifer Rowsell is a professor and Canada research chair at Brock University’s Faculty of Education. Her research interests include multimodal work with children, adolescents and teenagers in schools and community hubs; exploring how younger generations think and interact through technologies, videogames and immersive environments; and longitudinal research in homes connecting artifacts and material worlds with literacy and identity practices. She is Co-Series Editor with Cynthia Lewis of the Routledge Expanding Literacies in Education Series and the Digital Literacy Editor for The Reading Teacher.
