Earlier versions of this essay appeared in 1995 in Millennium Film Journal, No. 28 "Interactivities," and in 2003 in the ZKM/MIT Press publication Future Cinema. Since the ZKM publication, The Erl King, one of the works from which I developed the ideas laid out here, has been digitally 'emulated.' It now runs on a single off-the-shelf computer, while the1983-85 version requires several pieces of peripheral equipment, some custom built, communicating with a fairly specialized computer, vintage 1982. A number of interesting points emerged during the process of developing the emulation, and the last part of this essay will begin to address some of these issues. "He looked into the water and saw that it was made up of a thousand thousand thousand and one different currents, each one a different colour, weaving in and out of one another like a liquid tapestry of breathtaking complexity; and Iff explained that these were the Streams of Story, that each coloured strand represented and contained a single tale. Different parts of the Ocean contained different sorts of stories, and as all the stories that had ever been told and many that were still in the process of being invented could be found here, the Ocean of the Streams of Story was in fact the biggest library in the universe. And because the stories were held here in fluid form, they retained the ability to change, to become new versions of themselves, to join up with other stories and so become yet other stories; so that unlike a library of books, the Ocean of the Streams of Story was much more than a storeroom of yarns. It was not dead but alive."
Salman Rushdie, Haroun and the Sea of Stories (London: Granta Books, 1990), p. 71.
It is easy to read the nostalgic tone of Rushdie's 1990 "children's story" as the wish for a return to innocence, to a state of story-telling purity beyond the reaches of politics and intrigue. At the time he wrote Haroun and the Sea of Stories, purportedly for his eleven year old son, Rushdie was, of course, too sophisticated—and too embittered by the outrageous circumstances of his life—to profess any kind of innocence or naivete. The novella abounds with metaphors of cruel suppression and mindless censorship. The nostalgia embedded in the Sea of Stories is better seen as a reminder of the creative process a writer (or any artist) craves in his or her darker moments: an ocean of stories, wide and deep, effortlessly tapped. The power of the image is in its erasure of the line between writer and reader—Haroun's father, a professional storyteller, need only drink from his personal faucet plumbed to the Ocean and report to his audience: this alone constitutes writing.
Can the image be turned around? Can we imagine the Ocean as a source more for readers than writers? Could there be a "story space" (to use Michael Joyce's resonant expression1) like the Ocean, in which a reader might take a dip, encountering stories and story-segments as he or she flipped and dived? In these waters, turbulences created by the swimmer's own motion might cause an intermingling of the Streams of Story. The Ocean, as I imagine it, is a dynamic narrative region, a Heraclitean river into which one can never step twice, a lake of Heisenbergian uncertainty where the very attempt to examine a particular story-stream transforms it. What a goal to create such an Ocean! And how perfect as an ideal for an interactive fiction!
Fiction, Cinema and Cybernetics The idea of interactive fiction is not new. But the notion of a time-based medium that is not interactive is no older than mechanical recording. Before the recording of sound, all music was 'interactive,' though in this context the term sounds silly. Musicians are always affected by external factors, from the other performers in the ensemble, to the 'vibe' of the audience, to the acoustics of the hall, etc. The same can be said for theatre. The risk of variability is an essential ingredient of the thrill of live performance. And before recording, obviously, all performance was live. Though it is true that few works were deliberately designed to be influenced by external factors (but consider Tinker Bell in Peter Pan), audiences were always aware of their own potential power. Theatre and music mythology contains many stories of performers either energized or ‘thrown’ by the audience. One of the aspirations of an interactive cinema is to return the medium to this earlier condition, where the potential effect of the viewer on the performance is assumed.
In the process of developing an interactive narrative cinema, I realized early that it will not have the shape of narrative that we have learned to expect since cinema and television—media of the 'moving image'— have come to dominate our story-structures.. The very idea of user impact opens to question the concepts of end and beginning, of crisis and conflict, of development itself. Traditional notions of cinematic narrative must be rethought.
My own work is in the pull of a pair of forces that defined the late twentieth century—the Cinema and Cybernetics, the Projector and the Computer. In 1995, when this essay was first published, I had made two installations incorporating computers and moving images. In 2002 there were five, and in 2004 the first of these pieces, The Erl King, has been recreated as a fully digital piece. In all these works the participant's (inter-)actions affect the temporal conglomerate of images and sounds. The computer itself is not considered a medium or a tool, but a device that controls and presents existent media. Thus the questions that arise are about how cinema changes when its apparatus is linked to a computer—just as one can investigate changes in the structure of cinematic communication when recorded sound is added to the moving image.
The first two questions that came up can be posed in quite traditional terms. What kind of story will fit the medium, and what will be the grammar of its telling? Crudely: where is the change—in content or structure? One entry point is to find a narrative for which the sequence of events is not salient, since if the viewer is to wander around and through the story, the order in which the depicted events are accessed will need to be open to variation. And this requirement led me directly to Freud's studies of dream interpretation.
A Branching Structure "I dreamed that it was night and I was lying in my bed. Suddenly the window opened of its own accord, and I was terrified to see that some white wolves were sitting on the big walnut tree in front of the window. There were six or seven of them. The wolves were quite white, and looked more like foxes or sheep dogs, for they had big tails like foxes and they had their ears pricked like dogs when they pay attention to something. In great terror, evidently of being eaten up by the wolves, I screamed and woke up."
Sigmund Freud: "From the History of an Infantile Neurosis," in The Wolf Man by The Wolf Man (New York: Basic Books, 1971), p.173.
Freud transcribed and published the case history of the Wolf Man in 1914-15, soon after the end of the patient's analysis. It is the apex of Freud's early period, where the central concepts of condensation, displacement, wish fulfillment, the primal scene, etc., reach their full fruition, never to be resolved again in quite the same way.
The analysis revolves around the patient’s dream image of the staring wolves, -. Freud describes the process of gradually uncovering the components of the dream, linking each element with an event, a character, or an emotion remembered but perhaps suppressed. The dream's significance for the dreamer, manifested in the overwhelming emotional effect it had on him and the fact that it remained in his memory for decades, led Freud to seek further explanation. He finally accounts for this power with the proposal that the dream encapsulates the dreamer's greatest fears and desires, as transformed memories of the events that first produced them. For my purposes the details (and—it goes without saying—the `truth') of the dream-analysis are not important.2 I wish only to appropriate certain aspects of Freud's methodology in my own search for a paradigmatic story structure suitable for an interactive cinema.
Condensation is the key concept. The dream is formed by compressing and combining a set of mental objects. The dream can function in the dreamer's mental framework as the distillation of a set of emotional charges. Its powerful affect comes from the fact that, in an important sense, it embodies a set of memories and the specific emotions linked with them. Repeatedly Freud stresses that there is no universal symbol translation table—every element of the dream image, and every property of every element, is understood by the dreamer in his own individual way. Each element substantiates a combination of particular fears, hopes, desires or beliefs, transformed, by the laws of the unconscious, into a component of the dream image. Seeing the images through the dreamer's eyes—identifying the underlying atomic parts and understanding how they are altered by the dreamer's mental process into the dream image—is understanding the dream. In this understanding is written a page of the biography of the dreamer.
Freud's notion of a dream is a conception of a narrative-type based on a hermeneutic method. Unraveling a dream reveals the narrative of the dreamer's interlocking emotional states. But it is not a narrative that unfolds in time—all its elements are simultaneously present. Freud goes to great trouble to convey this atemporality, but even for him it is a notion that eludes expression. After all, his own mode of communication—writing—is, of necessity, linear, one word following another, forming paragraphs that follow one another, etc., while his conception of the dreamwork is non-linear, unsuited to the writing forms available in the early 20th century.
"This task [of forming a synthesis from fragments that emerge in the analysis of a patient] ... finds a natural limit when it is a question of forcing a structure which is itself in many dimensions onto the two-dimensional descriptive plane. I must therefore content myself with bringing forward fragmentary portions, which the reader can then put together into a living whole."
Sigmund Freud: "From the History of an Infantile Neurosis,” p.173.
Considered as a narrative structure, the underlying elements of the dream can be revealed in any order whatsoever, and the same story will emerge. Thus, it is truly a narrative without specificity of sequence.
The Interpretation of Dreams A film might try to approximate the structure of Freudian dream analysis in a story structure that step by step unravels the components of an evocative image. However, the linearity of cinema sequence tends to freeze material into narrative hierarchies, one element gaining in significance while another loses, depending on each one's context and their overall order. How better to reproduce the minimal significance of sequence, the irrelevance of order, than through interactivity? For in an interactive work the sequence of events can be determined by the viewer. And by the time the viewer becomes aware that sequence is determined by his or her responses, sequence may already have stopped being a criterion of narrative significance. In normal cinematic circumstances, the weight of an event is given largely by its context: now, with sequence under the control of the viewer, the significance of any given element will be in flux, changing from screening to screening. In these circumstances the viewer's understanding of the events of the narrative can undergo a radical transformation, based entirely on the knowledge that things could have been different.3 Later in this paper I shall make an attempt at describing the "subjunctive" state of mind evoked by the interactive cinema.
.It would be a gross oversimplification to suppose that the elements associated with a particular dream-image are, by themselves, sufficient to define the biographical narrative underlying the dream. The ways the elements are transformed and combined into the dream are equally significant, so these methods of aggregation must be considered part of the story. Often this syntax and its application can be expressed only verbally. It is difficult, for example, to imagine an effective visual expression of the transformation of something into its opposite (from "staring" to "being stared at," or from the ornate motions of sexual intercourse to the stillness of the white wolves), or the transfer of a particular quality from one object to another (as the color white is lifted from sheep and flour and attributed to the wolves). Freud's interpretation of the dream is far more than a simple compacting of memory-images into one conglomerate: the grammar of the image-elements' metamorphoses and rearrangements is as significant as the elements themselves.
I am not suggesting that the principles of condensation and displacement cannot form the foundation of a visual narrative, but only that some depiction of the types of transformation will have to be incorporated alongside the results of the transformations. The point is to develop a type of narrative that can retain its identity and make sense independent of the sequence of events. Thus, in Sonata, I found that I needed to make the formative elements of the dream, i.e. the grammar of the transformations, into explicit components of the dream-narrative—without them it became a mere medley of scenes connected only by association.
Desire Cinema, of course, cannot be internally affected by its viewers. Turning one's head, far from affecting the visual experience, removes one from the cinematic world and into the mundane space of the screening room. The straight razors and wolves on screen will yield only to forces that are profilmic, within the diegesis, or (commonly) both.
Furthermore, the impossibility of impacting on the cinematic is one of the sources of our pleasure in it. "Don't go up(/down) the stairs!" we inwardly cry out during Hitchcock's Psycho, first to the private detective Arbogast, and later to the heroine Lila Crane, all the time knowing that, no matter how powerfully felt, our distress will not influence their behavior. The experience of suspense would be fatally distorted by the elimination of the inevitability of the characters' actions. If Lila might turn back because of our pleas, the entire effect of the horror film would dissolve. Much of cinema's power over us is in our lack of power over it. In this sense, suspense is a paradigm of cinematic response. It could be argued that the introduction of viewer impact on the representation is a destructive step for the cinema. The removal of the possibility of suspense is the removal of desire from the cinematic, and, ultimately, the removal of the very fascination of the medium.
So my project became: to find interactive forms in which desire can be sustained. It will require the construction of a new cinematic grammar. If it is to be successful, this search, this construction-process, must foreground the temporal aspect of cinematic communication.4 Time. Time. Time. Time always moves relentlessly, tautologically, forward, as long as one is alive. "Real time," clock-measurable duration can always be distinguished from time subjectively felt. The duration of cinema is rigidly defined by the apparatus, fully predetermined by the physical substrate of images projected serially at a regular pace set by an electric motor. A film begins and ends necessarily and predictably. Relative to the beginning, the end is dependent on, and only on, the length of the filmstrip. Whatever its images, however they are organized, a film has a physical beginning, middle, and end. Whether and how this linear temporality structures the image-material in a particular film is a major issue (perhaps the major issue) for a filmmaker. It could even be argued that the stance taken by a filmmaker towards temporal structure, how time is articulated in a particular film, is an index of where in the spectrum of cinematic practice (from Hollywood to experimental) a given work falls.
Duration. This gives us the first sense of cinematic time, duration or running time: the clock-time required for the filmstrip to traverse the projection apparatus.
Story Time. The second sense of time is that of the world depicted in the film. This notion is more or less restricted to the cinema of story-telling, and the time depicted in ‘diegesis’ or depiction. Mieke Bal, in Narratology,5 calls this the time of the fabula or fictional world imagined in the story. A cinema narrative may jump forward, eliminating decades (or centuries) in a single cut; or slide back, perhaps using one of the various narrative strategies that fall under the category of flashback; or remain in the present, so that a given passage of film denotes a continuous passage of time. This latter case (the most frequently used) still allows for a broad range of variation: one continuous two minute portion of a story can occupy five minutes of screen time, while the next portion compresses seven years into as many seconds.
Experienced Time. Detective Arbogast's walk up the stairs seems painfully extended, so that his stabbing at the first landing is a dreadful shock. This shock puts us on edge for Lila's later descent into the basement, which stretches time even further as each step seems to last a full minute, since we now (rightly) expect the worst to be waiting for her. On one other end of the scale, a contemporary action film can make us feel as if no time passes during its 100 minutes; while in another discourse entirely, a film like Wavelength (Michael Snow, 1968) insists on equating duration and depicted time, a position transformed into an ideology in the 1970s by such 'structural/materialist' filmmakers as Peter Gidal. The relationship of film grammar, plot, and the experience of time is a fertile area of study, especially since the compression of time is probably one of the major determinants of the longevity of cinema as a cultural phenomenon.
The question is: what happens to cinematic time when viewer input becomes a component of the screen amalgam? To what extent does the incorporation of viewer impact keep time real, canceling out the magnetism of cinema itself—when does it cease to be cinema and become "multimedia" in its drab information-delivery costume, the slick transmission of data in fields of "hot spots," "buttons," and "point-and-click menus?" As I revise this essay (2002), this wasteland has become transmogrified into the paralytic gloom of the World Wide Web, and most people associate interactivity with clicking though endless linked Oracle database fields.
The Kuleshov Effect The temporal grammar of classical film continuity can be summed up in a single example, which, like many cinema myths, is often described but never seen. The "Kuleshov Effect" scenario consists of a close- up of a Russian actor, intercut with several emotion-laden images (a dead woman in a coffin, a child playing, a bowl of soup). This supposedly produces in the audience an apprehension of the actor's face as saturated with appropriate emotion. But more interesting than this (presumably a commentary on the actor's ambiguous, doleful expression) is the idea, taken as "obvious and certain" by Pudovkin, that the character is seen to be "looking at the soup"—the man and the soup are linked, across the cut, into a single continuous space. Of course it cannot be as simple as this, as the sense of continuous space requires the support of a number of factors such as eye-line, lighting, shadow direction, etc.—but the point is clear. I'll suggest a recasting of this fable later in this paper. Here I introduce it only to restate the homily that in cinema spatial unification is easily maintained through temporal and spatial disruptions,6 given a particular sequence. Sequence determines space. And sequence logically requires time.
The Liberation of the Filmstrip A standard linear unit of cinema has an A-B-A structure: e.g., the look / cutaway / look of a point-of-view moment, the shot-reverse-shot of a dialogue scene, or the performer-audience-performer of the musical. This atomic structure defines continuity of time and space in the cinema.
The equivalent in my interactive cinema is formed by a sequence in which the middle term is produced by an action of the viewer. If the viewer does not act, the first shot continues. But on action by the spectator the B-shot appears, then, after an appropriate period, the A-shot reappears, perhaps transformed by the interspersed shot, perhaps unchanged. In Sonata this structure is used as a bridge to an alternative point of view (for example of another fictional character, or of the author); as a jump to an earlier (or later) time in the story; as a glance at a different depiction of the narrative situation (e.g., a classical painting of the Judith and Holofernes theme rather than its continuing narration by a story-teller); or as the momentary introduction of a parallel narrative line. Does the sequence still denote a continuity of space and/or time? The interpretive mode the viewer takes toward the new material is associative. Because the new image or scene was produced, i.e. brought on screen, by the viewer, he is forced into connecting to the image it replaces—an act of association, rather than spatio-temporal suturing. In Sonata the mental act is reinforced by two strategies:
1. an automatic return to the previous image, so that it seems that the image produced by the viewer interaction is a temporary interruption of a continuing logic;
2. audio continuity: the sound from the first image continues through the interruption, which reinforces the impression that the viewer's actions are disturbing the natural flow, thus demanding that a sense be made of the new complex.
In the environment created by this structure, duration becomes variable, not fixed. Though the plot of the unfolding narrative is not affected by the viewer's interactions, the screen can now contain multiple diegetic times simultaneously, and the viewer quickly becomes accustomed to navigating between them.
Experienced Time, on the other hand, becomes open and indeterminate. At one extreme the viewer can find him- or herself in the extended time-instant of the computer hacker or videogame player compulsively acting on the screen image. It is this semi-hypnotic state that allows the computer programmer to spend twenty-four hours at a stretch in front of a CRT, "jacked in" as the novelist William Gibson puts it. In his fiction Gibson compares the state to that produced by imagined mind-altering drugs. In this mental condition, the user's impact on the screen output is paramount, while awareness of content and interpretive distance are subordinated to action. Videogame environments are often designed to stimulate this condition—the content is minimal and ancillary to the actions of the user, which are immediate and powerful, either floridly destructive and dependent on hand-eye coordination, or effortlessly navigational, most often a combination of the two. Unlike a videogame, however, changes in interactive cinema are driven by plot and consequence, and consequently compulsion will not be the overriding ingredient of the mental state of the viewer. Here the need for evaluation, interpretation and understanding are in the foreground, though the obsessive need to fully explore the narrative space can serve well as an incentive and accompaniment.
Freed from the predicament whereby the apparatus alone dictates the temporal experience, time can now expand or contract based on the extent of the viewer's involvement or attention, no longer only because of the hills, gullies, and plateaus, the changes in elevation of the plot. One can imagine the user of an interactive cinema alternating between compulsive input, loss of self in the flow of the narrative, and a sense of distance and control of one’s own experience of time, as the tides of the story ebb and flow based on one’s own actions on and in it.
The notion of suspense, for example, can be retained but transformed. If the viewer identifies with a character, seeing him as transfixed with horror at one moment and overcome with relief at the next, there may be some hesitation about accessing the cause of his distress. Now a new emotional affect, begins to emerge. "Don't look behind the door!" we wordlessly warn—but now whether the character opens the cellar door is determined by us, and the vacillation, the holding back, related to a particular experience of suspense, will put the viewer, unexpectedly, in a different grip of the screen. The new pull is a hook of agency—whether we have to face the horror that both terrifies and fascinates is now our decision, and in an effective work we will be equally compelled in both directions.