Fusing Fiction and Technology: Literature's Leap into Augmented Reality

 

The aim of immersion in Augmented Reality (AR) is to engage users with visual and other sensory stimuli, creating a sense of being in a different reality where the physical world fades away. In the 1970s, video art can be considered an early form of AR. This era of art, highlighted by theorists like Stephen Heath, was characterized by its dependence on specific equipment defining the art form.

For instance, Bill Viola's work, "Ocean Without a Shore", showcased on James Cohen Gallery’s website, is a vivid example: a Color High-Definition video triptych using two 65″ plasma screens and one 103″ screen mounted vertically with six loudspeakers. Similarly, the SFMOMA catalog details Peter Campus' 1975 work "Dor" as a "closed-circuit color video installation," where a discrete camera films visitors, projecting their live images onto a nearby wall. These descriptions underscore that such artworks are not merely about the visual content but are intricate setups involving cables, cameras, and screens, focusing on the configuration and interaction of the technology with the viewer.

This concept can be extended to a modern era and a new medium—the networked electric image that began with live video and has evolved with Augmented Reality (AR). AR represents the first screen media of the network age. It goes beyond simple movie broadcasts, digital recordings, or even computer-generated effects. Instead, AR is fundamentally about live media manipulation, making it a unique medium.

Each new medium often evolves by adapting elements from its predecessors, a process known as remediation. AR itself remediates both film and special effects, and the next medium will likely build on AR. While the nature of this future medium is still unknown, one possibility could be an advanced form of AR that integrates computing with the human brain more seamlessly, enhancing our sensory experiences to a new level of coherence.

In her article "Video: From Technology to Medium," Yvonne Spielmann explores the evolution of video into a distinct medium. She describes how a medium develops from the advent of a new technology into a unique form of expression with its own media language and aesthetic vocabulary.

Specifically for video, Spielmann points out that once it establishes a media-specific set of expressions, it can be differentiated from existing media like television. She discusses the technical characteristics video shares with other media, how it builds on these attributes, and how video-specific images create a unique departure from other forms. These concepts are critical to understanding how Augmented Reality (AR) is also going through a process of remediation, distinguishing itself from existing media and paving the way for new modes of expression.

The concept of remediation illustrates how new media often begin by emulating the features of older media. This tendency can limit the new medium's development by anchoring it to the past, rather than exploring and amplifying its unique capabilities. To advance a medium effectively without merely replicating past forms, it's crucial to focus on innovation and the distinct possibilities it offers.

Consider this anecdote to illustrate the point: when filmmakers first started using the camera, many shot films that resembled stage plays, capturing the action from a fixed, single-point perspective. This approach initially ignored the potential of film to use multiple angles, close-ups, and edits to tell a story in a dynamic new way. It wasn't until filmmakers began experimenting with these techniques that the medium of film truly distinguished itself, offering viewers experiences that were impossible in traditional theater.

Similarly, when introducing a new medium like augmented reality or virtual reality, the challenge is not to simply transfer existing content into a new format but to rethink how this content can be experienced uniquely and innovatively. By pushing the boundaries and using the medium to do what was previously unimaginable, creators can realize its full potential and guide it toward a future where it stands on its own, distinct from the media that preceded it.

Cross-Pollination: The Flourishing of Two Shared Fields

On October 13, 1966, New York's elite and their admirers gathered at the 69th Regiment Armory on Lexington Avenue for the inaugural night of “9 Evenings: Theater and Engineering.” This event marked the first major collaboration between artists, engineers, and scientists, featuring cutting-edge technology. Among the attendees were luminaries like Andy Warhol, who praised the event, and the Beat poet Allen Ginsberg, who was approached by a young Susan Sontag. Marcel Duchamp, the pioneer of modern art, was present, perhaps reminiscing about his provocative 1913 Armory show. Emerging artist Chuck Close was also there, sitting next to Duchamp.

The event was vibrant with notable figures such as fashion designer Tiger Morse, who stood out in her glowing white vinyl outfit. Robert Rauschenberg was acknowledged as the inspiration behind the event, but John Cage, known for his composition of silence, "4’33”", stole the show. He created a sound collage live from telephones around the city, which gradually included the audience interacting with onstage appliances.

The following night, Rauschenberg presented Open Score, a unique tennis game where each racket hit triggered a loud "boing" and turned off one of the forty-eight lights, eventually plunging the venue into darkness. The performances varied from magical to chaotic, embodying an Andy Warhol aesthetic yet heavily influenced by Duchamp—from the concept of the ready-made to the iconic Campbell’s soup can.

The event was a resounding success, with the opening night selling out all 1,727 tickets and turning away 1,500 hopeful attendees. Over the course of the nine evenings, 11,000 people attended, with several nights selling out. It was clear to everyone present that they were witnessing the dawn of a groundbreaking art movement poised to revolutionize the traditional art scene. A New York Times reporter captured the significance of the event, noting that a bomb at this location would have extinguished the entire New York art scene, highlighting the concentration of influential figures gathered there.

Over the nearly five decades since the intersection of art, science, and technology first captivated audiences, their interplay has continued to produce groundbreaking results. The art that has emerged from this interaction is as varied as it is provocative—ranging from beautiful to subversive to unsettling. This fusion often pushes the boundaries of what is traditionally expected in each field. Notable artists like Picasso and Kandinsky incorporated cutting-edge scientific ideas into their art, while scientists have explored concepts such as the aesthetic appeal of scientific theories and the beauty inherent in the structure of scientific inquiry.

The concept of cross-pollination, typically associated with the natural process where pollinators like bees, birds, and bats transfer pollen between flowers to enable plant reproduction, can also apply metaphorically to the synergy between distinct disciplines like art and science. This synergy enhances the creative and functional output of both fields, similar to how pollination boosts agricultural productivity.

Augmented Reality (AR) exemplifies this modern technological synergy. While AR is a sophisticated technology, it does not exist in a vacuum. Instead, it interacts dynamically with other media forms, enhancing and being enhanced by them. This relationship is akin to the Internet, which amalgamates various forms of media (audio, video, and print) into a new medium with its own unique stylistic and formal qualities. Just as the Internet has evolved its distinct language and form, AR merges and transforms existing media, creating new interactive and immersive experiences. This blending of media not only showcases AR’s potential to innovate but also highlights its capacity to adapt and integrate traditional media into new, engaging formats. Here is one example.

In June 2016, Lucasfilm's Industrial Light & Magic Experience Lab (ILMxLAB), which was established in 2015 to design immersive experiences for VR and AR platforms, announced a partnership with Magic Leap to develop Star Wars-related content. This collaboration aims to transform mixed reality from a mere novelty into a genuine medium for filmmakers and other creators to enhance and expand universes like Star Wars. Rony Abovitz, President and CEO of Magic Leap, highlighted ongoing efforts to test these experiential storytelling moments, emphasizing their potential to add depth to existing narratives.

Magic Leap's teaser demonstration videos showcase AR technology using flat planes to simulate conventional screens for activities like viewing emails or watching movies. This approach represents a transitional phase, incorporating familiar elements from traditional media into AR to make the technology more accessible. However, this also indicates a need to rethink these traditional formats within AR’s unique context to enhance user experience significantly.

So the question is, can literature cross-pollinate with AR?

The answer to whether literature can effectively merge with augmented reality (AR) is a resounding yes. A historical example supporting this possibility is the event on October 13, 1966, which marked the first large-scale collaboration involving artists, engineers, and scientists. This pioneering gathering saw ten artists and thirty engineers using groundbreaking technology of their time, creating a foundational event that not only demonstrated the potential of such collaborations but also established a precedent.

This event not only showcased the possibilities of technological and artistic collaboration but also set a foundational model for future explorations in AR, suggesting that writers, too, can significantly contribute to and expand the narrative capabilities of augmented reality.