Beyond the Frame: Exploring New Dimensions in AR and VR Storytelling

 

Augmented Reality (AR) is revolutionizing how we interact with art and narratives by moving beyond the traditional constraints of passive viewership. Instead of observing artworks from the outside, AR places the viewer inside the artwork itself, effectively dismantling the conventional boundaries defined by frames. This immersive approach allows participants to become part of the art, transforming their role from passive observers to active participants. The integration of AR technology means that artworks are no longer static; they are dynamic environments that respond to and engage with the viewer's actions.

This technology enriches the viewer's experience by syncing the participant's physical actions and movements with a range of sensory inputs, including visual effects, sounds, and potentially even tactile feedback. This multisensory engagement deepens the connection between the artwork and the viewer, opening up new avenues for artistic expression and interpretation. Artists can now create pieces that adapt and respond to the viewers' interactions, which can alter the narrative or visual elements of the artwork based on those interactions.

Moreover, AR introduces a novel narrative concept known as "storyworlding," which diverges significantly from traditional cinematic storytelling. In conventional films, the narrative is linear, with a clear progression of plot and character development aimed at evoking specific emotional responses. In contrast, storyworlding in AR is non-linear and interactive. It is driven by the participant's exploration and decisions within the virtual environment. This narrative form emphasizes emotional and character-driven storytelling, where the story unfolds through discovery and interaction rather than being presented in a predetermined sequence.

The potential of AR goes beyond simulating reality—it opens up new avenues for storytelling where creators can defy the laws of physics and bring the fantastical to life. Why limit AR to reproducing the real world when it can be used to craft experiences that are beyond the reach of reality? The inclusion of ghosts in AR storytelling leverages this potential brilliantly.

Ghosts, as ethereal beings that vanish and reappear, defy conventional physical laws and exist beyond the perceivable spectrum of the human eye. They naturally align with the capabilities and limitations of early AR technology.

In AR, ghosts become a thematic fit because they embody the very essence of the technology—making the invisible visible and the impossible possible. For instance, in AR games like "Ghostbusters Paranormal Blast," the imperfections of early AR tech, such as erratic tracking and computer vision, don't detract from the experience. Instead, they enhance it. The unpredictable nature of ghosts, who might float through walls or suddenly appear, meshes well with the glitchy, sometimes unstable, reality of AR. This not only preserves the magical and mysterious quality of encountering a ghost but also subtly plays into the expectation that ghosts don’t adhere to the rules of the physical world.

Moreover, AR's ability to overlay the digital onto the real world allows it to act as a portal to other dimensions. It allows users to interact with a parallel universe where the paranormal becomes normal. This isn’t just about visual trickery; it’s about expanding the narrative possibilities of AR to include the fantastical and the surreal. AR can transform our environment into a stage where the supernatural plays out—an experience that traditional media cannot replicate. There are interesting parallels with literature.

Creating imaginative worlds is crucial in literary and textual theory, beginning with Aristotle's idea that the poet's role is not merely to recount what has happened, but rather to depict what could happen under the laws of probability and necessity. This might initially seem to limit literature to only realistic representations of the world we know, bound by its physical, logical, and perhaps even psychological and economic laws.

However, this understanding of possibility doesn't need to be so narrow. By expanding our view of what is "probable and necessary," we can include a broader range of literary genres, such as fairy tales, science fiction, the fantastic, magical realism, and even the absurd, symbolic, allegorical, and dreamlike. Literature isn't just about re-arranging the familiar laws of our reality; it's about creating entirely new worlds that follow their own unique sets of rules.

While distinct, these worlds may share some characteristics with the real world, but they operate according to their own internal logic. This logic needs to be consistent enough so readers can understand what is possible within these fictional worlds, thus allowing them to appreciate the narrative and artistic necessity of the unfolding events. In essence, the writer's job is to construct compelling and internally coherent worlds that captivate our imagination by defining what can and cannot happen within their boundaries.

A key question in virtual storytelling is how audiences will choose to experience these narratives. Will they prefer to be passive observers, like ghosts who can move around the scene without interacting, or active participants who influence the story with their actions? Currently, much of the talent in Virtual Reality (VR) filmmaking comes from Hollywood and the video game industry, each bringing a different perspective to storytelling. Hollywood's approach tends to focus on observational narratives where the viewer watches events unfold, while video game creators offer interactive, player-driven stories.

Interactive storytelling allows for exploration and deeper engagement, often encouraging repeat interactions. However, the challenge with many video game narratives is that while they allow for exploration, they often lack the emotional depth and tight narrative structure found in well-crafted films and books, where every element is deliberately paced and placed.

In the future, there's likely to be a place for both styles of VR storytelling. What excites me is the potential for large-scale, socially shared narratives that combine traditional storytelling's depth and emotional impact with the interactive and exploratory elements of video games, creating a new form of immersive entertainment.