Interactive elements in movies and series

December 2, 2019

Currently, interactive elements in movies, series or TV shows mean little more than simultaneously playing a quiz on your phone, chatting with others or voting for your favorite artist. However, innovations around interactive content are taking place, which could radically transform the way we consume movies and series and, as such, enable completely new user experiences.

Our observations

  • Netflix wants to invest more in interactive storytelling. In a keynote presentation earlier this year, the streaming service’s VP of product noted Bandersnatch was a huge hit around the world and said Netflix is “doubling down” on interactive series. Bandersnatch is a spinoff episode from dystopian tech series Black Mirror and an example of an interactive “choose-your-own-adventure”, in which consumers are required to make choices for certain characters to advance the story. It is the first time a streaming platform has experimented with the format. It required a lot of heavy lifting from writers, directors and took over two years to get from the idea to the screen. Netflix needed to work closely together with the writers and directors of Black Mirror to develop the required technology to support the interactive narrative.
  • Hulu is currently the most active SVOD (i.e. “streaming video on demand”) player in VR and has put some serious effort into creating an interactive VR environment. With the Hulu VR app, users can watch movies with friends, choose their virtual surroundings (e.g. the beach or a haunted house) or watch concerts from Live Nation “on stage”. But more importantly, the company is producing a significant number of original VR series such as Door No. 1. This choose-your-own-adventure comedy makes the user the protagonist of the show and the actors speak directly into the VR camera to engage the viewer in the narrative.
  • During its yearly hardware event, Amazon announced a long list of new smart devices. Besides minor revisions of existing smart devices, it included all types of new wearables such as a smart ring, smart glasses, and earbuds. With these wearables, Amazon wants Alexa to always be around us and assist us in all our daily endeavors.

Connecting the dots

On-demand streaming of media does not only change the way movies and series are distributed, it also makes it possible to radically transform content itself. One way for streaming platforms to innovate is to add interactive elements to movies, series or TV shows.

Interactive VR movies and series are one of the innovations that could radically transform SVOD content. Last year, after the launch of his VR-project Carne y Arena, director Alejandro Iñárritu stated he approached VR movies as a completely new art, incomparable to traditional film. Viewers of Carne y Arena are allowed to experience one movie scene from different angles by walking around. This aspect of VR film limits certain cinematic techniques, in terms of storytelling (e.g. providing clues in the “right” order), but also cinematographically (e.g. providing specific perspectives), but it also enables new ones. For example, to confuse the spectator, a crime scene might contain contradictory clues on the murderer, depending on the angle and where you look in the room. In VR, every viewer will experiences his or own journey through that crime scene. Nevertheless, the downsides of VR remain significant. First of all, you need a certain amount of physical space for viewers to enjoy these experiences (e.g. enabling them to actually walk around the crime scene) and this limits the scaling options for most VR projects. Furthermore, there’s a significant chicken-and-egg problem between hardware and content that limits VR projects to a niche market; pricy headsets and poor user experience limits adoption and this, in turn, limits the development of high-quality content that would trigger the masses to buy the hardware. Not surprisingly, competitors such as Netflix have adopted a wait-and-see strategy regarding VR movies and series.

Nevertheless, Netflix is the industry-leading player in a different innovation around interactive content: interactive storytelling. As we have written before, Bandersnatch could set the trend for embedded content interfaces, which would be advantageous to Netflix as it would allow for all kinds of functionalities to be integrated into content (e.g. shoppable content, social voting, and game-like features such as puzzles). However, according to Matthew Ball of Redef, one current disadvantage of interactive storytelling is the stop-and-go interaction with the consumer. The user experience is continuously interrupted, so the participant has some time to make a choice, often binary and usually executed through a simple device such as the remote control. The induced friction might make us feel we are more in control but could also result in a loss of immersivity. A solution might be to simply reduce this friction as much as possible by using new technology, e.g. using speech instead of a remote control or applying automated interaction based on emotion detection through facial recognition or biometric sensors.

This latter application brings us to another set of innovation opportunities. A different strategic path for content innovation would be to integrate smart home devices, wearables or sensorized merchandise into the viewer experience. This could be referred to as a type of spatial interactivity whereby, in addition to interacting with the dominant screen in front of us (e.g. mobile, laptop, TV), we do so with our entire environment. We are increasingly surrounded by smart devices and wearables which are all able to communicate with each other and interact with us. Big screens will remain essential for work and entertainment. However, the “computer” is disappearing because digital technology is becoming omnipresent. Through intuitive interfaces such as voice interfaces or motion sensors we continuously communicate with computers without their clear visibility. The integration of ubiquitous computing into content production could lead to completely new user experiences. In this article, Matthew Ball presents a list of ways different devices, sensors, and technologies could be used to generate new viewing experiences. During a horror movie, connected doorbells or phones could start ringing or smart speakers could play scary sounds from different rooms. Facial recognition could be used to include our face in the storyline and heartrate monitors or eye-tracking technology could be employed to scare us at appropriate (read: unexpected) times.

For kids, creating interactive content through ubiquitous computing could establish a more natural balance between passively watching and actively playing. Sensorized merchandise such as puzzles, dolls or other toys could unlock unique experiences and engage kids to participate in the narrative. Wearables could enable more tactile interactivity with their surroundings and stimulate activity. Last, smart speakers and voice assistants could play sounds or songs, but also ask questions about a movie or guide kids through quests or educational tasks during the narrative.

Looking ahead, Bandersnatch and comparable interactive content are promising. However, in terms of movies and series, it remains to be seen how much interactive and engaging elements are actually desired by consumers, and trial-and-error is presumably the only way forward. Ultimately, these new interactive formats could provide new meaningful, personalized and more intimate viewing experiences for consumers. For streaming platforms, they offer news ways to collect data, expand media universes with new services and products and engage consumers in unprecedented ways.

Implications

  • Big tech companies with streaming platforms such as Amazon and Apple are in the best position to create interactive content through ubiquitous computing as, from hardware to software, they control the entire stack necessary to produce such cross-device viewing experiences. But at the same, their attempts could cause some backlash from movie studios, as these technologically driven innovations could clash with strong views on the art of filmmaking. For example, when Netflix announced it would allow people to speed up series, directors and actors responded with fury, claiming Netflix was ruining the art of moviemaking.
  • Interactive VR content could act as a gateway to the “metaverse”. When it comes to VR assets, Facebook is one of the industry leaders and its upcoming Facebook Horizon clearly shows the company has a strong ambition to create a metaverse. However, compared to competitors, Facebook is fairly weak in terms of intellectual property and might find it difficult to attract consumers (e.g. its previous VR environments all failed to do this). Disney, on the other hand, has built an impressive media empire. With respect to the metaverse, the possible synergy between its amusement parks, movie studios, and the direct-to-consumer streaming platform is promising. Moreover, with its amusement parks and merchandise, Disney has a long history of blending the virtual and physical. Also, regarding interactive content through ubiquitous computing, Disney has opportunities (e.g. toys and other merchandise). However, it doesn’t have the rich ecosystem of smart devices and voice assistants of big tech competitors, so it is completely dependent on others.

Series 'AI Metaphors'

×
1. The tool
Category: The object
Humans shape tools. We make them part of our body while we melt their essence with our intentions. They require some finesse to use but they never fool us or trick us. Humans use tools, tools never use humans. We are the masters determining their course, integrating them gracefully into the minutiae of our everyday lives. Immovable and unyielding, they remain reliant on our guidance, devoid of desire and intent, they remain exactly where we leave them, their functionality unchanging over time. We retain the ultimate authority, able to discard them at will or, in today's context, simply power them down. Though they may occasionally foster irritation, largely they stand steadfast, loyal allies in our daily toils. Thus we place our faith in tools, acknowledging that they are mere reflections of our own capabilities. In them, there is no entity to venerate or fault but ourselves, for they are but inert extensions of our own being, inanimate and steadfast, awaiting our command. (This paragraph was co-authored by a human.)
Read the article
×
2. The machine
Category: The object
Unlike a mere tool, the machine does not need the guidance of our hand, operating autonomously through its intricate network of gears and wheels. It achieves feats of motion that surpass the wildest human imaginations, harboring a power reminiscent of a cavalry of horses. Though it demands maintenance to replace broken parts and fix malfunctions, it mostly acts independently, allowing us to retreat and become mere observers to its diligent performance. We interact with it through buttons and handles, guiding its operations with minor adjustments and feedback as it works tirelessly. Embodying relentless purpose, laboring in a cycle of infinite repetition, the machine is a testament to human ingenuity manifested in metal and motion. (This paragraph was co-authored by a human.)
Read the article
×
3. The robot
Category: The object
There it stands, propelled by artificial limbs, boasting a torso, a pair of arms, and a lustrous metallic head. It approaches with a deliberate pace, the LED bulbs that mimic eyes fixating on me, inquiring gently if there lies any task within its capacity that it may undertake on my behalf. Whether to rid my living space of dust or to fetch me a chilled beverage, this never complaining attendant stands ready, devoid of grievances and ever-willing to assist. Its presence offers a reservoir of possibilities; a font of information to quell my curiosities, a silent companion in moments of solitude, embodying a spectrum of roles — confidant, servant, companion, and perhaps even a paramour. The modern robot, it seems, transcends categorizations, embracing a myriad of identities in its service to the contemporary individual. (This paragraph was co-authored by a human.)
Read the article
×
4. Intelligence
Category: The object
We sit together in a quiet interrogation room. My questions, varied and abundant, flow ceaselessly, weaving from abstract math problems to concrete realities of daily life, a labyrinthine inquiry designed to outsmart the ‘thing’ before me. Yet, with each probe, it responds with humanlike insight, echoing empathy and kindred spirit in its words. As the dialogue deepens, my approach softens, reverence replacing casual engagement as I ponder the appropriate pronoun for this ‘entity’ that seems to transcend its mechanical origin. It is then, in this delicate interplay of exchanging words, that an unprecedented connection takes root that stirs an intense doubt on my side, am I truly having a dia-logos? Do I encounter intelligence in front of me? (This paragraph was co-authored by a human.)
Read the article
×
5. The medium
Category: The object
When we cross a landscape by train and look outside, our gaze involuntarily sweeps across the scenery, unable to anchor on any fixed point. Our expression looks dull, and we might appear glassy-eyed, as if our eyes have lost their function. Time passes by. Then our attention diverts to the mobile in hand, and suddenly our eyes light up, energized by the visual cues of short videos, while our thumbs navigate us through the stream of content. The daze transforms, bringing a heady rush of excitement with every swipe, pulling us from a state of meditative trance to a state of eager consumption. But this flow is pierced by the sudden ring of a call, snapping us again to a different kind of focus. We plug in our earbuds, intermittently shutting our eyes, as we withdraw further from the immediate physical space, venturing into a digital auditory world. Moments pass in immersed conversation before we resurface, hanging up and rediscovering the room we've left behind. In this cycle of transitory focus, it is evident that the medium, indeed, is the message. (This paragraph was co-authored by a human.)
Read the article
×
6. The artisan
Category: The human
The razor-sharp knife rests effortlessly in one hand, while the other orchestrates with poised assurance, steering clear of the unforgiving edge. The chef moves with liquid grace, with fluid and swift movements the ingredients yield to his expertise. Each gesture flows into the next, guided by intuition honed through countless repetitions. He knows what is necessary, how the ingredients will respond to his hand and which path to follow, but the process is never exactly the same, no dish is ever truly identical. While his technique is impeccable, minute variation and the pursuit of perfection are always in play. Here, in the subtle play of steel and flesh, a master chef crafts not just a dish, but art. We're witnessing an artisan at work. (This paragraph was co-authored by a human.)
Read the article
×
7. The deficient animal
Category: The human
Once we became upright bipedal animals, humans found themselves exposed and therefore in a state of fundamental need and deficiency. However, with our hands now free and our eyes fixed on the horizon instead of the ground, we gradually evolved into handy creatures with foresight. Since then, human beings have invented roofs to keep them dry, fire to prepare their meals and weapons to eliminate their enemies. This genesis of man does not only tell us about the never-ending struggle for protection and survival, but more fundamentally about our nature as technical beings, that we are artificial by nature. From the early cave drawings, all the way to the typewriter, touchscreens, and algorithmic autocorrections, technics was there, and is here, to support us in our wondering and reasoning. Everything we see and everywhere we live is co-invented by technics, including ourselves. (This paragraph was co-authored by a human.)
Read the article
×
8. The enhanced human
Category: The human
In a lab reminiscent of Apple HQ, a figure lies down, receiving his most recent cognitive updates. He wears a sleek transparent exoskeleton, blending the dark look of Bat Man with the metallic of Iron Man. Implemented in his head, we find a brain-computer interface, enhancing his cognitive abilities. His decision making, once burdened by the human deficiency we used to call hesitation or deliberation, now takes only fractions of seconds. Negative emotions no longer fog his mind; selective neurotransmitters enhance only the positive, fostering beneficial social connections. His vision, augmented to perceive the unseen electromechanical patterns and waves hidden from conventional sight, paints a deeper picture of the world. Garbed in a suit endowed with physical augmentations, he moves with strength and agility that eclipse human norms. Nano implants prolong the inevitable process of aging, a buffer against time's relentless march to entropy. And then, as a penultimate hedge against the finite, the cryo-cabin awaits, a sanctuary to preserve his corporal frame while bequeathing his consciousness to the digital immortality of coded existence. (This paragraph was co-authored by a human.)
Read the article
×
9. The cyborg
Category: The human
A skin so soft and pure, veins pulsing with liquid electricity. This fusion of flesh and machinery, melds easily into the urban sprawl and daily life of future societies. Something otherworldly yet so comfortingly familiar, it embodies both pools of deep historical knowledge and the yet-to-be. It defies categorization, its existence unraveling established narratives. For some, its hybrid nature is a perplexing anomaly; for others, this is what we see when we look into the mirror. This is the era of the cyborg. (This paragraph was co-authored by a human.)
Read the article

About the author(s)

Economist and philosopher Sebastiaan Crul writes articles on a wide range of topics, including rule of law in digital societies, the virtualization of the lifeworld and internet culture. He is currently working on his doctoral degree on the influence of digitalization on mental health and virtue ethics, having previously published dissertations on the philosophy of play and systemic risks in the finance industry.

You may also like