The beneficial (and frightening) implications of virtualising reality
The mingling of virtual and physical worlds promises many beneficial and entertaining applications, but there are also some truly worrisome implications of augmented reality
Clear signals of the rapid development of virtual worlds emerged over a decade ago. Then, holographic representation Hatsune Miku – a virtual singer that employs sound-synthesising technology – performed live with a human band in front of thousands of fans. The same year, Eguchi Aimi became the newest member of Japanese performance group AKB48, and the excitement turned to astonishment when it became known that Eguchi was in fact a computer-generated composite of the other band members’ facial features.
In following years, the late rapper Tupac Shakur and late pop musician Whitney Houston made virtual concert appearances. The use of likenesses of real, but deceased, artists raised intellectual property issues and pointed to emerging ethical concerns.
Then, three years ago, China’s state-owned Xinhua News Agency introduced two virtual news anchors to present the news on Xinhua’s website and social media outlets. And this year, genealogy firm MyHeritage introduced a digital tool to animate the photos of customers’ deceased family members – the effect is as intriguing as it is creepy. Ethical considerations have now become a concern for everybody.
Hybrid realities are emerging. Also, in early 2021, South Korean broadcasting introduced a new cover of the 2002 ballad I Miss You by Kim Bum Soo with the twist that Kim Kwang-seok performed the song. Kim is still a well-known South Korean folk-rock singer, but died in the 1990s. Developers had aligned artificial intelligence (AI) with Kim’s songs so that the software could render any song the way Kim would have interpreted them. If he really would have done so in this fashion is up for debate, again raising questions about whether it is permissible to make assumptions on behalf of late artists.
The fashion industry has also experimented with virtual models in recent years. Computer-generated (CG) icons Shudu Gram and Miquela Sousa have become popular on social media and are increasingly sought-after models. Sousa garnered views on YouTube with her song Not Mine. The models are virtual, but they are increasingly competing with human models. For instance, Shudu Gram was used in a campaign for singer Rhianna’s beauty brand Fenty Beauty. Meanwhile, male virtual model Blawko has promoted the fashion brand Adidas Yeezy.
An entire industry is emerging that aims to cater to the developing market of virtual models and influencers. ModelingCafe’s website promotes the company as a “CG company that specialises in modelling”.
Virtual objects also are becoming increasingly marketable services outside of gaming platforms and virtual environments. In 2019, Sony introduced virtual foods for Aibo, the robotic dog the company introduced a couple of decades ago. Dog owners can download the “food” to a digital app to then transfer the food to a real-life bowl that the robotic dog will interact with, including making movements that indicate chewing. The virtual product serves to enhance the physical product’s features and entertainment value. Virtual and physical enter a mutually beneficial marriage.
And RTFKT Studios is designing custom virtual fashion items such as footwear and jackets and selling the rights for exclusive use by the buyer. In fact, just recently the market of non-fungible tokens (NFTs) has seen exploding interest. An NFT is essentially a ledger that enables owners to certify the uniqueness and authenticity of a digital asset.
Artist Beeple’s virtual image collage Everydays – The first 5000 days sold for almost $70m at an auction by Christie’s in March 2021. Virtual elements and objects are increasingly treated the way real-world objects are valued and used.
AR battle between good and evil
Virtual and physical worlds not only exist side by side but interact with and reinforce each other. Nowhere is such a relationship more apparent, genuinely beneficial and frighteningly disturbing as in the quickly expanding space of augmented reality (AR).
Uses of AR in military, industrial and enterprise applications have been a growth market for some time. In March 2021, Microsoft offered a glimpse into the future potential of the market. Indeed, the US Army entered an agreement to spend more than $20bn on the company’s HoloLens AR headset and related software and services.
Augmented reality’s potential not only includes beneficial applications and entertaining environments, but also truly worrisome implications
Unfortunately, AR’s potential not only includes beneficial applications and entertaining environments, but also truly worrisome implications. AR will create variable interactions in the real world, and thereby establish very different worlds for users of different applications and services. Polarisation of society might become an issue in what realities individuals might be able to see and interact with.
“Knowledge management systems have struggled with cumbersome interfaces, problematic approaches to enter information into such systems, and difficult-to-find knowledge on-demand and in real time – essentially the way workers would require such information to be useful,” says Tom Davenport, professor of IT and management at Babson College, highlighting AR’s promising use in knowledge management.
“AR offers the potential to capture, transfer and provide information in efficient and effective ways though,” he adds. “Workers in the field can instantly pull up information on components as the need arises – in fact, the virtual information can overlay on physical machinery in intuitive ways. Repair and maintenance crews can easily add knowledge they acquire on construction sites to make it available to other crews for future use.”
Devil in the detail
Meanwhile, author Mark Pesce foresees a world of constant surveillance by an increasing number of AR devices. “AR systems annotate the real world that users are looking at with digital information. To do so in usable ways, the system has to be in constant interaction with the users and their environments. Therefore, the system not only requires identification of the location, but also extraction of the context a user is in,” he says.
“Providers of AR-enabled systems will be able to compile very rich and personal information that is currently out of reach for web and social media applications. Transparency of individuals’ behaviour, and even intentions, will reach into real-world interactions and experiences,” he adds.
Pesce illustrates his concern by highlighting Facebook’s attempts to develop AR glasses. He cautions that such a device, if worn by millions of users, could map the world –people and pets, objects and devices, and more – in detail that has previously been impossible.
The question then becomes who will be able to leverage, exploit and potentially misuse such information. Combining web-based interactions and locations with real-world behaviour and places would allow the creation of data representations of individual consumers to a granularity that is downright frightening.
Virtual and augmented reality will establish environments and a future that will unfold on a spectrum ranging from entertaining and potentially educational, to exploitative, even criminal
Pesce paints the potential for a very dystopian world, but also more banal considerations exist that nevertheless establish puzzling implications. Two issues stand out.
First, who will be allowed to add information to real-world locations and objects and who will be allowed to monetise such information. The considerations could become contentious. More than a decade ago – after the accident of BP’s oil rig Deepwater Horizon – artist Mark Skwarek created an app that changed the appearance of the company’s logo to an image of an oil-pipe leak. Also, what kind of private information could be attached to unaware people’s homes.
Second, as more and more companies, services and applications annotate the real world with layers of information for a wide range of use cases, individuals will experience very different worlds as they access their preferred services. Which layers will users of competing services see – a restaurant’s menu, the restaurant’s real-estate value or the restaurant owner’s criminal record?
Distinct services would create distinct worlds that users would interact with. Free applications could provide basic information, with premium, pay-for applications providing more detailed data. Different levels of financial resources would create very different real-world perspectives and opportunities.
Other applications offer a lighter aspect of future uses of AR. Artist Kaws creates virtual artwork that he uses to populate real-world spaces, such as cartoonish animals that hover over public places. Also, owners of AR artwork could allow the public to see it depending on the time of day or change the art depending on the season. Urban environments could become more exciting, educational and flexible.
Commercial applications abound, but societal and cause-driven uses exist as well.
In the end, VR and AR will establish environments and a future that will unfold on a spectrum ranging from entertaining and potentially educational, to exploitative, even criminal. Policy-makers likely will react only after the most egregious uses of such technologies will lead to public outcry. The potential of such applications – for better or for worse – will remain difficult to envision for quite some time. But a very different future is currently in the making.
Martin Schwirn is the author of Small Data, Big Disruptions: How to Spot Signals of Change and Manage Uncertainty (ISBN 9781632651921). He is also senior advisor, strategic foresight at Business Finland, helping start-ups and incumbents to find their position in tomorrow’s marketplace.
