O
ver the last few years, two, linked aspects of the virtual human concept have been of particular interest to me: virtual personas and digital immortals.
Virtual Personas
Virtual personas are virtual humans based around a specific person and are increasingly being referred to as Digital Twins. Whilst that person could be an artificial person, such as a virtual influencer, my real interest, and probably the more challenging task, is the creation of personas based on real, living (not historical) people.
The reasons for wanting to create such a persona are driven by really mundane needs, such as how to keep corporate memory within an organisation, particularly if people are leaving or retiring early, or the nature of the job makes it hard to get in contact with previous post-holders. Wouldn’t it be nice if you could just have a quick chat with the previous people to do your job, asking for their views and opinions, mining their knowledge and experience, even if they were now working elsewhere, or for someone else, or retired—or you’re just too embarrassed to call them?
Digital Immortals
Digital Immortals are what happens when a virtual persona is under the control of its Subject (the person it is based on) and doesn’t just encompass their knowledge and experience from one job, but from their whole life. If the software could be developed to create such a persona, then, in many ways, that persona becomes little more than a hosting plan!
In considering a virtual human (VH) I typically think about two elements: the body and the mind.
The Body
The body is the manifestation of that VH in the “real” world, most probably as a digital avatar (such as most of the virtual influencers featured here on VirtualHumans.org, but probably not quite as pretty!)... possibly just as lines of text on Facebook Messenger, or WhatsApp, or, at the other extreme, as a Sophia-type robot or android.
Of course, there is no reason a VH should be limited to any one representation at a time. Speech is a key component of many forms of manifestation, and whilst Virtual Influencers can leverage pre-generated speech if necessary (and also make use of pre-generated animations), the VH’s I’m interested in need to be capable of dynamic real-time speech (and animation, gesture and expression) on any topic. This area of research seems to be developing more slowly than the pre-recorded and avatar aspects, although there is some neat work being done by applications such as Lyrebird.ai, in support of those who have lost the power of speech to progress this, and in “sandbox” type games.
The Mind
The mind is the software that drives the VH. To what extent such software needs to copy or mirror the way the brain and human mind work is one of active debate. For me, the key functional areas needed are:
- The ability to sense and attend (including digital as well as “physical” senses)
- Conversation (i.e. natural language recognition, understanding and generation)
- Needs, Goal and Task Management – possibly with a broad base like Maslow’s Hierarchy of Needs
- Memory (semantic, procedural and episodic)
- Emotion and Mood
- The ability to learn
- The ability to reason
- A long-term meta-management/meta-cognition function (which is also linked to personality)
These core architecture areas of a VH are shown below, and are all described in more detail in my book on Virtual Humans (www.virtualhumans.ai).
In our work creating Virtual Personas, we have examined how well current technologies let us create a copy of a real person. One particular area we have focused on is how we can use knowledge graph and semantic triples techniques in order to encode what a person knows (semantic and procedural knowledge) and has experienced (episodic knowledge). Here’s part of a typical knowledge graph:
In semantic triples, every piece of information is encoded as a subject (e.g. table), a verb (e.g. madeOf) and an object (e.g. wood). Those subjects and objects then sit in a vast graph linked by the verbs to each other, creating rich and complex paths between topics.
We discovered a key advantage of this approach was the ability to create far more human-like conversations than most chatbots. A typical chatbot receives a question, gives an answer (and maybe suggestions for further questions), then waits for another question. Very few human conversations are like that! In human conversation, you typically ask someone something, or make a statement, then they reply. They then proceed to talk until they come to the end of what they have to say on the subject, or, based on verbal or non-verbal cues, recognises you have heard enough and either want to move on or refocus the conversation on something more interesting that you’ve mentioned. That’s how “Virtual Barry”, our prototype Virtual Persona (and based on a real person called Barry), works.
Virtual Barry
When asked a question, Virtual Barry finds the best answer in the knowledge graph, but then if the user doesn’t quickly respond he gives a fact formed by a neighbouring triple in the graph, and then keeps responding with more information gradually drifting away from the original topic and onto related information. If the user then asks a new question then Virtual Barry starts the process again, with the new question as the focus. We’ve even shown how biometric and expression cues can be used to inform Virtual Barry as to when to move on!
In one study we did, people who knew the real Barry rated the Virtual Barry as having a 43% similarity to the real Barry when conversation and avatar (generated using the simple Sitepal system) were considered. Whilst I’d caution about reading too much into a single number, it was certainly higher than we’d thought to achieve at the start of the project.
Read the abstract from that study here, or skip to the next paragraph: This paper presents the preliminary findings from a study that created a Virtual Persona which models an individual’s personality traits. This was evaluated, and the findings indicate that users engaged with the Virtual Persona but sometimes held ambivalent views towards it. Trust in the technology, and in the accuracy and appropriateness of answers it provided, underpinned the themes relating to the level of engagement, sense of the uncanny, and sentience and expectations. It is argued that this system offers the possibility for the development of a Virtual Persona that learns post-death.
Another benefit of the knowledge graph approach was that once the Virtual Persona knew about the types of information it needs to know (events, anecdotes, people, organisations, projects, technologies etc), it could ask the real person (the Subject) to fill in the gaps, and whenever a new entity was mentioned in an answer, it would just cue that up to be asked about later. In such a way, an empty Virtual Persona could bootstrap itself, and we found that we could create a comparable Virtual Persona to Virtual Barry with a new Subject in only 40 hours, just by the Subject talking to the Virtual Persona.
When we started the project, we knew we would bump into a whole variety of ethical issues around the capture and the use of data – such as whether the company can use the Virtual Persona when a person leaves (one of the original drivers!), and whether the Subject can use their own Virtual Persona to cover for them when on holiday or sick (or even earn for them!). The big one, though, was what would happen if the Subject died.
Morality of Digital Immortality?
Whilst Virtual Barry is a pretty low-fidelity copy of the real Barry, there was, from almost day one, a very real issue of "Were Barry to die in a road traffic accident, then would his family be allowed access to, and control of, Virtual Barry?" As the fidelity of Virtual Personas increase over the coming years, then such issues are likely to become more and more pertinent. And this is not just something for a Black Mirror episode (Be Right Back), but something already in a South Korean Reality-TV show where virtual personas (well, virtual avatars) of deceased loved-ones are being recreated for a final, on-camera, goodbye. See "Meeting You" here:
What if the Virtual Persona didn’t just become a passive entity on the death of their Subject, but an active one? What if that Persona continued an active “life” reflecting the interests and wishes of the deceased subject, whether in business, social, political, cultural, financial or family affairs?
As more and more of our lives become digitised, from Facebook Messenger to Zoom chats, through online banking to gig economy working (through People-Per-Hour and the like), and romance (whether on Tinder or a virtual world like Second Life), the more feasible it becomes for our Virtual Persona to just take over our life (the novels Kiss Me First and The After Wife have interesting takes on this) post-mortem.
When I’ve carried out polls on this topic during lectures, it’s interesting that people are often far keener to have their own virtual persona living on after their death than they are to have a persona of a loved one – it seems they’d rather have a clean break than be virtually checked up on! Does this mean those Virtual Personas might end up living in their own city of the dead?
Having a spatial existence through a virtual world seems like it could be an important ingredient in “grounding” a virtual person (see Caprica), and so such a city may not be as fanciful as it sounds. Further, there is no reason why that city needs be on Earth – who better to explore a distant planet or star than a virtual city full of dead—but still “living”—scientists, explorers and artists! It's a lot cheaper than trying to ship biological people.
Whilst most of today’s Virtual Influencers seem to be the non-real-time creations of brand makers and art departments, a logical next step is for them to become more real-time so they can engage in a wider variety of activities, from live Q&A or TikTok chats to joining virtual parties and events. To be more than just a chatbot, and to build a personality that is believable and long-lived (and which perhaps even evolves), they’ll need some, or all, of the higher level functions described at the beginning of this article – they’ll need to become Virtual Personas.
Then, similar ethical issues over ownership as we had with Virtual Barry may arise, especially around the personhood of the Virtual Influencer and its own freedom of actions—shades of Willam Gibson’s Idoru or even Blade Runner. At what point does denying the Virtual Persona of a Virtual Influencer its own choices become a form of digital slavery? We are well beyond Turing Test territory here, and perhaps the test is more about how well the Persona can argue its own case, or, more tellingly, how well Virtual Personas could argue their collective case!
As for immortality? Why shouldn’t Virtual Influencers become amongst the first Digital Immortals? If they could access some of the wealth they are generating for companies and achieve some form of self-determination, then there is no reason why they couldn’t persist well beyond the goals, wishes, dreams or nightmares of their brand managers.
