Welcome! On this blog, we showcase interesting stuff from the New Media Design bachelor and master tracks (Tilburg University): student projects, lecturers’ research activities, updates about our lab, …
Interactive narratives making unsung heroes heard
By Renske van Enschot
In our Master course Interactive Storytelling, students create their own strategic interactive digital narratives. In these interactive narratives, readers become interactors who can engage with the characters, choose which storyline to follow, which perspectives to explore. The design of the interactive narratives is justified by theories about the experience and effects of (interactive) narratives.
In the ’23-’24 edition of the course, the theme for all interactive digital narratives was unsung heroes. Some stories are all over the news. Daily updates on the war in Ukraine, the effects of climate change, successes at big sports event, you name it. Other stories remain hidden. Our students were asked to focus on such a hidden story. A story of an unsung hero. A person who makes a real difference in the world but has been unnoticed. Our students designed admirable interactive narratives about a wide range of unsung heroes, from social workers and caregivers to combat photographers. All their interactive narratives give these unsung heroes the spotlight that they deserve.
Two interactive narratives let you step into the shoes of a social worker. Beneath the Surface focuses on the tremendous impact of social workers on their clients. This interactive narrative lets you help a recovering drug user get back on his feet again or assist Tyler in finding a safer home, away from his abusive mom. As a social worker in this game, you’ll face real-life challenges and make critical decisions that impact the lives of the clients you encounter. Social Heroes helps you understand how social work – involving many tough challenges – affects the social worker’s emotional well-being. You will learn how to comprehend and support a loved one who works as a social worker.
In recent years, people have increasingly been diagnosed with social anxiety disorder. People have become more aware of its existence, but is this enough to recognize the signs before a person is diagnosed? What is it like to witness such signs up close? Silent Struggles is an interactive digital narrative where you get the chance to experience a day-to-day life with your partner, Jamie, who has been behaving strangely. Will you be able to discover where those symptoms may come from? How will you navigate through these struggles together?
Screenshot from ‘A Loving Home & A Hopeful Heart’ interactive narrative
In A Loving Home & A Hopeful Heart, you will step into the shoes of Eline, as she and her partner open their home and hearts to a foster child. The road ahead may be filled with challenges, but they are ready to face them head-on. This interactive narrative not only offers you a glimpse into the lives of these remarkable individuals but also sheds light on the unsung heroes and victims within the foster care system. This interactive narrative hopes to inspire adults to become foster parents, making a profound difference in the lives of children who deserve love and stability.
Discover the story of Natasja, a kindhearted person who, motivated by her personal struggles, sets out on a quest to provide comfort and support to those in need. As you progress through the interactive narrative Woven Warmth, you will realize that even the smallest actions can have a huge impact and warm the hearts of the underprivileged and homeless people.
In Pollinator Paradise, you play the role of a homeowner trying to revitalize the bee population in their neighborhood. Bees are essential to our ecosystem but under threat due to habitat loss, chemical pesticides, invasive plant species, diseases and parasites, and climate change. By interacting with this interactive narrative, you become more attentive of the importance of a bee-friendly garden.
Animal Shelter makes you aware of all the difficulties involved in running an animal shelter. In this interactive narrative, you play the role of the temporary volunteer office manager for the Hart van Brabant animal shelter. You are in charge of making sure the animal shelter has enough money, ordering supplies, and managing the day-to-day operations at the shelter.
In the interactive narrative The Saviour of Mothers, you will find out how, in the 19th century, doctor Ignaz Semmelweis discovered the cause of so many women dying after child birth. You will go on a quest yourself trying to find out what is going on. This interactive narrative is aimed at students interested in the medical field to inspire curiosity and to better the treatment of patients and potentially save lives.
Pictures of the effects of war are seen by people all over the world on a daily basis. But have you ever wondered about the person behind the camera? Frame of War is an interactive narrative that allows you to see the point of view of a combat photographer. This narrative does not only show you the dangers of entering a combat zone, but also the story behind it. Will you choose for safety and perhaps go home empty-handed, or are you brave enough to take big risks to snap a picture for the front page of a big newspaper?
New NMD Project: Unlocking the Potential of Extended Reality Technologies for Learning
By Mincheol Shin
In recent years, XR technologies, such as Virtual Reality (VR) and Augmented Reality (AR), have become integral parts of our daily lives, significantly impacting the ways we learn, interact, and perceive the world. The increasing adoption of VR and AR has particularly caught the attention of researchers and practitioners interested in enhancing educational outcomes.
Photo Credit: Jesper Aggergaard(@aggergakker)
Despite the prevalent grouping of VR and AR in discussions about learning outcomes, it is crucial to recognize their fundamental structural differences. VR provides a sense of ‘being there,’ while AR introduces an object into the existing environment. This structural contrast implies potential variations in how these technologies influence information processing, knowledge acquisition, and attitudes.
Beyond structural differences, various unexplored factors may influence the learning outcomes of XR technologies. Content factors such as virtuality and emotional valence, coupled with individual factors like age, gender, personality, and cultural differences, could moderate the impact of these technologies on learning.
In light of these challenges and opportunities, Dr. Mincheol Shin will lead the new starter grant research project funded by the Dutch government, aiming to uncover how recent XR technologies influence information processing and knowledge acquisition. The project will receive support from experienced researchers in the Department of Communication and Cognition, including multiple faculty members in the NMD track: Dr. ir. Nynke van der Laan, Dr. David Peeters, Dr. Alwin de Rooij, Dr. Jan de Wit, and Dr. Tessa van Leeuwen. Additionally, a postdoctoral researcher will soon join the team.
Utilizing a mixed-methods approach involving experiments and in-depth interviews, alongside psychophysiology and neuropsychology measures, we hope to provide valuable insights into the educational potential of VR and AR. As XR technologies continue to shape our educational landscape, understanding their nuanced effects is crucial. Our research strives to contribute to the ongoing conversation about leveraging these technologies to enhance learning outcomes.
Tackling crises with interactive narratives – an overview of our students’ creations
By Renske van Enschot
In our Master course Interactive Storytelling, students create their own strategic interactive digital narratives. In these interactive narratives, readers become interactors who can engage with the characters, choose which storyline to follow, which perspectives to explore, et cetera. The design of the interactive narratives is justified by theories about the experience and effects of (interactive) narratives.
In the ’22-’23 edition of the course, the theme for all interactive digital narratives was tackling crisis. With global and local crises looming and happening everywhere, students were asked to design an interactive narrative on how people tackle these crises. The students designed impressive interactive narratives on a diverse range of subtopics, from loneliness and depression to criminal offenses to earthquakes. How can you navigate through these crises, face the challenges, and survive? The interactive narratives below give a helping hand.
Old & forgotten is an interactive narrative about 89-year-old widower Bob. This beautifully designed interactive narrative makes you aware of the everyday loneliness prevalent among the elderly. Is there anything you can do? Try it out.
In Tired of covid, you get to experience what it would be like to have long covid yourself. How to deal with all your daily struggles while managing your energy level?
Fragile takes you on a journey of a group of friends in which one (unknown to the others) is dealing with depression. Your goal is to figure out who it is while navigating through the course of the story. You will learn how to recognize signs of mental illnesses in loved ones and how to best act in the given situation.
Hands off lets you experience a day in the life of a police officer. How would you deal with obtrusive bystanders, reckless drivers, or even aggressive hooligans? By playing this interactive game, you put yourself in the shoes of a police officer and will see the consequences of increasing violence against police and emergency workers.
Folding or cutting is about the drug scene. In today’s society, drugs use is being normalized more and more. But do you know the consequences of your seemingly innocent pill? The drugs world is a dark place, with many consequences for people involved. This interactive narrative gives you a first-hand experience of the story of Amir. An innocent teenager who gets stuck in to the drugs world.
In Guilty?, you play the role of a judge in a complex case about sexually transgressive behavior. You have the opportunity to experience the case from different perspectives. Throughout the court case, you are asked to form an opinion about whether a boundary has been crossed. Ultimately, you face a final judgment; is the defendant guilty or not guilty? This interactive narrative shows that the scope of transgressive behavior can be more complex than people expect.
Internationals working or studying in a foreign country face many struggles. Am I a criminal invites you to step into the shoes of an international student facing an issue that many people face in the Netherlands – a telephone scam. Which decisions are you going to make? How are you going to decide what is best for you and your future in the Netherlands?
Escape the quake prepares you for an earthquake that might happen while visiting countries like Japan, Indonesia, Turkey, and Greece. Your goal is to survive the chaos and bring yourself to safety. Do you know what to do when the earth starts to shake?
Artistic Process Futures and AI
Artificial Intelligence (AI) is rapidly becoming enmeshed in our professional and private lives. The ubiquity of this technology raises a host of ethical questions, value clashes, and unforeseen consequences that must be confronted sooner rather than later. One domain in which AI is exerting a growing influence is the arts sector. Developments such as Chat-GPT and DALL-E by OpenAI are exciting in that they present seemingly robust new capabilities in AI and creativity. However, the futures such technologies unlock are also unpredictable. Given the speed with which such technologies are emerging and becoming adopted, the need to engage target audiences to weigh in on possible AI futures in this domain is critical. Artists and other affected parties should have a voice in AI-based innovations. After all, it is in everyone’s interest to ensure that AI innovations are introduced into society in a responsible manner.
To help give artists a voice in the creation and promotion of AI tools, a research project was developed called Artistic Process Futures and AI The project was developed by an interdisciplinary team consisting of Simone Ashby and Alwin de Rooij (New Media Design Programme, Tilburg University), Julian Hanna (Culture studies, Tilburg University), Michelle Kasprzak (Willem de Kooning Academy, Hogeschool Rotterdam), and Bert-Jaap Koops and colleagues (TILT, Tilburg University). The research was generously funded by TSHD Digitalisation of Society seed money grant 2022.
Our initial phase of research consisted of two workshops that took place at the New Media Design Lab at Tilburg University in 2022-2023. During the workshops, we constructed stories about different possible, plausible, and preferable futures in which artistic work might exist.
Stage 1 was a guided brainstorm for ‘prehearsing the future’ and envisioning future scenarios, a method developed by the Brussels-based transdisciplinary collective and ‘Hybrid Reality Lab’, FoAM (fo.am). Participants engaged in: (a) mind mapping artistic process futures; (b) discussing driving forces behind art and AI, and ranking these in terms of their relative importance and uncertainty; (c) selecting critical uncertainties (e.g. artistic authenticity, power relations); and (d) creating starter scenarios offering different visions of artistic process futures.
In Stage 2, participant groups made the scenarios tangible by designing speculative artefacts (i.e., ‘design objects from the future’). Inspired by speculative design methods (e.g., work by Bleecker, Foster, Girardin and Nova, 2022) each group created and presented their speculative artefact and participated in a wider discussion on possible and preferable futures.
During Stage 3, participants co-authored a manifesto that communicates a collective vision of artistic process futures and AI. Given the role of manifestos in gathering a list of demands (or requirements), the idea was for participants to conceive of a lasting record, to be made accessible to a wider public.
The toolkit based on our methodologies will be made public in late 2023. Watch this space!
A Responsible UX Design manifesto
By Simone Ashby
As part of the Responsible UX Design course at Tilburg University, my students spent their final class session collaborating on a unified manifesto aimed at synthesizing their ideas around the core principles and values that underpin responsible UX design.
An end-to-end process
This activity was a follow-up to the starter manifestos that students generated during our very first class session together, using the Tech edition of the MANIFESTO! card game (Ashby et al., 2019). The idea was to start and end the course with meaningful engagement about the design decisions and socio-technical factors that researchers and practitioners must be attuned to for more equitable, humane, and sustainable technology innovation. The two manifesto activities also provided opportunities for students to develop and reify their own ethical frameworks, positions, and processes with respect to the design of new media products, and critically reflect on the responsibilities of UX design practitioners and researchers. As a result, students satisfied two corresponding course goals, both of which intersect with the ‘Character’ dimension of the Tilburg Educational Profile.
 |  |
Tableau vivant
Just prior the starter manifesto activity, student groups warmed up by enacting a tableau vivant depicting the MANIFESTO! tone and provocation cards they were dealt. This made for a fun ice-breaker activity, as well as providing students with an opportunity to bond with their fellow classmates and embrace the performative spirit of the manifesto.
 |  |
starter manifestos
Using the MANIFESTO! cards and game sheet, students co-authored starter manifestos, consisting of a short introduction (or preamble) and a list of principles. Again, the tone and provocation cards (along with a set of opening cards, e.g. ‘Imagine’) were meant to offer inspiration and help students write attention-getting statements of principle. Students were encouraged to consider: what’s wrong with the status quo, what needs to change, a set of goals or demands they wanted to inspire with their starter manifesto, and possible steps towards achieving the desired change.
 |   |
manifesto sprint
After exploring a variety of topics in the form of course lectures, breakaway activities, and group PechaKucha presentations, the class returned to the manifesto activity, this time with the objective of transforming their ideals and values into a single unified Responsible UX Design manifesto. To engage all of the students around this common goal, and in one location and time span, I designed this as a 90-minute round-robin sprint activity.
- Students began by voting on a tone for their manifesto: Hopeful.
- Working in groups of seven, across five * 10-minute intervals, students gathered at one of five sprint stations to develop the manifesto introduction and lists of principles (organized by theme). One person per group was randomly appointed to act as the sprint station patron and remain at the table for the duration of the writing sprint. This offered continuity across the successive sprint cycles, so that newly arriving groups did not have to puzzle over what had been discussed already for a given theme/sprint station.
- After 10 minutes, each group rotated to the next station, adding, rewriting and polishing the text that was left by the previous group until each group visited each of the five sprint stations.
Each group then nominated a student to act as ambassador to co-lead the editing and polishing of the final draft. During this final draft stage (30 minutes), the five student ambassadors worked together to combine the different parts of the manifesto (introduction, plus four sets of principles) into one unified manifesto, come up with a title, even out the tone, and add/edit/cut elements.
At the end of the session, two students read out the manifesto to the class, using a megaphone for dramatic effect. It was an exciting end to what I hope was an engaging, inspiring, and fun activity for all involved.
 |
References
Ashby, S., Hanna, J., Matos, S., Nash, C., & Faria, A. (2019). Fourth-wave HCI meets the 21st century manifesto. Proceedings of the Halfway to the Future Symposium, 1-11. [url]
Making sense of the sensory gap at the Sencity Festival
By Michel Witter
During my PhD project (begun in September 2021), I am investigating the potential of sensory augmentation technology for bridging the sensory gap between deaf and hearing people. This sensory gap occurs when sensory differences between individuals cause unrecognizable cues or an absence of relevant cues needed to support the shared experience and the mutual sharing of meanings.
At the Sencity Festival (17 September 2022), I conducted a study in order to better understand the deaf and hearing sensory gap. The method I used is based on the emBODY research tool1 which provides a way for participants to express their felt experience. The figure below shows a gender neutral depiction of a human body, where the participant can mark or highlight where they feel the current experience in/on their body. Next to the picture, there is room for the participant to offer more information about where and how this experience is felt.
In order to understand the sensory gap, participant pairs with different “abilities”, e.g., a deaf person together with a hearing person, were assigned a specific artwork at the festival and asked to go there and use the inquiry form to describe their individual experience. Each was additionally asked to express how they thought the other participant experienced the given artwork.
Most visitors who stopped by my booth at the festival responded positively to my research and its goals. People were also enthusiastic about participating in my study. The results of the study still need to be analyzed. However, I can report that despite all of my preparations, I learned to give more consideration to social dynamic behaviors at such events. For example, asynchronous visits to my booth by groups or pairs of hearing vs. deaf attendees made it hard to form pairs of people with different abilities.
Still, the Sencity Festival was a welcome opportunity for me to conduct the first study for my PhD research. I encourage everyone to visit the next Sencity event on the 25th of March in 2023, when the Sencity Festival will mark its 20th(!) anniversary. Please, take a look at www.sencity.today and follow their socials announcing the next event. While awaiting the publication of my prospective paper, more information about my research can be found at https://www.avans.nl/over-avans/nieuws-en-pers/nieuwsberichten/detail/2021/09/stimulans-onderzoek-naar-overbruggen-communicatiekloof-tussen-doven-en-niet-doven-door-sensory-augmentation.
1Nummenmaa, L., Glerean, E., Hari, R., & Hietanen, J. K. (2014). Bodily maps of emotions. Proceedings of the National Academy of Sciences, 111(2), 646–651. https://doi.org/10.1073/pnas.1321664111
Proudly presenting our students’ interactive digital narratives
By Renske van Enschot
In our Master course Interactive Storytelling, students create their own non-fiction interactive digital narratives. In these interactive narratives, readers become interactors who can engage with the characters, choose which storyline to follow, which perspectives to explore, et cetera. The design of the interactive narratives is justified by theories about the experience and effects of (interactive) narratives.
In the ’21-’22 edition of the course, the theme for all interactive digital narratives was diversity and inclusion. The students rose to the occasion and designed powerful interactive narratives on a diverse range of subtopics, from social anxiety to street intimidation to poverty. We proudly showcase some of the narratives below.
Are you sure and Eyes on me let you experience what it is like to suffer from social anxiety. How do you navigate school, friends, and party invites while dealing with anxiety?
Fading is about what it’s like to build and sustain a good relationship with a mother with evolving Alzheimer’s. Are you able to reconnect with your mother after she fails to recognize your face?
Blind spots allows the interactants to walk in the main character’s shoes to come to better understanding of how visual impairment affects people and their daily media usage. It is aimed to activate designers to create more inclusive designs for different websites and online platforms.
Between two worlds. The life of a CODA makes you realize what it is like to be a child of deaf parents.
Roll with me enables interactants to learn how to support a friend or family member in a wheelchair.
Summer camp and Two sides of the same coin focus on how poverty affects the daily lives of school children. You get to try out how to include poor children in everyday scenarios. A new face in class also aims at school children, discussing what it’s like to have a refugee in class.
It does(n’t) matter helps interactants realize what it’s like to be the victim of street intimidation and what the (limited) impact is of the choices that you make.
Open Your Eyes revolves around the increasingly common problem of discrimination at work. What it is like to work a shift as an Asian nurse in a Dutch hospital? During your day you will have some tough decisions to make, which will have a lasting effect on your stress level.
Diversity University enables you to spend a day in the shoes of an international student to see what issues they face and, more importantly, learn what you can do to make sure all nationalities feel welcome on campus.
Sparkside is all about pronouns. They can’t be seen from someone’s appearance or name, but they are crucial in expressing one’s identity. In this interactive narrative, you step into the footsteps of a teacher who tries to learn how to use pronouns correctly.
The appeal of virtual chocolate
By Nadine van der Waal
Virtual Reality (VR) is considered a useful tool within food-related applications. An example is Virtual Reality Exposure Therapy, whereby patients are repetitively exposed to virtual food objects without eating, assuming that this leads to a decrease in their food cravings or anxiety. VR could also be used as a measurement tool for food choices, for example in a Virtual Supermarket. However, it is important to know whether responses to virtual food are similar to responses to real food in order to assess the effectiveness of using VR as a tool for food-related applications. Therefore, I conducted a study — together with Nynke van der Laan, Loes Janssen, Marjolijn Antheunis, and Emiel Culleton — to examine the effects of virtual and real food (vs. non-food) cues on people’s physiological and psychological responses.
Students who participated in the experiment were exposed to four different stimuli: real food, virtual food, real non-food and virtual non-food (see image). Participants were told not to swallow during the one-minute exposure. Physiological responses to the stimuli were measured with respect to the amount of salivation, which participants had to spit out after each exposure, and which was weighed by the experimenter. As an additional psychological response measure, we elicited the participant’s level of craving on a scale from 0 to 100.
The results regarding real food cues were in line with our expectations and resemble those observed in previous studies: exposure to real food leads to stronger cravings and salivation than exposure to non-food. When looking at responses to virtual food cues, we found that psychological responses were similar, such that exposure to virtual food led to higher cravings than exposure to virtual non-food. Nevertheless, the cravings were somewhat weaker after exposure to virtual food, compared to real food. This can be explained in terms of the discrepancies between the experience of virtual food and real food, like the absence of touch and smell in VR.
However, we did not see these similarities in physiological responses between real life and VR for these appetitive stimuli. This was unexpected, as VR is generally known for the ability to elicit strong physiological fear responses, for example when one is exposed to feared stimuli (such as spiders), or arousing stimuli (such as standing on top of a building). It would be interesting to further examine whether physiological virtual food cue responses occur when adding other ‘real-world’ cues, for example smell or touch of food objects.
This study was published in Food Quality and Preference. You can read the full paper here.
Join the HCI community in The Netherlands (CHI Nederland)!
Our NMD students are trained to become research professionals in the field of human-computer interaction (sometimes also referred to as computer-human interaction, for instance in the largest conference in the field, which is abbreviated as CHI). Although there has been a professional association connecting all HCI professionals located in The Netherlands for a long time, this association had been somewhat dormant. Fortunately, several HCI enthusiasts, including our own teachers, Jan de Wit and Karin Slegers(!), have committed themselves to reviving this association. Therefore, we proudly present to you: the new and revitalised CHI Nederland!!
CHI Nederland will be increasingly present in the coming months (and hopefully in years to come!) through the organisation of events focused on themes such as eHealth, HCI education, and inclusive design, to name just a few. Everyone in HCI (both students and professionals) are warmly invited to join this community of HCI professionals in The Netherlands, or even to become active members, by joining one of the task forces that are currently working hard to shape the future of CHI Nederland.
You can find more information on the CHI Nederland website, including links to the social media channels used for communication within the community and information about upcoming events. We hope to meet many of you there!
Showcasing our students’ interactive narratives
By Renske van Enschot
In our master course Interactive Storytelling, students create their own non-fiction interactive digital narratives. In these interactive narratives, readers become interactors who can engage with the characters, choose which storyline to follow, which perspectives to explore, et cetera. The design of the interactive narratives is justified by theories about the experience and effects of (interactive) narratives. In the ’20-’21 edition of the course, the theme for all interactive digital narratives was diversity and inclusion (see our previous blogpost on this theme). The students came up with impressive interactive narratives on a diverse range of subtopics, from inequality on the work floor to the prejudices towards the LGBTQ community. We proudly showcase some of the narratives below.
Aged lets you explore the challenges that elderly people face at work or in everyday situations.
On the same subtopic, Meet May lets you step into the shoes of May, a 71 y/o grandmother who is confronted and struggles with innovative technology.
Is this your friend? is about an elementary school friendship between cultures. Interactors learn how different cultures experience and cope with diversification, and get to decide how the story ends.
Town is about implicit biases concerning gender, sexuality, disabilities, ethnicity, and visual appearance. The interactor is invited to explore different stories and reflect on his/her own biases.
In Listen up, you get to experience what it is like to have a hearing impairment on the work floor. Learn what it takes to feel included at work as a young hearing-impaired person.
Hop on a job lets children aged 11-14 yo experience different occupations. This interactive narrative hopes to reduce prejudices towards low status occupations and create awareness of the required skills and necessity of these jobs.
In We regret to inform you, you become a recruiter and discover which choices recruiters have to make, which prejudices they may have and how this affects the applicants who want to secure their place in the labour market.
And do have a look at these interactive narratives as well:
- Adoption: Dutch on the inside – Asian from the outside
- Autism: But you don’t look autistic at all
- Down syndrome: My special friend Hannah
- Dutch children’s holiday Saint Nicholas: Who’s walking on my roof?
- Gender discrimination in the workplace: Hard work pays off
- Hearing impairment: Unheard
- LGBTQ: Being allowed to love whoever you want
- LGBTQ: Class Dandelion
- LGBTQ: Sexual Diversity
- Rwandan genocide survivors: Bèkske – Rwanda Empowerment Coffee