Over the last two weeks, all of my efforts for the Ruby Bridges project have been focused on the Prologue experience. This included creating a crowd that surrounds the user, adequate audio, attaching the camera to a moving Ruby, bringing all of these animations into the same scene together, and a smooth transition from the introductory sequence to the actual experience.
The crowd building was a real technical challenge for us, and we still haven't completely nailed it down. For playtesting purposes we took the captured data we had for four figures and duplicated it into a crowd, then instantiated that crowd once the scene started. Eventually what I would like to do is use a crowd simulation to offset the animations of the figures- looking at the crowd as it is, it's very easy to spot patterns where we duplicated groups and where figures are floating above the ground plane. It would also help us create a more faithful representation of the scene; I looked at some images taken from Ruby's fist few days of school to gauge where the crowd would be harassing her along the sidewalk and how close they were to her. Based on these the crowd was most aggressive on the sidewalk around the school, but were kept away from the front doors as the school had a fence all the way around it.
For the transition into this scene, we wanted to give the user context for where they were and whose shoes they would be standing in. Upon starting the experience, the user is in a almost completely dark room listening to audio of Ruby talking about her first day of school from her perspective. Text cues come up with Ruby's name and what interview we're pulling the audio from, followed by the school, date of the event, and the location as she's talking. The scene then fades and the user reappears in front of the school.
While listening to some interviews with Ruby talking about her first day, I noticed some of these podcasts and interviews included audio of the crowds yelling at her. I was able to cut up this audio and loop the crowd yelling into the scene, along with some stock effects of neighborhood environmental noises. Tori recorded some of our classmates yelling specific phrases, such as "We don't want you here!" and "We're FOR segregation!" to add into the audio amongst the crowd. With the volume all the way up, this audio can be very chaotic and confusing. After a few moments just standing there and the headset on, I found it easy to lose track of where I was. The audio completely obscures anything in the outside world. The added chants grounds the user in the event and the time period.
On Friday, Tori and I were able to demonstrate the current version of our prototype at the ACCAD Open House. Other than the two of us and the occasional classmate, we haven't been receiving much feedback from sources outside of the Design world. We were able to get some fantastic feedback from a wide variety of people of all ages, races, and experience with virtual reality. The topic itself raised a lot of interest with those walking by, and after a quick background on who Ruby was and our intentions with the project, most were eager to see what we had.
After taking off the headset, we had a table set up with the children's book and post-it notes for guests to provide written feedback for us. We only had two written notes, but most of the guests asked questions and gave us their impressions afterwards.
One of the most frequent comments we received was "wow, it feels like you're really there! It's very immersive." I do take that with a grain of salt, especially as many of the guests were experiencing virtual reality for the first time. However, the fact that we were able to gain that reaction from so many of those who experienced a prototype with primitive forms and non-recognizable humanoid figures was very promising. Guests gave different reasons for feeling this way- the audio being powerful and negative, the crowd surrounding the user, seeing the crowd animated in VR.
Some guests cited brief dizziness during the movement as Ruby up the sidewalk. I myself experienced this when testing the prototype before the Open House. The fact that it was significant to mention after only a 3 second motion is important as we're going to be putting a longer walk and animation in the scene in the future. After the motion stopped, the users adjusted to the world. Part of this could be resolved by having guests sit for this experience- it can be disorienting to be standing while the character is moving. Though if we continue with the interactive portion of the experience, guests would ideally be standing and moving around. I have seen other solutions in VR ports of games like Skyrim where the edges of the screen are blurred on the periphery to reduce the feeling of sickness while the player is moving, and the blur fades once the player has stopped. This may be a good area to explore when we have longer animated sequences in the scene.
I had several conversations with guests who are instructors or educators, and all mentioned seeing the uses for this in the classroom.
One guest asked me if I would be working with educators in the development of this experience. Ideally, yes- this experience is meant to be implemented in the classroom, not to replace the classroom itself. It's very far in the future, but gaining feedback from instructors as to how they could best utilize this would be absolutely necessary.
Several guests commented on whether the experience would be appropriate for elementary-age students, after asking what our target audience is. To be honest, there is very little research on how kids those ages react to virtual reality. There have been studies that suggest kids ages 6-18 perceive virtual experiences to be much more "real" than adults (as discussed and referenced here), and that children ages 6-8 can create false memories after experiencing a virtual event (source). While we want to stay faithful to Ruby's account, Tori and I will have to discuss the implications of how "real" of an experience we create.
Following up on that question, another guest asked whether we had considered leaving the avatars of the characters as these robotic figures rather than assigning them race. She was interested in how the user might project onto these figures if a race was not assigned and thus change the experience for the user. I understand her point and this is a question being addressed on several studies dealing with racial bias and stereotyping- in that realm, leaving the user "colorblind" may be an interesting area to study. One such study involves changing the race of the user's avatar and observing how users of different races demonstrated bias when the avatar was different from their own race (finding it reduced explicit bias, with no impact on implicit- an interesting study to consider when we're having users experience Ruby's walk. Source). However, our purpose is to craft a world similar to that Ruby experienced to promote empathy, understanding, and connection between the student and Ruby. Race is a vital point to her story and understanding that this is just one of many moments during this time where she would encounter aggressive racism is vital to this experience.
The question of interaction was addressed when discussing the scene where the user would be able to explore the world. Guests asked what kind of interactions they might experience- would the crowd react to their presence? Would they be able to move around the scene? One guest suggested using gaze-tracking to trigger the crowd into throwing things at you when walking around the scene. In past critiques, the suggestion of having the crowd's heads all turn to follow you no matter where you are would certainly be intimidating (or even menacing).
It really comes down to what we want the user to gain from that freedom to explore. Initially it was to provide background knowledge of the event and learn more about the long-term effects/major components in the scene- how Louisiana fought her attendance, how the community reacted, what the rest of Ruby's education was like. The major question is how to go about delivering this information. Looking at perspective-taking, the user could embody different characters in the scene and listen to their internal monologue as a way of understanding different points of view. Or the user could walk around as their own avatar objectively, as if at a museum.
An Open House guest gave me a great case study for this "virtual museum" experience created by Assassin's Creed Origins. The game takes place in Ancient Egypt, and your character is part of a vast open-world environment. Ubisoft recently released a Discovery Mode for the game, featuring guided tours through landmarks and buildings. The player can run around the landscape at will as their own character. When a tour is activated, a guided trail is illuminated along with interactive checkpoints that features a narrator and extra written information/artwork added into a menu archive for later inspection.
This seems to be a great way to keep player autonomy and the general elements of gamification consistent in the game while still conveying the relevant information. I own the game and have yet to explore Discovery Mode myself, but I will be doing so this week and discussing ways to move forward with Tori.
Tori and I will be meeting this week to discuss our next steps and compiling the feedback received from the Open House. With the current course, we will likely be working on the crowd simulation and the user animation for Ruby. The current walk is very short, and we will need to work on the animation cycles (and creating an idle state) so the characters do not just stop after a three second experience. We will also be testing out model applications for the crowd and adjusting the audio.