Here are the slides from my final presentation:

[Show as slideshow]

And the a video of the application:
*notes: while this was taking using the simulator to make the video easier, the game it does indeed work on an iphone or ipad. also, the accelerometer and the simulator were not working, so i had to click with the mouse to move the ball of light. the accelerometer does work in the actual game

And here’s the abstract of my final paper (link to full paper at bottom):
Games are primarily about playing within, yet pushing against the limitations of a structured ruleset. This struggle with and against these constraints is what makes gameplay enjoyable. Yet games are ruled by their visual world. Our eyes have evolved to quickly and accurately detect spatial relationships, and most games rely on this perception to create a visual game world. But what about audio? Our ears, though not as accurately, can still detect spatial relationships when 3D (or surround) sound is manipulated to create soundscapes. This audio could create its own game world, and opens up doors to compelling and novel game play.

Yet the area of sound in games, let alone sound as game, is widely under-researched and under-developed. This paper and game prototype explore using audio as the interface for interaction, using 3D sound through stereo headphones and a mobile device. The mobile-device game is an effective platform for testing since the player is often already wearing headphones and seeking an escape from the world around them.
Pitch, the game prototype developed alongside of this research, has the player direct a ball of light to a hidden endpoint using aural directional cues. It creates space through sound, and encourages the player to explore that space.

While a successful first prototype, much research and experimentation still needs to occur in the world of audio games. The game as-is is compelling, but ultimately loses the player’s interest. However, the results of this initial experiment were promising. With more research, developers will be able to make interesting and novel audio games that cater to the general public.

Read the whole paper >

Some personal thoughts that didn’t work with the final paper:
Personally, I’m fairly pleased with how this project came together. First of all, I really stuck to the timeline that I set for myself, and was able to change some parts of the game based on user-testing. This was a first for me, and it absolutely made the entire project feel better-developed. Actually, I think that the most excited I’ve been about this project was when, after I had grown bored and tired of the concept after hours of development and testing, others who were playing it for the first time found it innovative and fun. I tend to second-guess myself after working on the same idea for a long period of time, and this made me realize that I should show my work to people throughout the process (even if I come to feel that the idea is boring or trite).

And now for my note on sound. I learned that I can’t do everything with this project. There’s a reason that sound designers spend years perfecting their craft – electronic sound is hard. Really hard. If I could do this project all over again, I would be in talks with a sound designer from day 1. Despite this personal knowledge gap, I really did enjoy working with sound for a final project. I learned a lot, not just about creating sound, but about how to create effective, engaging, and descriptive sound. And, I learned, that it’s an area not often studied in relation to games, which I found surprising.

This project was absolutely a solid start to my thesis work. I intend to do something mobile / web / mac app related, and just making something, anything to work on the iPhone felt good. While I used openFrameworks for this prototype, it made me want to learn Obj C / cocoa so that I could create cleaner, more engaging interactions. It got the wheels spinning in my brain about potential thesis ideas.

This week marked significant progress in the application. I was able to get the code to a point where I could have an actual user test. And get feedback! I worked really hard to get the code to this point because I wanted to actually be able to have time to implement user feedback (for the first time in my Parsons career).

Here’s a video of Nick testing it (note, I took out the audio because he was using headphones):

Some of the feedback I received from a few different people:

• sound – hypnotic, feels significant; i like how the farther away you get, the quieter it is
• somehow make sure the ball doesn’t start at the win-spot
• ball seems to have a mind of its own with the velocity – this could get frustrating
• the change between the left and the right side is too jarring as you get closer tot he end
• definitely think about adding levels
• when the sound gets closer to the win-state, somehow make it more full-bodied. change it a bit when you’re close
• “I stopped wanting to look at it.” I wasn’t really watching it anymore – watching it took my concentration away
• click to see the light; have the light normally hidden
• needs a sound (like a success ping) at the win state
• i wanted my heartbeat to match the rhythm of the sound; surprised how into it i got
• the emotional resonance is there
• double-check the delay with the ball; gets frustrating
• incorporate different worlds based solely on sound – soft, marshmallowy world; metallic world; wood world; etc.

And, my to-do list!

• incorporate loading screen, instructions screen, and win screen
• fix clipping of sound
• curate the sounds better – somehow become a sound designer in the next week (yikes!)
• add sounds to the edges of the game world
• incorporate a pause function

As of now, I have one application working where you have to move the ball by touching it, but it has SOUND. I have another application working with the accelerometer, but no sound. Now to combine them!

Outstanding tasks:

• user test, user test, user test
• get sound working with accelerometer
• implement random placement for win-area
• create / implement game screens (start, pause, end)
• SOUNDS: curate, create, noise, etc.

As I’m listening to the 3D sound working, it is certainly creating that immersive environment that I was looking for. While exciting to have work, it’s making me realize that I will need to spend a lot of time on creating and curating future sounds. I’m not a sound designer and I’m realizing that more and more every day. Keep on plugging, though!

While this past week has been spent mostly on production and development, I need to not forget about concept. I’ve been doing some research into audio games, and there’s surprisingly not a lot out there. In fact, there’s not a whole lot of research done on audio in games apart from the actual sounds themselves (very little research on concept or emotion as it relates to these sounds). It does make me bit worried about my finding appropriate sources for my paper, but, at the same time, it’s exciting (and intimidating!) to be working in an area that is largely untouched by the research community at large.

I made great progress after Tuesday’s class. I FINALLY got my developer’s license and certificate set up with my computer, xcode, and my iphone. FINALLY.

to start, i uploaded an initial prototype app to my phone. in this prototype, the user’s fingers moves the ball around the screen. sound in your left ear or in your right ear tell you where you are in relation to the sweet spot. once the user finds the sweet spot, a “win!” screen appears.

*notes: this is best played with headphones. it’s difficult to hear the 3d sound in the video, but it’s there! also, the “win” font is terrible, but it’s the only ttf font i had on my computer.

Both Mohini and Shan had excellent feedback for me. They suggested that, while it would be interesting to feel unsettled while playing the game, they often go for games that they can zone out to.

Because of the way the accelerometer works in the iPhone/iPad, they suggested that this game lends itself well to curved paths, which would further reinforce the idea of calming the player. They then suggested that I read up on zen theory and include some principle ideas from that, such as winding paths and a zen statement at the end of each level (which I will test but I feel makes the game more of a novelty).

Finally, they gave me some good suggestions as far as aesthetics, and including white noise to create a fully immersive experience.

I also met with Zach Gage, as I was having a difficult time wrapping my head around how I could actually code this. He suggested that I remove the maze entirely and test first with users merely reaching an end point with the panning sound. He mentioned that that in itself is enough of a challenge, and then to work from there in increasing difficulty through obstacles.

Finally, in talking with other classmates, they suggested a way for viewers to get hints if they need them.

I wanted to put my sketches [thanks Liza, for the rendering part!] online, as I think these are an integral part of the process.

[Show as slideshow]

And here’s the presentation I gave to the class [pretty much clarifies the above sketches]:

[Show as slideshow]

things to research:

• games for the blind
• preexisting sound-based games
• NARRATIVES

During and after our working class period, I worked through some of the user scenarios I would like to prototype. I feel like the game could go in either of two directions – an event-based installation (perhaps in a gallery or at a conference) or an ipad app. I re-sketched out the two scenarios after class (the ones in class were messy, but can be found here and here).

As of now, I’m still pretty unsure as to which direction to take this project, but hopefully some more sketching and prototyping will help me to decide.

User Scenario: iPad

User Scenario: installation kinect

User Scenario: installation wiimote or opencv

User Scenario: idea for leveling

first user test / prototype

map out a small maze on the floor using dowel rods. blindfold a player, and direct them using different sounds. figure out whether it is too disorienting to not see the maze at all first, or if they user should see the maze. to be completed this week.

I have always been interested in the relationship between senses and emotions – from sound and disgust, to touch and annoyance, to vision and joy, etc. This semester, my right, dominant arm is wrapped up in a sling, creating an irritating constraint. And I’m frustrated. And I’m angry. But ultimately, I’m persevering. The irritation of doing tasks one-handed has become somewhat of a game with me. How can I creatively open a jar? Or tie my shoes? My mind, out of necessity, has turned a frustrating challenge into a fun one. Can I relate this to an interactive environment?

I seek to visually handicap the user. More specifically, I seek to explore the relationship between sound and (dis)orientation. What emotions will be felt by the user in a blank, disorienting environment? For my final project, I propose to complete one level of a maze similar to my game proposal. Connected aurally by headphones, the user must complete a maze in which they have little to no visual cues. If they hear a sound in their left ear, they must travel left. A sound in their right, and they go right. Sounds in both mean forward. If they hit a wall, they will feel a negative cue – perhaps a vibration of some sort. Users will compete for time on leaderboards, and time will be docked if the user goes the wrong way. The final representation of the project is still undecided – will it be an immersive, large-screen interactive environment with the Kinect or Wiimote? Or a laptop in a darkened room? Also, should there be a story component to the project, or is a lack of story even more disorienting? Despite these unresolved questions, I think that the project will be an interesting exploration in connecting sound with (dis)orientation, and creating frustration, but a game-like environment by purposefully handicapping the user.

into the dark…

You tumble back into consciousness with a fit of coughing. The air is dank and tight, gripping your chest. Where are you? How did you get there? You touch your aching head – is that blood? It’s dark, too dark to see your hands in front of your face. But then, is that a sound in the distance? You start to walk toward the sound but your footsteps crowd it out. You then walk into a large… is that rock? Your deafened ears strain for the sound … and then you hear it again. It seems to be guiding you out of the cave. You start to follow the sounds.

Objective: To reach the light. This is the end of the level. Sounds ranging from ambient to intense will guide a player through a mysterious narrative to discover their condition. It will draw on aspects of Plato’s Parable of the Cave as inspiration for the discoveries the protagonist makes about their reality but also transform the visual nature of the Parable to a sonic landscape.

Game Mechanic: This is an aural version of the game Labyrinth Maze.

Paper Prototype: Player dons a blindfold. Game master stands next to player and hits the left button repeatedly when the player is supposed to go left, the right button repeatedly when the player is supposed to go right, both buttons repeatedly when the player is supposed to go forward, and no button when the player turns backwards.

Real-World Game: Using the gyroscope built into iPhone, the player must tilt the iPhone based on whether the sound is played in their left ear, right ear, or both. More specifically, when the player is on the correct path the music will play at equal volume in each ear. When the music gets stronger in one direction they must turn the iPhone in that direction to maintain the proper trajectory (e.g. if the music starts to pan left the player must turn the phone to the left as well). Players can restart from the beginning of the level by pressing a ‘clear’ button on the screen. Score is kept and rewarded for avoiding obstacles and decreased for colliding, and optionally shared with the world on a central server and website. Scores can also be posted to social networking sites. As levels get harder, there are more obstacles in the way, with narrower openings for turns. Failure to make a turn will cause the level to start over.

Moving Forward

Haitham gave some excellent feedback that the sound should actually be someone, instead of anonymous. I will pursue that angle further. In fact, I am considering adapting a level of this (and further developing the storyline) for my final project. Maybe I’m crazy, but I think it would be really interesting to pursue this combination of sound and interactivity.

Defining Game Mechanics

Miguel Sicart

This article delved incredibly deeply into the academic world of gameplay and game mechanics. For a novice, I’ll admit, I found it overwhelming, especially since I have yet to play many of the games that he mentions throughout the article.

What I did gather from it, is that game mechanics are nearly as ethereal as “play” or “game.” In fact, despite the longer article to explain it, I think Järvinen said it best with, “game mechanics are best described with verbs.” Of course, Sicart’s later, more comprehensive definition, “game mechanics are methods invoked by agents designed for interaction with the game state,” says it quite well.

I did find it interesting when Sicart said, “Game mechanics can be invoked by any agent, be that human or part of the computer system.” I had never before really thought of the computer as an engaging player in the game, capable of using game mechanics to influence outcomes. I don’t disagree with his statement, but I do tend to believe that these actions, these mechanics, are inherently human because they are a result of choice. Maybe I’m just used to my good old 2d platformers.

Meaningful Play & Designing Play

Katie Salen & Eric Zimmerman

For a n00b like me, I found this reading exceptionally helpful. We frequently talk about how all of us play games, and how just about every interaction has some basis in game mechanics. Often, however, I found this assumption troubling, as I was unaware of the spectrum we were using to define game and play. The circular chart on page 304 (game play as a subset of ludic activities as a subset of play) cleared it up for me instantly.

I really enjoyed rethinking the word play into one of its other definitions – that of the play of a car. “Play is an expression of the system, one that takes advantage of the space of possibility created from the system’s structure.” And, what really tickled me about this reading, was that they could not describe how it does this without using the root word ‘play’: “[it] is in many ways the opposite of rules… play opposes and play resists. But it does so playfully, making use of existing structures to invent new forms of expression.”

Beyond me sharing quotes that delighted me, I was interested in the concepts of paida (improvisational play) and ludus (structured play), and how they relate to transformative (or emergent) play. I recognize that there is such a thing as structured play – regulated play within a defining structure (masks vs. theater) – but I believe all forms of play must have a basis in paida. For play to be interesting to the player, there must be a form of improvisational discovery, delight, challenge, fantasy, something to heighten interest and intrigue in the play. I don’t think that paida and ludus are nearly as black and white as the reading makes them seem.

Pokewalkers, Mafia Dons, and Football Fans: Mobile Play with Me

Katie Salen

Mobile technology is a game changer (get it? ha ha ha). But it is. And it is not only because it brings the gaming world to the 5 minutes waiting in line at the checkout, but because it makes the social aspect of gaming so much easier. You don’t have to worry about scheduling times to meet up, and you can brag to all your friends on social networks. I get it. But I thought this article really lacked and important component of mobile gaming: that of one’s physical location in the world (gps).

I disliked how this article focused on wrapping the mobile world to the predefined ideals of already-existing game mechanics. It did not speak extensively to an exciting game mechanic that can only be utilized by mobile technology – that of location-based (gps) games. Yes, there has been the mechanic before of location, but on a much smaller scale. With mobile technology, we can bring gaming not just into our everyday lives, but have it influence our decisions on moving about our physical world. I find this exciting, and think that it is an important concept to explore in an article about mobile gaming.

I Have No Words & I Must Design: Toward a Critical Vocabulary in Games

Greg Costikyan

Despite his references to many games that I haven’t played, or, worse, haven’t heard of, I think that Costikyan nailed the definition of a game: “an interactive structure of endogenous meaning that requires players to struggle toward a goal.” Finally, something I can work with.

I was most interested in his section on endogenous meaning. Before, when thinking of games, I certainly understood that they were important to the people who played them, but I didn’t understand why, until the sentence, “A game’s structure creates its own meaning.” It is vitally important to the players while they are in the game, but it is fantastical in the sense that it loses its importance as soon as one leaves the game world. This augments the idea of a rule structure by giving meaning to that structure.

It was incredibly helpful for me to read about Marc LeBlanc’s taxonomy of game pleasures. Finally, I was able to understand the components beyond the ruleset that make a game “fun.” It definitely gave me some good ideas for my game for Module III.

Nature and Significance of Play as a Cultural Phenomenon
Johan Huizinga

This was one of the hardest readings to get through. Maybe it was my frame of mind, maybe it was that Huizinga’s tone was inherently more difficult to follow than the other readings, who knows. The article started out promisingly enough, by giving an excellent background on the cultural significance of play. I enjoyed his deconstruction of the seemingly simple, but ultimately complex word of “play.” And, for the first time in all of these readings, he brings up the word “tense.” “Though play as such is outside the range of good and bad, the element of tension imparts to it a certain ethical value in so far as it means a testing of the player’s prowess.” This is certainly an important element of games.

However, when he discusses religion and play for pages on end, is where he lost me. Honestly, I don’t have much to critique, other than that the religious aspect of play was not shocking, and did not necessarily relate to me on any deeper level than it existing as an astute observation. Overall, this article was helpful in cementing the necessary role of play in our anthropological history, but beyond that, seemed to get a bit ahead of itself.

Prompt: Investigate the implications of digitally augmented hybrid environments though developing your own alteration or intervention in an existing space. Support your ideas through demonstrated observation and analysis. Use a design scenario and low-fidelity prototype to communicate your ideas.

My Solution: decided to try to come up with a solution for a frustrating experience I often have with thrift store shopping – I never want to try on the clothes. So I ask, how can we use augmented reality to improve a frustrating shopping experience?

Here’s the presentation:

1: what is augmented space?

◄ Back

Next ►

Picture 1 of 12

In the critique following my presentation, this project turned from wishful thinking into an idea that might be viable. If I were to create qr codes with the measurements of the clothes standard to the vintage industry, and a mobile application to streamline the interface, it might be a very valuable application. Maybe this is something I could pursue for a final project?

We were asked to design an object that would play a critical role in our lives. I chose to explore the sense and emotion of touch, and the lack of this sense in our increasingly digital world. I devised TouchMail, a low-fidelity prototype of a plush object that you must interact with through the sense of touch before you can send an email.

While I still believe in the concept, I agree with the class that the interaction felt disconnected. There was too much back and forth between the object and the computer screen, which, going into it, I recognized, but really needed the critique to help me to brainstorm how to increase the object’s effectiveness. I really liked the idea of the object taking a photograph of the person as they’re interacting with it and automatically attaching it to the outgoing mail. I think that that would help to solve the emotional disconnect felt by the user.