‘Fantasia: Music Evolved’ and its origins in the Kinect-hacking scene

The developers at Harmonix aren’t afraid to hit the reset button if something isn’t working correctly. Chances are, strumming a plastic Stratocaster changed quite a bit before you ever even started playing “Creep” by Radiohead in Rock Band. Same goes for stepping to the beat of Lady Gaga’s “Poker Face” in Dance Central, too. That willingness to start from square one time and again? Well, it’s carried through to the developer’s latest Kinect title, Fantasia: Music Evolved, out now for Xbox 360 and Xbox One, as well. The team’s aim, seemingly regardless of project, is for whatever you’re doing in one of their titles to seem perfectly obvious and natural.

“There’s a huge willingness to throw stuff away and start over,” Fantasia‘s lead programmer Mike Fitzgerald says. “It feels like [the final product] just works, when in reality it took a long time and a ton of work to make [gameplay] invisible.”

The Police’s “Message in a Bottle” in Fantasia: Music Evolved

To do that this time around, Harmonix turned to the Kinect-hacking scene for its Disney-funded project. At the outset, the team was keeping a close eye on what garage-based developers (and likely a few rock stars) were doing with Microsoft’s do-all sensor, using its SDK as they saw fit for all manner of things. Harmonix brought in Jason Levine. He’s well-known in the Kinect community, and has done live stage performances using Redmond’s camera setup to track his body position for real-time visualizer backgrounds. He seemed like a perfect fit to consult on a game that ultimately turns you into a conductor on songs ranging from “Night on Bald Mountain” to more contemporary fare like “Royals” from Lorde.

Levine’s position-tracking input can be seen in the game: the silhouette at the bottom of the screen that reflects your motions back to you. That bit became one of the game’s core design elements, letting you see what it was the Kinect was watching you do in real-time as a sort of positive reinforcement. “It’s different from Fruit Ninja [Kinect] in that you have to manage your silhouette,” lead designer Jonathan Mintz says. Meaning, it’s getting the rhythm of your movements synced with the actions onscreen — not just swiping at fruit randomly as it flies in front of you. “We don’t care about positions; what we care about is timing,” he adds. “We let the player find a style of motion that works for them; then they listen to the music and watch the [gesture] cues to get a sense of rhythm.”

Jason Levine at New York City’s Hardware Hack Lab

The inherent problem with basing a game off of hacks, apparently, is teaching others how to use them. “If you build a tool for yourself — like a 3D DJ controller-like Kinect hack — you can perform it really well,” Mintz says, “but it’s got this really steep learning curve.” That can make it hard for anyone else to use. “It’s probably more frustrating than learning an instrument, where at least you know what fret you’re holding.” he adds. He likens it to learning a theremin, an electronic instrument that you don’t even touch for it to produce different sounds. “You have to learn how to move in space and you can get these outrageous results.”

To combat this with Fantasia, individual movements are taught to the player on a song-by-song basis until the training wheels come off and songs start getting more and more complex.

A group of French theremin players

Mintz says that while creating a hack might look impressive, making it fun is completely different. That’s where partnering with Disney has its advantages. Mintz says that Walt and Co. afforded the team “a lot” of time to get the actual game aspects of their hack right and, perhaps most importantly, to make it enjoyable. Implementing a structure that guides players through the complexities of the title at a deliberate pace before taking the training wheels off completely was paramount as well. “That’s where having the time to figure out the structure that would help as many people be able to do that as possible was really great,” Mintz says. In practice, the progression in the game feels pretty natural and after a few songs of training, the skills that make it feel like you’re behind the music control come in.

“Getting something functional on the hardware is doable, right? That’s why you see all these cool hacks out there,” he says. “Taking the time to build that into a game context where there’s a really strong design around it, where there are goals and things for the player to explore with it? That seems like the harder part.”

It’s difficult because any tech demo can be fun for five minutes, but stretching it into a 10-hour or more experience that people actually want to come back to takes work, along with, naturally, some talent and a willingness to keep exploring new avenues when older ones aren’t panning out. It takes a bit of a maturity to not have tunnel vision or get stuck on an incorrect solution to a problem, too — something forged in the hobbyist scene. If something isn’t doing what you want, you either have to find a creative way around it, or just take a step back in the project and start fresh.

“Night on Bald Mountain,” from Fantasia

In the embryonic stages, Fantasia was more like a puzzle-based point-and-click adventure, but with gesture controls. That led to an issue of trying to avoid overwhelming the player with the user interface so that he or she wouldn’t literally be flailing about, not knowing what to do next — actions that clashed with the game’s target audience of kids and families.

“It always felt to me that it was giving players a point-and-click adventure’s inventory puzzle, but the inventory was anything you could physically do in front of the camera,” Mintz says. There was much waving about in vain attempts to solve puzzles, and the feature was ultimately scrapped, but it led to Fantasia‘s 3D cursor system in the end. What’s in place now is nigh-invisible, and surprisingly intuitive.

There was even a two-handed mode at one time, where each extremity represented a cursor, and you were spreading paint around a given scene. While it might seem like a waste, these failures eventually led to the game’s final form: more or less putting you in Mickey Mouse’s wizard cap to conduct an orchestra (or pop song) — often two hands at a time, and remix music set to some pretty wild visuals.

“It’s a matter of seeing [a hack] in a game context and with a whole host of other problems,” says Fitzgerald. “Not the least of which is what will people pay you for? [laughs]“

[Image credit: AFP/Getty Images (Theremin players)]

Filed under: Gaming, Home Entertainment, HD, Microsoft

Comments