IE 11 is not supported. For an optimal experience visit our site on another browser.

Kinect and Sesame Street hack the next generation of TV

\"Kinect Sesame Street TV\" and \"Kinect Nat Geo TV\" will bring interactive, immersive experiences to live action television and children's books.
\"Kinect Sesame Street TV\" and \"Kinect Nat Geo TV\" will bring interactive, immersive experiences to live action television and children's books.Microsoft

By Gregory Ferenstein

Fast Company

Microsoft has unveiled a sharp idea for the next generation of television: interactive, live-action TV shows, produced in partnership with Sesame Workshop and National Geographic.

"Kinect Sesame Street TV" and "Kinect Nat Geo TV" will use the Xbox 360 and Kinect motion controller to provide downloadable, linear episodes that run like normal television shows but also give children opportunities to play simple games with familiar characters and play around with the shows' environments.

A series of interactive children's books is also in the works. Dubbed "Project Columbia,” they allow burgeoning readers to explore the otherwise static world of picture books with games, sounds, and augmented reality.

"An analog television show is not designed as a one-on-one experience," Dr. Rosemarie Truglio, Vice President of Education and Research at Sesame Workshop, tells Fast Company.

But with an immersive environment, fluffy characters tailor the learning experience to different skill levels and invite children to explore imaginary and exotic locations.

A TV/Video Game Hybrid

Microsoft views the Kinect TV programming as a Goldielocks compromise between the relaxed experience of television and the cognitive demand of a video game.

"There are no idle states,” says Dave McCarthy, Xbox General Manager for Kids and Lifestyle Entertainment. If children are intent on being couch potatoes, the show proceeds as normal (but not without some humorous prodding to play along). Thus, children interact with the live-action, analog version of the characters, not an animated replica.

"We collaborate right at the kind of script and theme creation level," notes McCarthy. "Right on the set, we will shoot a bunch of different outcomes depending on how the child reacts."

The characters' environment, too, becomes a virtual playground, where children are encouraged to identify shapes, count objects, and discover hidden items. In the Kinect National Geographic episodes, children can explore wilderness environments as the animal itself, foraging for food in a animal-wrapped avatar that molds to their bodies.

The experience, Microsoft hopes, is intuitive enough for families who want a game-like experience without the difficulty of learning rules, dealing with frustrating win-lose states, and managing progression through an extended virtual world.

Sometimes "we just want to chill out and watch TV," says McCarthy. Given that children do still watch TV, Microsoft wants to place a product where children still naturally spend their time and will gently invite them to play with Kinect at some point during the experience.

Seriously Playful

For decades, Sesame Street's research-tested call-and-response system has been haunted by an elephant in the room: When a character like Snuffleupagus asks children how many peanuts he's eating, there's no guarantee that children actually call out the right answer.

With Kinect "If they get a wrong answer, we give them these helpful hints, so that they're getting more information,” says Truglio. So-called "scaffold learning” pairs discovery with direct instruction and hints, to give children the freedom to experiment, but not so much leeway that they quit from too many failed attempts, or inadvertently end up creating false perceptions of the world.

Additionally, as every parent knows, children seem hard-wired for kinesthetic learning. They grab, prod, slam, and toss every item they can get their tiny little hands on.

"Young children, they learn through play,” says Truglio. "They need to be able to manipulate things with their hands, they need to use their bodies." Whether it's counting objects while tossing coconuts or physically reaching out to three-sided figures to identify a triangle, the immersive experience helps imprint lessons in a way that transfers to the outside world.

Sesame Street 2.0 also hopes to prepare children for the new media world.

"Children are born into this media-saturated world," says Truglio. Yet, "a young child is often excluded from new media," she says, noting that tiny-buttoned controllers and keyboards require too much dexterity for the still-forming fingers of very young children.

Game controllers have also, by default, excluded parents, who are uncomfortable with button-mashing themselves. "Parents need an opportunity to be more playful with their children," argues Truglio, who has previously talked with Fast Company about the importance making games inviting for non-gamer parents so they can be involved in the video game playing of children.

Xbox opens up a children's book

The final launch product, still in early testing, is "Project Columbia," Xbox's interactive children's book. Unlike static words on a page, "Project Columbia" helps children understand concepts by playing with them in an immersive environment.

"We make sense of the world best when we have a reference in the world that relates to our previous experience. And, those previous experiences are deeply tied to our body," says Alex Games, Xbox Education Design Director. For instance, he says "near and far only make sense with reference to where you position yourself in the scene."

In the "Project Columbia" promotional video above, children learn superlatives and the importance of chores simultaneously, seeing a room go from messy, to messier, to messiest, and then understand the meaning of each degree by physically moving their bodies to clean up the environment.

"Kinect Sesame Street TV" and "Kinect Nat Geo TV" will be available for purchase next spring with a "season's worth of content," says McCarthy. "Project Columbia" is being targeted for the next holiday. Microsoft also plans to partner with Pixar (for a game called "Rush") and Double Fine productions (for "Happy Action Theater") for similar interactive, immersive TV experiences for the holiday season and later in 2012.

McCarthy says the Xbox team envisions using interactive technology for all sorts of traditional children's products, including toys. At E3 this year, Fast Company got a sneak peek at Kinect technology that transforms camera snapshots of physical toys into a digital version in the Kinect environment, which replicates a pretty accurate version of color, shape, and size.

Indeed, for some children, analog objects are simply broken computers, as adorably demonstrated by this child, who treats the magazine like a broken iPad:

For more from Fast Company, check out these stories:

Follow Greg Ferenstein on Twitter, Google+, or Facebook. Also, follow Fast Company on Twitter.