Eye tracking software can gauge your intent and boredom in VR

One of the singular things about virtual reality is the freedom to look in any direction. But that’s also one of its biggest narrative problems. How does a storyteller retain control when the viewer is free to decide where to look? The answer, it seems, is in the eyes.

Eyefluence, a company that’s rooted in optics, AI, machine learning and mechanical engineering, has built an interface that lets a user communicate with a virtual environment through sight alone. The idea is to convert looking into action. So the software enables you to use your eyes to do anything that you would do with a finger on a smartphone. No more typing, clicking, swiping or even talking. With a display in front of you, you would be able to navigate a menu, launch applications, pan, zoom and scroll, and even slip in information simply by looking.

Beyond the boost in productivity, though, one of the most compelling applications of this eye-machine interaction is in immersive storytelling. The eyes, when distracted or focused, can give away what a viewer is feeling in a moment. The Eyefluence software is designed to take advantage of those clues to know when you’re interested in a scene, captivated by a character or feeling bored.

“Your eyes are the fastest moving organs in your body,” says Eyefluence CEO and founder Jim Marggraff in a short film called The Language of Looking. The movie, embedded below, is a part of the annual Future of Storytelling summit that brings together inter-disciplinary innovators to discuss the challenges of telling stories in a digital world. Here, Marggraff explains the difficulty of immersive storytelling and how sight can be used to fire up an interface that pushes the narrative forward in virtual reality.

After watching the film, I spoke to Marggraff to find out how the eyes can have an impact on immersive storytelling in particular.

How does “the language of looking” fit into storytelling in VR?

There are a lot of problems in storytelling that need to be grappled with. I’ve sat with filmmakers and talked about the challenges that they have. [It starts with] shifting the mindset, to say we’re going to make the user a consequential participant in the story, meaning that what they do has a consequence in the arc of the story. It’s a new thinking. Typically, as a storyteller, you want complete control; you guide the [viewers’] eyes, their moods so they’re sensitive to the beats of the story as it unfolds. Essentially, every scene directs them and manages their emotions throughout. But now, by reclassifying the user as a participant, when you [let them] have consequence in the story, you give them a degree of autonomy.

“By reclassifying the user as a participant, when you let them have consequence in the story, you give them a degree of autonomy.”

Some of the known challenges with the medium are teleportation, locomotion and nausea. But more significant are some of the challenges in maintaining a sense of rhythm in the flow of the story. If I let you run off and start examining something around a corner, you don’t necessarily know how to stay engaged with that, you could put yourself in a boring position. How do we maintain that flow and the urgency in the beats of the story? It can be done. The software looks at the participant’s behavior to decide when it needs to move them along, when to deliver key points on the story level. So we know where you are, what you’re looking at, what you’re interested in and where you’ve lost interest. We can guide you back to the storyline at any time.

What is it about sight that makes it an appropriate solution for challenges in VR?

There are so many things we can do to take advantage of knowing what you see, what you’re aware of, what you’re not aware of and actually changing things around you without you even knowing. That all can happen with a deep understanding of how your eyes and your brain perceive information.

With the eyes, you can navigate in a large information space more rapidly than any thing else. Inside VR environments, for instance, where you have large amounts of information in a headset, you can look around. We give you the means to not just see a function [like a message or a browser] but activate it and move into a new space. For instance, you can search for photographs and find them more rapidly than before. It’s a mixture of purposeful and non-purposeful motions, to be able to search through a list of 1,000 names and find the one you’re looking for with your eyes only without scrolling, flicking or tapping. The eyes are the fastest moving part of your body. It’s as quick as thinking and looking. It’s quicker than even speaking to get things done.

But it’s also about what I like to call “sensuality” — it gets your senses engaged and the result is very satisfying. We’ve gotten feedback from people who say: “The system feels as if it’s reading your mind.” It’s not. It’s reading your intent and that comes from the signals. It’s a new kind of language that needs to be learned.

In the film you mention your collaboration with Rival Theory, the VR content studio that generates characters for virtual reality storylines. In what ways do the Eyefluence techniques work with these characters?

It works with characters that are in a live film or rendered, both have their beauty and challenges. Let’s consider a rendered character like [Rival Theory’s] that’s also an AI. It can have memory. It has a personality that evolves over time in relation to you as a participant, specifically with you. It knows you and the relationships you have. It comforts you in an upsetting event when you’ve cried. For example, if the character is a child who loses his best friend who slipped through a crevice while climbing and you see the child and you console them. That child character forms a connection with you based on eye interaction. The AI forms a memory of it and it can come back any time. It builds a bond between you and the character. We know the power of being able to look at someone, see when they avert your gaze, when their eyes well up in reaction to something you’ve said. This kind of connection has not existed before within the medium.

iPhone 7 Again Said to Ship With Lightning-to-3.5mm Headphone Dongle

With just over six weeks remaining until Apple is expected to unveil the iPhone 7 and iPhone 7 Plus, leaked photos and rumors surrounding the smartphones are starting to become clear and consistent.

Deutsche Bank is the latest group to add its expectations to the mix, issuing a research note obtained by Business Insider that corroborates several previously rumored features coming in this year’s refresh.

New photo of iPhone 7 and 7 Plus rear shell molds (Image: KK低调 on Weibo)
The bank, which allegedly made supply chain checks, believes the 4.7-inch model will have an improved camera and optical image stabilization, while the larger 5.5-inch model is set to gain a dual-lens camera and 3GB of RAM.

Both models are also expected to have no headphone jack, a touch-sensitive home button with haptic feedback, a new darker color, “professional class” waterproofing, and improved sound, possibly by way of stereo speakers.

Perhaps the most interesting bit is Deutsche Bank’s belief that the iPhone 7 will ship with a Lightning-to-3.5mm headphone jack dongle in the box, rather than Lightning-based EarPods as previously speculated.

Many aftermarket Lightning-to-3.5mm adapters are bulky and clunky looking due to the need for a digital-to-analog converter, however, so it remains to be seen if Apple could create a dongle that meets its high standards of design.

Every single one of Deutsche Bank’s predictions have surfaced previously:

– No 3.5mm headphone jack: WSJ, Fast Company, Mac Otakara, and many others
– Dual-lens camera for iPhone 7 Plus: Ming-Chi Kuo, Mac Otakara, and many others
– 3GB of RAM for iPhone 7 Plus: Ming-Chi Kuo
– Touch-sensitive home button: DigiTimes, Cowen and Company
– Improved waterproofing: WSJ, Fast Company, DigiTimes, Commercial Times, Weibo
– Stereo speakers: DigiTimes, Mac Otakara
– A new darker color: Mac Otakara and leaked photo
– Lightning-to-3.5mm adapter: Mac Otakara

Apple is widely expected to announce the iPhone 7 and iPhone 7 Plus in September. Other rumored features include a faster TSMC-made Apple A10 processor, repositioned antenna bands, faster LTE and Wi-Fi, a slightly larger battery, and a minimum 32GB of base storage. A larger 256GB model may also be available.

Related Roundup: iPhone 7
Discuss this article in our forums

Experiment with face swaps in a snap with this new search engine

Ah, the future. You don’t even have to use Cosmopolitan Virtual Makeover to try on different hairstyles and other looks these days even if you aren’t proficient with Photoshop. All you need is a selfie to try out curly hair, the 1930s or even a different culture.

It’s possible using Dreambit, a face-swapping search engine that automatically analyzes any photo you upload and figures out how to crop it into images you search for. Your results incorporate your face, cropped and colorized and adapted to the images you receive. It sure does beat the heck out of doing it all manually.

Created by computer vision researcher Ira Kemelmacher-Shlizerman at the University of Washington, Dreambit is an interesting tool that can and will be used for tons of silly applications, but the serious implications it has are infinite as well. In a press release, Shlizerman noted how the engine could be used in missing persons cases as it can adapt how victims in said cases can change their looks over time. As the software’s still in beta, however, it’s far from use by law enforcement just yet.

Dreambit will be on display at SIGGRAPH next week, but unfortunately it’s not available to the public just yet. You can, however, sign up for beta access right now if you’re so inclined.

Via: TechCrunch

EFF sues US government to void ‘onerous’ copyright rules

The Electronic Frontier Foundation (EFF) has sued the US government in an attempt to overturn a part of the DMCA that it says violates the First Amendment. The provisions, contained in Section 1201, restrict user access to purchased content including videos, music and the software used in cars, appliances and devices. “The First Amendment preserves our right to … research and talk about the computer code that controls so much of our world,” says EFF Staff Attorney Kit Walsh. “Section 1201 threatens ordinary people with financial ruin or even a prison sentence for exercising those freedoms.”

Section 1201 was originally enacted to combat piracy, but has instead has “served to restrict people’s ability to access, use and even speak about copyrighted materials,” the EFF says. It can inhibit, for example, a buyer’s ability to repair their own device, remix music or video and study software installed in cars, computers and other devices. Firms often sue users even when they make “fair use” of content for things like satire or criticism, a practice that’s full protected by free speech laws.

The organization is suing on behalf of two individuals well known in the tech world, inventor Andrew “bunnie” Huang (who recently worked with Edward Snowden on a cellphone signal-blocking device) and John Hopkins security researcher Matthew Green. Huang is interested in developing devices that would allow consumers to add Twitter comments, captions and other overlays to high-definition video, but fears running afoul of the law. “Section 1201 prevents the act of creation from being spontaneous,” said Huang, who added that America will soon fall behind other nations in tech if the law isn’t changed.

YouTube is a frequent target of DMCA takedown orders.

Matthew Green and his students have uncovered security flaws in vehicles, Apple’s iMessage text system and the encryption used for the web. He calls such work a “precious commodity,” but says that a lot of companies don’t feel that way, even when it ultimately benefits them. “Companies use the courts to silence researchers who have embarrassing things to say about their products, or who uncover too many of those products’ internal details,” Green said on his blog.

Working with its own counsel and a private law firm specializing in tech law, the EFF is challenging the anti-circumvention and anti-trafficking provisions contained in Section 1201. While the organization is suing the US government, it’s also likely to face the MPAA and RIAA, which represent the recording and motion picture industries in the US. Those associations are often the ones filing lawsuits and takedown notices against consumers, and have reportedly spent tens of millions of dollars lobbying in Washington.

Via: Fast Company

Source: Electronic Frontier Foundation