Google Assistant will be the core of new hardware

Google CEO Sundar Pichai kicked off today’s keynote event by emphasizing the company’s shift in focus from “mobile first” to “AI first”, and saying it wants to create a personal Google for each user. The company demonstrated the fruit of its efforts earlier this year by giving us a taste of its Assistant bot in its Allo messaging app. But that isn’t enough. Pichai said for Assistant to be truly helpful, it needs to be available universally, and so the company today showed how it will be bringing the helper to phones and homes.

Hardware chief Rick Osterloh detailed how the company is working on hardware built around Google Assistant, starting with the new Pixel phone. Google Assistant starts listening on the new Pixel phones when you hold down on the Home button (similar to how you’d activate Google Now before), or when you say the hotword (“Ok Google”).

In a demonstration, Google Assistant brought up (and rattled off) a list of events from the Hearst Greek Theatre in response to a query about happenings there. Then, when asked to play music by The Lumineers, the helper brought up a YouTube video of the band’s music and automatically started playback. You can also ask Assistant to show you your pictures from a specific date, and it will return images that match that criteria.

Pichai said Google improved the quality of its artificial intelligence from 89.6 percent in 2014 to 93.9 percent today, and this helps the system better recognize pictures so Google Photos can more accurately find the images you’re looking for. This helps Assistant be more helpful to you. The Pixel phones come with Photos built in.

To bring the Assistant into our abodes, the company is also adding it to the Google Home smart speaker. You can use it to manage everyday tasks and play tracks from YouTube Music, as well as Spotify, Pandora and Google Play Music. In addition to the voice trigger, a touch panel at the device’s top lets you activate Assistant. You can tell the speaker to turn the volume up or down, or play specific compilations. Plus, if you don’t remember the exact title of a song, you can say something like, “Ok Google, play that Shakira song from Zootopia,” and Assistant will know you’re looking for the song “Try Everything.” That’s powered by Google’s search engine.

You can also get answers to your questions through Assistant in Home, such as what singer Adele’s real (or full) name is. It’ll understand follow-up questions, as demonstrated onstage when Assistant understood that it was still talking about Adele when asked, “How many awards has she won?”

Assistant also understands traveling and navigation information. Google showed how a query for “nearby camping stores” returned companies such as The North Face and REI in the area. Then, it provided traffic information for and the time needed to get to each of those locations.

At the start of the day, Assistant can also run down your list of upcoming activities and reminders, and does so somewhat colloquially. For instance, in a demo, it said, “By the way, remember to cook dinner for your kids tonight,” instead of something more stilted, like “You have a reminder for ‘cook dinner for your kids.’”

Assistant can also control your connected home appliances, and Google is partnering with Nest, IFTTT and a growing list of brands to integrate the service. Plus, Chromecast (and Chromecast Audio) owners can also use Home to play videos, songs and pull up specific Google Photos on your TV (or speakers).

Lead engineer for Assistant Scott Huffman said the company plans to launch an open developer platform to let anyone create actions for Assistant. It will let partners come up with so-called Direct Actions for easy, straightforward tasks such as turning on the lights. Another method is conversational actions, which creates a back-and-forth between you and Assistant to get you what you need, such as an Uber ride. More information on Actions will be revealed in December, said Huffman, but the goal is to get Assistant working with every app and platform available.

Google promises that Assistant will get better with time, but it already looks to be a solid amalgamation of all of the company’s services that you can access from basically anywhere. That is, if you have a Pixel phone or Google Home.

Click here to catch all the latest news from Google’s fall event.