Over the past year or so, Google has explored a number of Gemini-branded and other AI features in its consumer-facing apps. Here's everything that's been announced and when they might be available.
Pixel
At the end of Made by Google 2023, a Zoom Enhance was teased for the Pixel 8 Pro that “intelligently fills in the gaps between pixels and predicts finer details.” Leveraging an on-device “custom generative AI image model,” Google calls it useful when you forget to zoom.
It is an incredible application of creative AI, which opens up a range of possibilities for creating and editing your photos. So the kind of zoom you used to zoom in on in science fiction—it's in the phone in your hand.
In October, Google said it was “coming later.” After three Pixel feature drops, its arrival is yet to come. It's unclear if the model Google is referring to is the Gemini Nano with multimodality. At this point, it may also debut with the Pixel 9 Pro as the phone's headlining photo feature.
Google Home
In the Google Home app, creative AI will be used to summarize events into a “smooth view of what happened recently.” It will use “quick and easy summary” bullet points, while you'll also be able to interactively “ask your house” to find video history clips and get automation. “Experiential features” are coming to Nest Aware subscribers in 2024.
Fit bit
Fitbit Labs will let Fitbit Premium users test and provide feedback on experimental AI capabilities.
One such feature is a chatbot that lets you ask questions about your Fitbit data in a natural and conversational way. This “personalized coaching” tailored to fitness goals aims to generate “actionable messages and guidance,” responses that can include customized charts.
- “For example, you can take a closer look at how many active zone minutes (AZMs) you get and correlate that with how restorative your sleep is.”
- “…the model may be able to analyze your sleep patterns and variations in sleep quality, and then make recommendations about how you can change your exercise intensity based on those insights. can do.”
Behind the scenes, this is powered by a new Personal Health LLM from Fitbit and Google research built on Gemini. As of March, it's coming “later this year” to “a limited number of Android users enrolled in the Fitbit Labs program in the Fitbit mobile app.”
Google Photos
Ask Photos lets you ask questions about photos and videos in your library. In addition to searching images, it can extract information and give you text responses. Powered by Gemini, example questions include “Show me the best picture of every national park I've seen” and “What themes should we have for Lena's birthday parties?” It can be used to “suggest top photos” and create captions for them. While Ask Photos is an “experimental feature” that will launch soon, Google is already teasing more capabilities in the future.
Gmail + Google Workspace
In Gmail for Android and iOS, you'll find a Gemini button in the top-right corner that lets you bring up the mobile equivalent of a side panel to enter full text. Gmail is also getting contextual smart replies that offer more customized, detailed, and nuanced suggestions. It will be introduced to Workspace Labs in July.
At CloudNext 2024 in April, Google also explored the ability for voice prompting to help me write in mobile Gmail. Meanwhile, an “Instant Polish” feature will “turn a rough note into a full email with one click.”
On the desktop web, the side panel is available in Gmail, Google Drive, and Docs/Sheets/Slide. Gemini is coming to Google Chat to summarize the next conversation and answer questions.
Google Maps
Back in February, Google announced that Maps would use LLMs to power an “Ask about” chatbot. You can use this to find locations that match your prompts for follow-up questions. It's powered by over 250 million locations and details about user-submitted photos, videos and reviews.
Chrome
Gemini Nano is coming to desktop Chrome to power browser features like “Help me write.” It should be available on most modern laptops and desktops.
Google search
In addition to launching the AI ​​review, Google previewed several upcoming features that are first coming to Search Labs:
- You will be able to take the original AI review and make it “easy” (just a few sentences) or “breakdown” (long answer).
- Multiple-argument reasoning capabilities allow you to ask a complex question all at once instead of breaking it down into multiple questions.
- Food and travel planning
- AI-powered search results page
- Video Search: Record a video and ask questions about it.
Android
Gemini Nano Pixel with multimodality “later this year” and power features like on-device/offline talkback description, and real-time scam alerts that listen for telltale call patterns. Google will share more details later this year.
At I/O 2024, Google also previewed that Gemini on Android will soon become an overlay panel instead of opening a full-screen UI to display results. In addition to preserving context, it will let you drag and drop a generated image into a conversation. For Gemini Advanced subscribers, the “Ask this video” and “Ask this PDF” buttons will view Gemini Digest videos and documents, respectively. It's starting “over the next few months.” Additionally, Dynamic Suggestions will use Gemini Nano with multimodality to understand what's on your screen:
For example, if you activate Gemini while discussing pickleball, suggestions might include “Find a pickleball club near me” and “Pickleball rules for beginners.”
Another addition that will be especially useful on mobile are the Gemini extensions for Google Calendar, Tasks, and Keep. This will let you take a snapshot of a page with multiple upcoming dates that Gemini can turn into calendar events. In the coming months, a “utilities” mobile app will give Gemini access to Android's Clock app.
We're also looking forward to the arrival of Mobile Gemini on Pixel tablets this summer.
Gemini
Live will let you have a two-way conversation with Gemini. To make the experience more natural, Gemini will return short responses that you can interrupt to add new information or ask for clarification. With 10 different voices to choose from, Google envisions Gemini Live as helpful for interview preparation or speech practice. It will be available to Gemini Advanced members in the “coming months.”
“Later this year,” Gemini Live will let you launch a live camera mode. Just point to something in the real world and ask a question about it. This project is powered by Astra.
Gems are customized versions of Gemini that let you have a “gym buddy, sous chef, coding partner or creative writing mentor.” Gemini Advanced members will be able to create customizations, while all users will have access to pre-built gems, such as the Learning Coach.
Just describe what you want your mini to do and how you want them to respond — like “You're my running coach, give me a daily running plan and be positive, encouraging and encouraging.” Gemini will follow these instructions and, with one click, add to them to create a gem that meets your specific needs.
Gemini Advanced users will also get an “in-depth planner” that goes beyond just suggesting activities but actually creates a detailed itinerary with travel times and stops, as well as people's interests. Gemini will use Gmail to search for flight/trip details, Google Maps recommendations for restaurants and museums near your hotel, and other activities.
FTC: We use auto-affiliate links to generate income. More.