close
close

first Drop

Com TW NOw News 2024

Apple is working with third parties, including Google, on the iPhone 16’s visual search feature
news

Apple is working with third parties, including Google, on the iPhone 16’s visual search feature

Apple’s relationship with Google as its search partner is taking a new turn with Apple’s introduction of visual search, or “Visual Intelligence,” as the iPhone maker called it Monday during the company’s “Glowtime” event. Apple already pays Alphabet about $20 billion a year to make Google the default search engine in its Safari browser. Now, iPhone 16 users can access Google’s search engine — and its visual search capabilities — with a click of the device’s new Camera Control button.

OpenAI’s ChatGPT, which will be accessible through Siri, was also shown off as a third-party partner in a demo where you could point your phone’s camera at your class notes and get help understanding the concept or problem with the press of a button.

With Camera Control, Apple explained how users can quickly take a photo or start recording video, then slide their finger over the button to frame their shot and adjust options like zoom, exposure, or depth of field in a new camera preview experience. But the button also gives iPhone 16 users access to Apple’s new “visual intelligence” search feature, which is where the partnership with Google comes in.

When the iPhone 16 first introduced, Apple’s Camera Control seemed like Apple jargon for “shutter button,” but as the event progressed, Apple explained that you can do more with this new hardware feature. With Visual Intelligence, there’s more than just an easy way to learn about what’s in your camera’s field of view; you now have another way to access third-party services without having to launch separate apps.

Essentially a visual search tool, similar to Google Lens or Pinterest Lens, Apple described Visual Intelligence as a way to instantly learn about anything you see. In a few examples, Apple showed how you could click the Camera Control button to pull up information about a restaurant you spotted while out on the town, or use the feature to identify the breed of dog you saw on your walk. The feature could also turn an event poster on a wall into a calendar entry with all the details inside.

Craig Federighi, Apple’s senior vice president of Software Engineering, casually mentioned at the time that the feature could also be used to access Google Search.

“The Camera Control is also your gateway to third-party tools, making it super quick and easy to tap into their specific domain expertise. So if you see a bike that looks exactly like the type you’re looking for, just tap Google to search for where you can buy something similar,” he said.

Image credits: Apple

The demo showed a person tapping the Camera Control button while pointing their iPhone at a bicycle, then viewing a series of similar options available for purchase in a pop-up window overlaid on the camera view. The grid of images and descriptions of the matching bikes was followed by a smaller on-screen button that read “More results from Google,” indicating that you could continue your Google search with another tap.

What Apple didn’t explain is how or when pressing the Camera Control button would signal that a third-party partner would be called in to provide an answer rather than a built-in Apple service — like Apple Maps, which was shown off in the restaurant demo. The company also didn’t fully explain how users would be able to manage or configure this feature. Instead, Federighi said, somewhat vaguely, “Of course, you always have control over when third-party tools are used.”

A Google spokesperson said the company had nothing to say about the partnership at this time. Apple did not respond to a request for comment.

What’s interesting about this feature is that it presents a new paradigm for interacting with software and services beyond what Apple provides with the iPhone. And it comes at a time when the concept of an App Store is starting to feel dated.

AI technology lets users ask questions, perform productivity tasks, be creative with images and videos, and more. These are things that consumers once turned to apps for, but can now do through a new interface of talking and texting with an AI assistant.

Rather than rushing to build its own competitor to ChatGPT, Apple is positioning itself as the platform to reach out to third-party services, including AI technologies, search services, and likely other providers in the future. What’s more, it can make these connections through behind-the-scenes deals with partners — such as its partnership with OpenAI for certain AI features — rather than tapping into transactions that happen within its apps as a way to generate revenue.

It also prevents Apple’s reputation from being damaged when a third party like ChatGPT makes mistakes (as AI often does) or when a Google search doesn’t return useful results.