Smartphone Artificial Intelligence Finds Its Calling in Cameras

Huawei has managed to trump its smartphone competitors in a world’s first by announcing the P20 Pro, a smartphone with triple rear cameras co-engineered with Leica, packing in a 20MP monochrome, a 40MP color, and an 8MP 3x telephoto lenses into a single advanced module. While this is an impressive lineup of optical sensors, it is the use of Artificial Intelligence (AI) to power the cameras that create a differentiated User Experience (UX).

Huawei’s new smartphone launch takes the use of AI in cameras to a whole new level. The camera has long been a point of differentiation in the smartphone market, but the latest implementation in the P20 Pro goes way beyond any general mega-pixel count, providing excellent results through super slow motion, powerful zoom, and adjustments for low light/night conditions. The use of AI for powering assisted compositions across a range of photographic situations also provides a quite remarkable intelligent camera experience that has the ability to completely transform photography for the average user. Feeding the AI processor tens of millions of images and adding learning from professional photographers into the device allows for an automated experience that provides a series of enhanced features such as a predictive focus on moving objects, automatically switching the camera to select correct shot modes, and long exposure stabilization.

It is worth noting that, unlike Huawei P10, where the company accommodates both training and inference functions of AI in the device core Computer Processing Unit (CPU), Huawei P20 Pro comes with a dedicated Network Processing Unit (NPU) dedicated to AI functionality. The NPU performs the AI interference and some thin training function while the bulk of the training function is delegated to the cloud. This is mainly to preserve the battery life of the device while enabling a more efficient execution of AI functions.

As a result, the latest AI implementation from Huawei provides an impressive photographic experience, but it is not the only vendor to add this level of functionality to its lineup. Many Original Equipment Manufacturers (OEMs) now use AI as a highlight feature on their smartphones, through dedicated hardware and/or software implementations. Some major flagship devices contain own-built "neural engine" chips, notably those from Apple (as part of its A11 Bionic chip) and Samsung (with the Exynos 9810), while other chipset companies such as Qualcomm are also building AI engines into their offerings. Aside from the ability to enhance certain features and functionalities, the chief reason for adding a dedicated AI chip allows the device to securely process certain functions—notably inference—while relieving the strain on the main processor and other components. This creates better power efficiencies while also building learning in the device that allows the smart enhancement of main features.

From an application perspective, the major drive behind the use of AI platforms in smartphones is to handle images (animation and photography) and voice recognition, both of which have become highly important points of differentiation. Over the past year Apple, Google, Samsung, Huawei, and LG have all introduced AI, albeit using slightly differing approaches, which has enabled features ranging from the creation of better photographs and smart organization of images to enhanced facial recognition and the recording of animated 3D emojis. Also tapping into the AI are voice and visual search engines. Voice assistants such as Apple’s Siri, Samsung’s Bixby, Google Assistant, and Amazon’s Alexa have all benefited from the use of AI to provide more “human-like” two-way interactions, while visual search tools such as Bixby Vision and Google Lens allow users to act on image detection and identification through the camera.

These implementations of AI allow for the creation of “smarter” smartphones as they are capable of providing an enhanced and advanced user experience, while also enabling power efficiencies within the device. However, it is the ability for these new AI features to perform effectively and to work quickly that are of the utmost importance and will ultimately determine their success. For this to be achieved it is imperative that AI is built into the device as this approach will improve processing speeds and efficiency rather than being accessed through the cloud, which has the effect of limiting how quickly information can be processed effecting the UX.

In the short term, there is expected to be a continued focus by smartphone vendors to create their own AI chips. Prioritizing AI alongside other key smartphone features is going to provide a key point of differentiation for vendors, as witnessed by the latest Huawei launch. Such an approach will offer innovative user interfaces and experiences that seek to breathe new life into the saturated market to reinvigorate sales and maintain margins.

While Huawei may have stolen a march on its competitors with its latest AI implementation in the P20 Pro, it is likely to be short-lived, as more will arrive later this year from Samsung, Apple, and Google as they upgrade their flagship smartphones. Indeed, Huawei is also likely to provide another step up in AI in 2018 with its new Mate device. It is expected that the focus for AI-based features on these devices will still be around voice and imaging (including facial recognition and other biometrics) but it will be the ability to convert the use of AI into a near-seamless upgrade in features and the UX that will provide key points of differentiation. Moreover, as AI is still in its nascent stage, it is important that any such launches do continue to show some tangible benefits to the end user rather than just being used as a marketing exercise to present immature AI experiences that are far from ready for market.

In the longer term, the smartphone will undoubtedly be rooted in advancements of AI and machine learning, both of which are expected to achieve more widespread adoption. However, for this to happen, the use of AI chips needs to drive down the smartphone price tiers. This will require companies such as ARM and Qualcomm to be more heavily involved in AI implementations, while the major vendors need to start working toward creating dedicated chips for the mid- to low-end sectors of their portfolios. Owing to this expected trend in smartphone AI attach rates, ABI Research believes there will be around 60 million AI-enabled smartphones in circulation in 2018, and this is expected to grow beyond 1 billion by 2022.

Immersive experiences in smartphones are expected to continue evolving, based on touch and touchless User Interfaces (UIs), with many new interfaces developing with voice, AI, Virtual Reality (VR), Augmented Reality (AR), and gesture coming to the fore. In this era, mobile phones will be an extended part of the ecosystem through which many applications and services will be accessed via immersive UIs, driven in no small part by AI. Indeed, use of AI will ensure that a whole host of inputs and gestures on the smartphone can be accurately tracked and interpreted so that individual usage patterns and trends can be recognized. This will provide the end user with a greatly enhanced UX and extremely personalized assistant, taking the industry toward a “post-smartphone era” of new ecosystems and experiences.