Night-Time ADAS Testing: A Ray of Light for FIR Image Sensors in Automotive?

Far Infrared (FIR) or thermal imaging sensors have a number of strengths and inherent capabilities that can be applied to automotive perception, but uptake of the sensor modality has been very limited to date. The primary advantage of FIR is its performance in poor lighting conditions, delivering robust detection and identification of a range of targets at night or in inclement weather contexts. FIR imaging can also contribute to sensor fusion in daylight conditions, helping to distinguish genuine pedestrians from Two-Dimensional (2D) images. However, FIR imaging also suffers from consistently high costs intrinsic to operations in the Long Wave Infrared (LWIR) spectrum.

Conventional glass has poor refractive index and transmission range in the conventional operating ranges of most thermal imaging sensors (e.g., 8 μm to 14 μm). This has historically required the use of germanium lenses, which have excellent refractive index and transmission range for 2 μm to 17 μm, but are expensive, subject to geopolitical complications in sourcing, and prone to thermal runaway at temperatures over 80⁰ Celsius (C). The poor refractive index of LWIR through conventional glass also prevents the positioning of the FIR camera behind the windscreen, creating further integration challenges.

Therefore, in spite of its advantages, FIR adoption, to date, has been slow and, in fact, largely disconnected from active safety. What limited uptake FIR has enjoyed in the automotive industry has been confined to night vision—a Human-Machine Interface (HMI) gimmick made available as a high-cost option on low-volume premium models. As a result, FIR has failed to achieve the scale necessary to bring down costs and encourage adoption in active safety contexts.

However, a renewed focus on the performance of Vulnerable Road User (VRU) detection and collision mitigation in low-light scenarios is driving FIR sensor manufacturers to aggressively position thermal imaging as a core perception technology in active safety Advanced Driver-Assistance Systems (ADAS).

The work of safety ratings agencies such as the New Car Assessment Program (NCAP) and the Insurance Institute for Highway Safety (IIHS) in driving ADAS adoption is well understood, with 5-star ratings and Top Safety Picks efficiently communicating to consumers the value of life-saving technologies. What is less well understood is how influential these safety ratings agencies are in shaping ADAS specification, as well as ADAS adoption, albeit indirectly. While a technology-neutral approach means that specific sensors or compute architectures are not required to secure a passing grade, the detail of the testing protocols tends to imply a cost-optimal technology specification.

Currently, testing protocols in all regions are carried out in ideal conditions—conditions so ideal that they do not represent the full set of situations that the ADAS being tested will ultimately encounter in the field. For example, the EuroNCAP updated testing protocols for 2023 require that Autonomous Emergency Braking (AEB) systems should not be tested while it is raining, or while visibility is less than 1 Kilometer (km). Daytime tests must be carried out with homogenous ambient lighting, with no strong shadows and without driving toward sunlight. Even the recently introduced nighttime tests for VRU require testing with at least 5 lux—a level of ambient lighting better associated with twilight or streetlight-lit routes.

This has the effect of discouraging the fitment of sensors that, although effective at improving perception in some real-world scenarios, do not help the OEM to secure the vital 5-Star rating or Top Safety Pick.

In May 2023, NHTSA in the United States published a Notice of Proposed Rulemaking (NPRM) that would set out requirements for the fitment of AEB systems on passenger vehicles and light trucks, with a final decision expected soon. When developing suggested testing protocols to ensure compliance, NHTSA first investigated the performance of 11 existing AEB systems to assess their efficacy in certain real-world scenarios, including low light scenarios. Ultimately, only 1 of these systems was able to consistently prevent a collision with a crossing VRU target in a full range of lighting scenarios.

In 2019, 77% of pedestrian fatalities that involved being struck by a light vehicle in the United States occurred in “dark conditions.” This pattern seems to be consistent across regions, with the Japanese Transport Ministry likewise reporting that 70% of pedestrian fatalities take place at night.

Therefore, in their proposed rulemaking, mandatory AEB systems would be subject to a test that involves assessing the systems’ ability to detect VRU targets in a number of scenarios with ambient lighting no greater than 0.2 lux—a degree of lighting more closely associated with unlit routes without a full moon. This represents an important step in ensuring that the real-world performance of ADAS continues to improve, and the automotive industry must anticipate that regulators will increasingly seek to leverage the details of testing protocols to continue to advance public safety.

Unlike passenger vehicle ADAS, robotaxis are not awarded any kind of standardized safety score. Rather, these operations live and die by the real-world performance of their vehicles, and therefore by the ability of their perception suites to robustly detect all road agents in all scenarios. Currently, robotaxis from Cruise and Zoox feature multiple FIR imaging sensors.

However, any vendor of a high-cost component dependent on scarce or expensive materials should temper expectations. While safety ratings agencies and regulators value a technology-neutral approach, in practice, they must keep one eye on state-of-the-art ADAS sensing and processing. It is not realistic to expect that EuroNCAP, the IIHS, or a national type approval system would develop testing protocols that imply the adoption of a technology that is cost-prohibitive, highly exposed to supply chain disruptions, unproven, or for which there is not a competitive set of suppliers for OEMs from which to choose.

Therefore, it is incumbent on the FIR imaging ecosystem to make the first move and find technical solutions to reduce costs, and to secure their supply chain. AutoSens Brussels 2023 brought together a number of important vendors in the FIR supply chain, including FIR image sensor manufacturers, chalcogenide manufacturers, and windscreen manufacturers, collectively demonstrating a feasible path to market for FIR in ADAS.

Umicore, an electro-optic materials manufacturer, has launched a series of chalcogenide lenses that, while not as effective as germanium in transmitting Infrared (IR) wavelengths, employs materials that are far cheaper and more readily available across geographies. Furthermore, the refractive index is less adversely impacted by temperature increases, making AEC Q100-compliant design far more feasible. Meanwhile, Sekurit, a glazing specialist, has developed a specialized transparent crystal material that would facilitate the inclusion of a thermal imaging camera behind the windscreen.

Combined with a Video Graphics Array (VGA) resolution FIR image sensor from Lynred, vendors are collectivity targeting a sub-€100 price point for FIR ADAS sensors to deliver on the 0.2 lux Pedestrian Autonomous Emergency Braking (PAEB) testing requirement elaborated upon by NHTSA in the May NPRM.

This is almost unique in the automotive sensor space. Every other modality has increased performance over time—cameras, radar, and Light Detection and Ranging (LiDAR) have all improved in their specifications (range, resolution, dynamic range, Standard Dynamic Range (SDR), etc.). The above collaboration would see a system that is less performant that FIR cameras currently used for night vision in order to address a specific need at a low price point. While counterintuitive, this less performant system, which emphasizes cost and supply chain integrity over resolution and range, is likely to be essential for FIR imaging, including adopting more capable systems oriented toward unsupervised automation at scale in the automotive industry.