IlluminateAI’s technology changes the physics of image capture to enable smartphones and cameras to accurately measure color and how objects reflect light. Our software takes a series of images while delivering a known ‘reference illuminant’, subtracting out (unknown) ambient light, and measuring how the reference illuminant (known light) reflects from the object of interest. By recording images using light with a known spectrum and intensity, we are able to precisely measure object reflectivity. From accurate reflectivity we can calculate the color of the object for any given lighting condition. As described above, we can also use our reflectivity data to do color-matching, and to identify materials properties. The approach described here minimizes or eliminates the impact of unknown ambient light without using any supplemental hardware or reference standards. Instead of capturing apparent color, which changes with lighting conditions, we measure color and reflectivity independent of lighting conditions. Our computational imaging breakthrough allows accurate data capture to enable trustworthy visual AI.
Adding a reference illuminant allows cameras to measure color and reflectivity.
We overcome the challenge of measuring color in the presence of unknown ambient light by synchronizing a camera with a known reference illuminant, and taking a series of images. Our reference illuminant allows our software to measure the color and reflectivity of an object with a known source of light and eliminate (or mitigate) the artifacts caused by unknown ambient light. To illustrate how this approach works, we describe the smartphone-selfie embodiment of our software. To measure skin color (for the beauty and dermatology markets), we take a series of selfies while modulating the light from the phone screen.
As shown in Figure 3 below, the simplest embodiment of our technology relies on two images. The first selfie is acquired with the user-facing phone screen set to black. This first image captures an image of the user’s face in the presence of the unknown ambient light. The second selfie is acquired with the user-facing phone screen set to white. This second selfie captures an image of the user’s face in the presence of unknown ambient light, and the white light that came from the screen. We then subtract the first image from the second image. Assuming the ambient light is constant, we are left with an image formed with only the white light (from the phone screen) reflected from the face. In this way we are able to record an image of the user’s face using only light from the phone screen, and eliminate artifacts caused by unknown ambient light, shadows or mixed lighting.
Figure 3 – To measure skin color we reimagine the selfie. Our software takes a series of selfies to eliminate the unknown ambient light from the scene and create a new image made using only the light from the phone screen. In the simplest embodiment shown here we do this by acquiring two selfies: the first with the screen set to black, the second with the screen set to white. We then subtract the first selfie from the second to create a new image (3) which only contains light emitted from the phone screen, which has been reflected from the face. In this way our software is able to acquire an image of the face taken with a known source of light.
Adding a reference illuminant is a remarkably simple and elegant way to measure reflectivity and color, and ensure AI can make decisions based on measurement, not inference. In practice, the implementation of this approach requires precisely knowing the intensity and spectrum of the reference illuminant, the ability to subtract-out the unknown ambient lighting, and access to the compute resources to calculate color and reflectivity. This approach also requires precise synchronization of the camera with the reference illuminant. From a hardware perspective this tight integration is possible in modern smartphones and cameras thanks to advances in image sensor, display, and lighting technology. Finally, and of critical importance, smartphone compute is powerful and inexpensive, so the algorithms needed to translate the raw data into color and reflectivity can run locally on a smartphone or at the edge, providing instant access to accurate visual data, while also maintaining data privacy (by avoiding the need to upload any personally identifiable information to the cloud).
For the smartphone embodiments described here, laboratory techniques are used a-priori to create calibration profiles of the light emitted by the phone screen (for selfie orientation) and for the flash (for world-facing orientation). Similarly, laboratory techniques are used to calibrate the selfie and world-facing cameras of popular smartphone models. While most smartphone models require a unique hardware profile, the image sensors, phone-screens, and flash modules are factory calibrated by the smartphone OEMs. This minimizes the ‘within phone model’ variation to an extent that the variation is below the threshold for our measurement accuracy specification. The combination of a calibrated reference illuminant and image sensor provide the underlying information needed to calculate accurate color and reflectivity on a per-pixel basis.
A reference illuminant enables any camera to measure color & reflectivity
The approach of capturing a series of images while modulating a known reference illuminant can readily be applied to other use cases. For selfie applications our software modulates light from the phone screen as our reference illuminant (as described above). In world-facing smartphone applications our software synchronizes the camera flash with the world-facing camera to measure the color and reflectivity of objects. In this case a-priori techniques are used to create calibration profiles of the light emitted by the flash and the response function of the world-facing camera. For generalized camera applications, such as machine vision or security, a supplemental (or existing light) source can be used as a reference illuminant. This light emission of this source needs to be characterized and synchronized to the camera.
For specialized spectroscopy applications the reference illuminant can include target wavelengths which enable the measurement of specific physical properties. Light sources such as LEDs and lasers allow for high-speed synchronization and wavelength selection, and can therefore be used to turn ordinary cameras into high performance spectrometers (as determined by the illuminant spectrum) with minimal cost and complexity.
In this way our software transforms ordinary smartphones and cameras into powerful tools capable of accurately measuring the color and physical properties of objects in a scene.
Finally, it is important to note that the reference illuminant does not need to overwhelm the ambient light in a scene, it just needs to be detectable by the camera. Similarly the reference illuminant does not necessarily need to be visible to humans: high-frequency modulation, invisible wavelengths or other methods can be used to make the reference illuminant invisible. Finally, the reference illuminant approach can be used to measure color and object reflectivity in video-based applications by leveraging high-frame rate cameras which are synchronized to controllable light sources.
Figure 4 – Adding a ‘reference illuminant’ (RGB shown here) makes it possible for any camera to accurately measure the color and reflectivity of an object. While the appearance of an object will change with changes in the color and intensity of ambient light (as shown by the large vector/arrow reflecting from the surface), the reflection of a fixed reference illuminant is invariant (as shown by the small vector/arrow reflecting from the surface). We separate out the reflection of the reference illuminant from the unknown ambient light by acquiring a series of images. Note that the reference illuminant just needs to be distinct and measurable, it does not need to overwhelm the ambient light in the scene. The example here shows color measurement using the visible spectrum, but a reference illuminant can be any combination of wavelengths (Infrared, visible, or UV) in order to do spectral analysis of the object.