JP2022524023A - White balance using a reference illuminant - Google Patents

Abstract

Presented herein are computer programs and related computer implementation techniques for achieving high fidelity color reproduction in the absence of any known reflection spectrum. That is, color reproduction with high fidelity can be realized without using a portable reference such as a gray card or a color checker. To achieve this, a new reference spectrum, the "reference illuminant spectrum," is introduced into the scene that will be imaged by the image sensor. The reference illuminant spectrum is created by a multi-channel light source with known spectral characteristics. [Selection diagram] FIG. 6

Description

Cross-reference to related applications This application, entitled "High Fidelity Color Technologies", claims priority to US Provisional Application No. 62 / 818,055 filed March 13, 2019.

Technical Fields Various embodiments relate to computer programs and related computer implementation techniques for achieving high fidelity color reproduction by correcting the color values of pixels in a digital image.

The illuminant can be characterized by a color temperature measured in Kelvin (K) degrees. The color temperature of the illuminant is a temperature at which the color of the light emitted from the heated blackbody matches the color of the illuminant. In a light emitter that does not substantially mimic a blackbody, such as a fluorescent sphere or light emitting diode (LED), the temperature at which the color of the light emitted from the heated blackbody approximates the color of the light emitter is the correlated color temperature. It is called (CCT).

Ideally, the scene in a digital image is illuminated by one or more emitters of the same color temperature. However, such things rarely happen. Instead, the scene is usually illuminated by multiple emitters with different color temperatures (this is a situation referred to as "mixed lighting"). For example, indoor scenes can be illuminated by overhead lighting elements, indirect sunlight shining through windows, and so on. Such a situation may also occur due to various lighting conditions with different color temperatures. For example, a pair of digital images of an outdoor scene may be created with a slight time lag, but if the sun darkens in the meantime, the color temperature will be different.

White balance adjustment is the process of balancing a digital image in an attempt to return the color temperature to neural. At a high level, white balance removes unrealistic color casts and mimics what the human eye does when looking at white objects. This is done so that objects in digital images that look white to the human eye are rendered white in those digital images.

This patent or application includes drawings made in at least one color. Upon request and payment of the required fees, a copy with color drawings of the Gazette of this patent or application will be provided by the Office.

FIG. 1A shows a top view of a multi-channel light source containing a plurality of color channels configured to produce a variety of different colors. FIG. 1B shows a side view of a multi-channel light source illustrating how the illuminant can be present within the housing in some embodiments. FIG. 1C shows an electronic device including a rear camera and a multi-channel light source configured to illuminate the surrounding environment. FIG. 2 shows an example of an arrangement of illuminants. FIG. 3 shows an example of a separation mechanism arranged on the image sensor. FIG. 4 shows an example of a communication environment that includes a characterization module programmed to improve the color fidelity of an image. FIG. 5 shows a network environment including a characterization module. FIG. 6 shows a flow diagram of the process of calibrating an electronic device including a multi-channel image sensor and a multi-channel light source before deployment. FIG. 7 shows a first image captured using the automatic white balance (AWB) setting, a second image captured using the fixed white balance (FWB) setting, and captured using the FWB setting. A flow diagram of the process of calculating a calibration matrix based on a series of differentially illuminated images is shown. FIG. 8 shows a flow diagram of a process that employs a calibration matrix created for an electronic device during the pre-deployment calibration process (eg, by completing the process of FIG. 6). FIG. 9 illustrates how a multi-channel light source performs a series of lighting events, thereby illuminating the scene differentially. FIG. 9 illustrates how a multi-channel light source performs a series of lighting events, thereby illuminating the scene differentially. FIG. 9 illustrates how a multi-channel light source performs a series of lighting events, thereby illuminating the scene differentially. FIG. 10 illustrates why the reference illuminant white balance (RIWB) approach described herein is required to reproduce high fidelity colors. FIG. 11 illustrates why the reference illuminant white balance (RIWB) approach described herein is required to reproduce high fidelity colors. FIG. 12 illustrates why the reference illuminant white balance (RIWB) approach described herein is required to reproduce high fidelity colors. FIG. 13 illustrates why the reference illuminant white balance (RIWB) approach described herein is required to reproduce high fidelity colors. FIG. 14 illustrates why the reference illuminant white balance (RIWB) approach described herein is required to reproduce high fidelity colors. FIG. 15 illustrates why the reference illuminant white balance (RIWB) approach described herein is required to reproduce high fidelity colors. FIG. 16 illustrates why the reference illuminant white balance (RIWB) approach described herein is required to reproduce high fidelity colors. FIG. 17 is a block diagram showing an example of a computing system capable of implementing at least some of the operations described herein.

Various features of the techniques described herein will become more apparent to those skilled in the art by reviewing this detailed description in conjunction with the drawings. Embodiments are exemplified, but not limited to, in these drawings, and similar references may indicate similar elements. Although these drawings depict various embodiments for illustrative purposes, those skilled in the art will recognize that alternative embodiments may be employed without departing from the principles of these techniques. There will be. Thus, while certain embodiments are shown in the drawings, these techniques are suitable for various modifications.

Manually correcting the white balance during post-processing is not only a time-consuming process, but it also tends to result in inconsistent results, even when performed by an expert. Some experts tend to prefer warmer digital images, while others tend to prefer cooler digital images. To address these issues, modern digital cameras are designed to automatically perform white balance. For example, a digital camera may have an automatic white balance (AWB) setting, which, when selected, identifies the brightest part as a white point and then digitally bases it on its reference. Allows the scene to be evaluated by an algorithm designed to attempt to balance the colors of the image.

In situations where color is important, neutral references may be introduced into the scene to avoid problems with the AWB algorithm. For example, a professional photographer / videographer may carry a portable reference that can be easily added to the scene. An example of a portable reference is a neutral gray flat surface derived from a flat reflection spectrum. Another example of a portable reference is a series of flat surfaces of different colors that correspond to different reflection spectra. The former is usually referred to as a "gray card" and the latter is usually referred to as a "color checker". Regardless of its format, the portable reference will introduce into the scene a known reflection spectrum that the algorithm can use as a reference for automatic white balance adjustment.

When the scene does not contain a known reflection spectrum, the AWB algorithm can (and often does) produce a digital image that is visibly inaccurate with respect to color. For example, if red is dominant in the scene, the AWB algorithm may try to correct it by confusing it with a warm illuminant-induced color cast and bringing the average color closer to neural. be. However, this introduces a bluish tint, which can be quite noticeable. 10-16 include examples of some images for which the AWB algorithm was unable to properly scale the color values.

Therefore, the present specification presents a computer program for achieving high fidelity color reproduction without using a portable reference, and related computer mounting techniques. To achieve this, a new reference spectrum, the "reference illuminant spectrum," is introduced into the scene that will be imaged by the image sensor. The reference illuminant spectrum is created by an illuminant whose spectral characteristics are known.

As further described below, a single reference illuminant may not be sufficient to properly render the colors in the scene. This is because for any single illuminant spectrum, there are instances of color metamerism in which the pixels corresponding to the objects with different reflection characteristics have the same color values as those measured by the image sensor. That is, these pixels are not actually the same color, but look the same to the image sensor. Therefore, multiple reference illuminant spectra may be introduced differentially to mitigate the effects of color metamerism.

Embodiments may be described with reference to a particular electronic device, light source or image sensor. For example, the art can be described in the context of a mobile phone including a multi-channel light source with multiple different colored LEDs and a multi-channel image sensor with red, green and blue channels. However, those skilled in the art will recognize that these features are equally applicable to other types of electronic devices, light sources and image sensors. For example, the same features may be applied by a multi-channel light source configured to produce invisible light (eg, ultraviolet and / or infrared) in place of or in addition to visible light. Thus, embodiments can be described in the context of a light source with multiple "color channels", but these features are non-color channels (ie, channels with one or more illuminants that produce invisible light). ) Can be equally applicable.

Embodiments may also be described with reference to "flash events". Generally, a flash event is performed by an electronic device to flood the scene with visible light for a short period of time while a digital image of the scene is being captured. However, the features described herein are applicable to other lighting events as well. For example, an electronic device strobes the color channels of a multi-channel light source to determine the effect of each color channel and then drives at least several color channels simultaneously, thereby extending the visible light flooding the scene. Can be generated. Accordingly, embodiments may be described in the context of capturing and processing digital images, but those of skill in the art are equally applicable to capture and process a series of digital images representing frames of video. You will recognize that.

The art may be embodied using dedicated hardware (eg, circuits), programmable circuits properly programmed with software and / or firmware, or a combination of dedicated hardware and programmable circuits. can. Thus, an embodiment, when executed, causes an electronic device to introduce a series of illuminant spectra into the scene, capture a series of images in conjunction with the series of illuminant spectra, and then capture the series of images. It may include a machine-readable medium with instructions to establish an information spectrum on a pixel-by-pixel basis based on image analysis.

Terminology Reference to "an embodiment" or "one embodiment" in this description means that the features, functions, structures or properties described are included in at least one embodiment. The appearance of such a phrase does not necessarily indicate the same embodiment, nor does it necessarily indicate another embodiment that is mutually exclusive.

Unless otherwise explicitly requested in the context, the words "comprise" and "comprising" are not exclusive or exhaustive, but in a comprehensive sense (ie, "contains". , But not limited to it. " The terms "connected", "coupled" or any variation thereof are intended to include any connection or connection between two or more elements and are direct or indirect. It doesn't matter. These connections / connections can be physical, logical or a combination thereof. For example, the components may be electrically or communically coupled to each other without sharing a physical connection.

The term "based" should also be interpreted in a comprehensive sense, not in an exclusive or exhaustive sense. Therefore, unless otherwise stated, the term "based" is intended to mean "at least partially based".

The term "module" broadly refers to software components, firmware components and / or hardware components. A module is typically a functional component capable of producing useful data or other (one or more) outputs based on a specified (one or more) inputs. The module may be self-contained. The computer program may include one or more modules. For example, a computer program may include multiple modules responsible for completing different tasks, or a single module responsible for completing all tasks.

When used with reference to a multi-item list, the word "or" is any of the items in the list, all of the items in the list, and any combination of the items in the list. It is intended to cover all of the interpretations of.

The sequence of steps performed in any of the processes described herein is exemplary. However, these steps may be performed in various orders and combinations as long as they do not violate physical potential. For example, steps can be added to the process described herein, and steps can be removed from the process. Similarly, the steps can be replaced or rearranged. Therefore, the description of any process is intended to be open-ended.

Outline of Light Source FIG. 1A shows a top view of a multi-channel light source 100 including a plurality of color channels capable of producing various different colors. Each color channel may include one or more illuminants 102 designed to produce light of substantially similar color. For example, the multi-channel light source 100 may include a single illuminant configured to produce a first color, a plurality of illuminants configured to produce a second color, and the like. For the purpose of simplification, the color channel is referred to as a "light emitter" regardless of the number of individual emitters contained in the color channel.

An example of a light source is an LED. The LED is a 2-lead light emitter generally made of an inorganic semiconductor material. Although embodiments may be described in the context of LEDs, the art is applicable to other types of illuminants as well. Table 1 includes a number of examples of colors that can be used in LEDs, as well as the corresponding wavelength ranges and representative materials.

Other colors not shown in Table 1 may also be incorporated into the light source 100. Examples of such colors are indigo purple (cyan, 490 <λ <515), lime (lime, 560 <λ <575), amber (amber, 580 <λ <590) and indigo (indigo, 425 <λ <450) is included. Those of skill in the art will recognize that these wavelength ranges are included solely for illustrative purposes.

As described above, the multi-channel light source 100 may include a plurality of color channels capable of producing different colors. For example, the light source 100 may include three distinct color channels configured to produce blue light, green light and red light. Such a light source is sometimes referred to as an "RGB light source". As another example, the light source 100 may include four distinct color channels configured to produce either blue light, green light, red light, and either amber light or white light. Such a light source may be referred to as an "RGBA light source" or an "RGBW light source". As another example, the light source 100 may include five distinct color channels configured to produce blue light, indigo purple light, slate gray light, amber light, and red light. As another example, the light source 100 may include seven distinct color channels configured to produce blue light, indigo purple light, green light, amber light, red light, purple light and white light. good. Therefore, the light source 100 may include 3 channels, 4 channels, 5 channels, 7 channels and the like.

The 3-channel and 4-channel light sources improve conventional flash techniques, but may have a wavy spectral distribution and a narrow range of high fidelity. As a result, the multi-channel light source 100 often contains at least 5 different color channels. As the number of color channels increases, so does the quality of light, the CCT range, the quality of the entire range, and spectral sampling. For example, a 5-channel light source with a well-selected illuminant can be designed to deliver full spectrum white light over a wide CCT range at ΔuV ± 0.002 (eg, 1650K to over 10000K). .. In addition, by using a 5-channel light source, the spectral distribution can be sampled in a substantially continuous (ie, non-wavy) manner.

Since the LEDs generate less heat, they can be placed close to each other. Therefore, when the light source 102 of the multi-channel light source is an LED, the light source 100 may include an array consisting of a plurality of dies arbitrarily arranged in close proximity to each other. However, it should be noted that the placement may be limited by the "white wall" space between adjacent dies. The white wall space is approximately 0.1 millimeters (mm), which can be limited based on the overall desired diameter of the light source 100 (eg, to 0.2 mm or less). For example, in FIG. 2, the array contains eight dies associated with five different color channels. Such an array may be sized to fit in the same dimensions as conventional flash techniques. The array may also be based on a standard manufacturing die requiring, for example, a lime-violet-violet-red-blue area ratio of 2-1-1-0.5-0.5. The array may be driven by one or more linear field effect transistor-based (FET-based) current-controlled drivers 110. In some embodiments, each color channel is driven by the corresponding driver. These drivers 110 may be attached to a substrate 104 arranged below the light emitter 102, or may be embedded in the substrate 104.

By driving each color channel independently, the multi-channel light source 100 can generate white light with different CCTs. For example, the multi-channel light source 100 may emit a flash of light that illuminates the scene in conjunction with the capture of the image by the electronic device. Examples of electronic devices include mobile phones, tablet computers, digital cameras (eg, single-lens reflex (SLR) cameras, digital SLR (DSLR) cameras and light field cameras (also referred to as "prenoptic cameras")) and the like. Is included. The light produced by the multi-channel light source 100 can improve the quality of images taken in the context of consumer photography, prosumer photography, professional photography and the like.

By controlling the multi-channel light source in such a way, it is possible to improve the accuracy / accuracy of the spectrum control over various operating states as compared with the conventional lighting technique. All conventional lighting techniques are designed to emit light in the desired segment of the electromagnetic spectrum. However, light (ie, the segment of the electromagnetic spectrum) changes based on factors such as temperature, aging, and brightness. Unlike traditional lighting techniques, multi-channel light sources can be processed so that the output of each channel is always visible. Using this information, the controller 112 (i) adjusts the current provided to each channel and / or (ii) compensates for the spectral shift to maintain the desired segment of the electromagnetic spectrum. The above factors can be compensated by adjusting the ratio of channels to. Examples of the controller 112 include a central processing unit (also referred to as a "processor").

In some embodiments, the multi-channel light source 100 can generate color light by individually driving the appropriate color channels. For example, the controller 112 drives a single color channel (eg, a red channel for producing red light) or multiple color channels (eg, a red channel and an amber channel for producing orange light). Thereby, the multi-channel light source 100 may generate color light. As mentioned above, the controller 112 may also cause the multi-channel light source 100 to generate white light with the desired CCT by driving each color channel simultaneously. In particular, the controller 112 may determine the operating parameters required to achieve the desired CCT based on the color mixing model. The operating parameter may specify, for example, the drive current that will be provided to each color channel. By changing the operating parameters, the controller can adjust the CCT of white light as needed.

The light emitter 102 is illustrated as an array of LEDs located on the substrate 104, but other arrangements are also possible. In some cases, different arrangements may be preferred due to thermal constraints, size constraints, color mixing constraints, and the like. For example, the multi-channel light source 100 may include a circular arrangement, a grid arrangement, or a cluster arrangement of LEDs.

FIG. 1B shows a side view of the multi-channel light source 100 and illustrates how the illuminant 102 is present in the housing in some embodiments. The housing may include a base plate 106 surrounding the illuminant 102 and / or a protective surface 108 covering the illuminant 102. Although the protective surface 108 shown herein is in the form of a dome, one of ordinary skill in the art will recognize that other designs are possible. For example, the protective surface 108 may instead be arranged parallel to the substrate 104. Further, the protective surface 108 may be designed so that the upper surface of the protective surface 108 is substantially coplanar with the outer surface of the electronic device when the multi-channel light source 100 is fixed in the electronic device. The protective substrate 108 may be made of a substantially transparent material such as glass or plastic.

The substrate 104 may be composed of any material capable of appropriately dissipating the heat generated by the illuminant 102.
Instead of a metal substrate, a non-metal substrate such as one made of a glass fiber woven fabric with an epoxy resin binder (eg, FR4) may be used. For example, the substrate 104 composed of FR4 can dissipate the heat generated by the plurality of color channels more efficiently without experiencing the retention problems typically encountered by metal substrates. .. However, some non-metal substrates cannot be used in combination with high power emitters commonly used for photography and video capture, so the substrate 104 may be made of metal, ceramic, etc. Please note.

The processing components required to operate the illuminant 102 may be physically separated from the light source 100. For example, the processing component may be connected to the light emitter 102 via a conductive wire that passes through the substrate 104. Examples of processing components include a driver 110, a controller 112, a power supply 114 (eg, a battery) and the like. Therefore, the processing component need not be located within the light source 100. Alternatively, the processing component may be located elsewhere in the electronic device where the light source 100 is installed.

As further described below, the multi-channel light source 100 is designed to operate in conjunction with an image sensor. Therefore, the multi-channel light source 100 may be configured to emit light in response to a determination that the image sensor has received a command to capture an image of the scene. The instruction may be created in response to an input indicating a request to capture an image. As shown in FIG. 1C, the image sensor (here, the camera 152) may be housed in the same electronic device as the multi-channel light source. The requirement may be provided in the form of a tactile input along the surface of a touch-sensitive display or a mechanical button accessible along the outside of the electronic device.

In some embodiments, the multi-channel light source is designed for easy installation within the housing of the electronic device.
FIG. 1C shows an electronic device 150 including a rear camera 152 and a multi-channel light source 154 configured to illuminate the surrounding environment. The multi-channel light source 154 may be, for example, the multi-channel light source 100 of FIGS. 1A to 1B. The rear camera 152 is an example of an image sensor that can be configured to capture an image in conjunction with the light generated by the light source 100. Here, the electronic device 150 is a mobile phone. However, one of ordinary skill in the art will recognize that the techniques described herein may be readily adapted to other types of electronic devices such as tablet computers and digital cameras.

The camera 152 is typically one of a plurality of image sensors contained within the electronic device 150. For example, the electronic device 100 may include a front camera that allows an individual to capture a still image or video while looking at a display. Rear and front cameras can often be different types of image sensors intended for different uses. For example, the image sensor may be capable of capturing images with different resolutions. As another example, the image sensor can be combined with a variety of different light sources (eg, the rear camera may be associated with a stronger flash than the front camera, while the front camera is located near the single channel light source. , The rear camera may be located near the multi-channel light source).

FIG. 2 shows an example of an array 200 of the illuminant 202. If the illuminant 202 is an LED, the array 200 may be manufactured using standard dies (also referred to as "chips"). A die is a small block of semiconductor material in which a diode is located. Typically, diodes corresponding to a particular color are mass-produced on a single wafer (eg, composed of electronic grade silicon, gallium arsenide, etc.), and then the wafer is made into multiple pieces. And cut (“diced”), each of the pieces contains a single diode. Each of these parts is sometimes referred to as a "die".

As shown in FIG. 2, the array 200 includes a plurality of color channels configured to produce light of different colors. Here, for example, the array 200 includes five color channels (blue, violet, lime, amber and red). Each color channel may include one or more illuminants. Here, for example, three color channels (ie, blue, violet and red) contain a plurality of illuminants and two color channels (ie, violet and amber) contain a single illuminant. The number of illuminants in each color channel and the arrangement of these illuminants in the array 200 may be modified based on desired output characteristics such as maximum CCT, minimum CCT, maximum temperature and the like.

The array 200 is capable of producing light in excess of 1,000 lumens, but some embodiments generate light in less than 1,000 lumens (eg, 700-800 lumens during a flash event). Designed to do. In some embodiments, the illuminant 202 is arranged in the array 200 in a highly symmetrical pattern to improve spatial color uniformity. For example, when the array 200 is designed to produce white light by driving multiple color channels simultaneously, the emitters corresponding to those color channels are symmetrical to facilitate mixing of the color lights. Can be placed.

The array 200 may be designed to be installed in the housing of an electronic device (eg, the electronic device 150 of FIG. 1C) in addition to or in place of conventional flash components. For example, some arrays designed to be installed in a mobile phone are less than 4 mm in diameter, while some arrays designed to be installed in a mobile phone are less than 3 mm in diameter. The array 200 may also be less than 1 mm in height. In some embodiments, the estimated total area required for the array can be less than 3 mm ² before installation and less than 6 mm ² after installation. Such a design allows the array 200 to be placed within the cell phone without the need for significant relocation of components within the cell phone.

One of the advantages of a compact die array is the ability to achieve good color mixing and proper field of view (FOV) without the use of collimators, diffusers or lenses. However, a collimator 204 (also referred to as a "mixing pipe") designed to ensure proper spatial color uniformity of the light produced by the illuminant 202 may be placed around the array 200. .. At high levels, the collimator 204 may facilitate more uniform color mixing and better control of the FOV of the light emitted by the illuminant 202. The collimator 204 may consist of a non-flexible material (eg, glass) or a flexible material (eg, silicone). The collimator 204 may be in the form of a tubular body. In some embodiments, the outlet opening of the tubular body is narrower than the array (eg, the outlet opening may have a diameter of 2.5 mm, 3 mm or 3.5 mm), while other embodiments. In the form, the outlet opening of the tubular body is wider than the array (eg, the exit opening may have a diameter of 4.5 mm, 5 mm or 5.5 mm). Therefore, the tubular body may have an inclined inner surface that focuses or disperses the light produced by the illuminant 202.

The array 200 may be used in place of or in addition to conventional flash techniques configured to generate flash in conjunction with image capture. Therefore, the electronic device (eg, the electronic device 150 of FIG. 1C) may include a single-channel light source and / or a multi-channel light source.

Although embodiments may be described in the context of LEDs, one of ordinary skill in the art will recognize that other types of light sources may be used in place of or in addition to LEDs. For example, embodiments of the present technology include lasers, quantum dots (“QD”), organic LEDs (“OLED”), resonant cavity LEDs (“RCLED”), vertical cavity surface emitting lasers (“VCSEL”), and superluminous diodes. ("SLD" or "SLED"), blue "pump" LED under the phosphorescent layer, up-conversion phosphorescent material (eg, fine ceramic particles that provide a response when excited by infrared light), nitrided Phosphorus light substance (eg CaAlSiN, SrSiN, KSiF), down conversion phosphor light substance (eg KSF: Mn4 +, LiAlN), zinc rubidium phosphate, yttrium-aluminum-garnet (YAG) phosphor light substance, lutetium-aluminum- Garnet (LAG) phosphorescent material, SiAlON phosphorescent material, SiON phosphorescent material, or any other combination thereof may be adopted. For example, the array 200 may contain phosphorescent converted colors such as lime produced by the YAG phosphorescent coating on the blue LED. In such an embodiment, a highly efficient blue LED pumps the YAG phosphorescent coating with photons that are almost completely absorbed and then re-emitted in a wider yellow-green band. This can also be done to create other colors such as red, amber, green, indigo and purple. As another example, a plurality of VCSELs and / or a plurality of QDs are placed on the substrate so that the VCSEL and / or the QD emits electromagnetic radiation to the outside when the substrate is installed in the housing of the electronic device. Can be arranged in a given pattern. The type of illuminant used to illuminate the scene can affect the schedule of lighting events. In other words, some illuminants may be required to be tuned in terms of timing. For example, because phosphorescent-based illuminants generally exhibit delayed excitation and de-excitation, phosphorescent-based light sources overlap (ie, a second phosphorescent-based illuminant is activated. It is activated (eg, strobe irradiation) by an early on / early off method to avoid the first phosphorescent material-based illuminant still emitting some light when it is done. May be good.

Overview of image sensor An image sensor is a sensor that detects information that constitutes an image. In general, an image sensor realizes the information detection by converting a variable attenuation of a light wave (eg, when passing or reflecting an object) into an electrical signal. Here, the electrical signal represents a small burst of current carrying information. Examples of image sensors include semiconductor-charge-coupled device (CCD) and complementary metal-oxide-semiconductor sensor (CMOS) sensors. Both types of image sensors perform the same task: converting the captured light into an electrical signal. However, because CMOS sensors are generally cheaper, smaller, and consume less power than CCDs, many electronic devices use CMOS sensors for image capture.

Image sensors can also have different separation mechanisms. One of the most common separation mechanisms is a filter array that passes light of different colors to selected pixel sensors. For example, each sensor element may be made sensitive to either red light, green light or blue light by a color gel made of a chemical dye. Since the image sensor separates the incident light based on the color, it can be said that the image sensor has a plurality of sensor channels or a plurality of color channels. Therefore, an image sensor including a plurality of sensor channels corresponding to various different colors may be referred to as a "multi-channel image sensor".

FIG. 3 shows an example of the separation mechanism 302 arranged on the image sensor 304. Here, the separation mechanism 302 is a Bayer filter array containing three different types of color filters designed to separate incident light into red, green or blue light on a pixel-by-pixel basis. On the other hand, the image sensor 304 may be a CMOS sensor. Instead of using a photochemical film to capture the image, the electronic signal generated by the image sensor 304 is recorded in memory for subsequent analysis.

After the recording function is initiated (eg, in response to receiving an input indicating a request to capture an image), the lens focuses light on the image sensor 604 through the separation mechanism 602. As shown in FIG. 3, the image sensor 304 may be arranged in a grid pattern of separate imaging elements. In general, the image sensor 304 determines the intensity of the incident light, not the color of the incident light. Instead, the color is usually determined by the use of a separation mechanism 302 that allows only a single color of light for each imaging element. For example, a Bayer filter array can be used to separate incident light into three different colors (ie, red, green, blue) and average these different colors within a 2x2 arrangement of imaging elements. Includes three different types of color filters. Each of the given images may be associated with the placement of such imaging elements. Therefore, each pixel can be assigned different values for red light, green light and blue light. Another method of color identification employs separate image sensors, each dedicated to capturing a portion of the image (eg, a single color), and the results can be combined to produce a full-color image.

Overview of characterization Modules FIG. 4 shows an example of a communication environment 400 including a characterization module 402 programmed to improve the color fidelity of an image. The term "module" broadly refers to software components, firmware components and / or hardware components. Therefore, the aspects of the process described below may be implemented in software, firmware and / or hardware. For example, can these processes be performed by a software program (eg, a mobile application) running on an electronic device (eg, a mobile phone) that includes a multi-channel image sensor and a multi-channel light source, or these processes are multi-channel. It can be performed by an integrated circuit that is part of the channel image sensor.

As shown in FIG. 4, the characterization module 402 may obtain data from a variety of different sources. Here, for example, the characterization module 402 is driven by a first data 404 generated by a multi-channel image sensor 408 (eg, camera 152 in FIG. 1C) and a multi-channel light source 410 (eg, light source 154 in FIG. 1C). Acquire the generated second data 406. The first data 404 can specify an appropriate value for each sensor channel on a pixel-by-pixel basis. For example, if the multi-channel image sensor 408 contains three sensor channels (eg, red, green and blue), each pixel has at least three distinct values (eg, red, green and blue values). Will be associated with. The second data 406 can specify the characteristics of each channel of the multi-channel light source 410. For example, the second data 406 may specify the drive current of each color channel during a lighting event (also referred to as a “lighting event”), the dominant wavelength of each color channel, the illuminance profile of each color channel, and the like.

In some embodiments, the multi-channel image sensor 408 and the multi-channel light source 410 are housed in the same electronic device. In another embodiment, the multi-channel image sensor 408 and the multi-channel light source 410 reside in separate housings. For example, in the context of professional photography or video capture, multiple multi-channel image sensors and multiple multi-channel light sources may be arranged in different arrangements to capture / illuminate different parts of the scene.

FIG. 5 shows a network environment 500 including the characterization module 502. The individual may interface with the characterization module 502 via interface 504. As further described below, the characterization module 502 may be responsible for improving the color fidelity of the image produced by the multi-channel image sensor. The characterization module 502 may also be responsible for creating and / or supporting interfaces for individuals to display improved images, initiate post-processing operations, manage settings, and so on.

The characterization module 502 may be present in the network environment 500, as shown in FIG. Therefore, the characterization module 502 may be connected to one or more networks 506a-b. Networks 506a-b may include personal area networks (PANs), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), cellular networks, the Internet, and the like. In addition or instead, the characterization module 502 may be communicably coupled to an electronic device via a short range communication protocol such as Bluetooth® or Near Field Communication (NFC).

In some embodiments, the characterization module 502 resides on the same electronic device as the multi-channel image sensor and multi-channel light source. For example, the characterization module 502 may be part of a mobile application capable of manipulating a mobile phone's multi-channel image sensor. In another embodiment, the characterization module 502 is communicably coupled to a multi-channel image sensor and / or a multi-channel light source via a network. For example, the characterization module 502 may be run by a network accessible platform (also referred to as a "cloud platform") that resides on the computer server.

Interface 504 is preferably accessible via a web browser, desktop application, mobile application or over-the-top (OTT) application. Accordingly, the interface 504 is a mobile phone, tablet computer, personal computer, game console, music player, wearable electronic device (eg, watch or fitness accessory), networked (“smart”) electronic device (eg, television or home). It may be displayed on an assistant device), a virtual / augmented reality system (eg, a head-mounted display) or other electronic device.

Some embodiments of the characterization module 502 are hosted locally. That is, the characterization module 502 may reside on the same electronic device as the multi-channel image sensor or multi-channel light source. For example, the characterization module 502 may be part of a mobile application capable of operating a mobile phone's multi-channel image sensor.

Other embodiments of the characterization module 502 are performed by a cloud computing service operated by Amazon Web Services® (AWS), Google Cloud Platform® Microsoft Azure® or similar technology. .. In such an embodiment, the characterization module 502 may reside on a host computer server communicatively coupled to one or more content computer servers 508. The content computer server 508 may include a color mixing model, items required for post-processing (eg, heuristics and algorithms) and other assets.

Although embodiments may be described in the context of networked electronic devices, the characterization module 502 does not necessarily have to be continuously accessible over the network. For example, the electronic device may be configured to run a self-contained computer program that only accesses the network accessible platform while completing the pre-deployment configuration steps. In such embodiments, the self-contained computer program may download or upload information to or from a network accessible platform at a single point in time. After deploying the electronic device (eg, after the electronic device has been packaged for sale), the self-contained computer program does not have to communicate with a network-accessible platform.

White Balance Using Reference Emissive Spectrum Modern digital cameras are designed to automatically perform white balance when an image is generated. For example, an electronic device has an AWB setting that, when selected, is designed to identify the brightest part as a white point and then balance the colors of the digital image based on that reference. Allows the scene to be evaluated by the algorithm. Such an approach tends to be quite effective in producing high fidelity colors when the scene contains known reflection spectra that can be used as a reference. For example, professional photographers / videographers typically add gray cards or color checkers to scenes that can be used as a reference for AWB algorithms.

When the scene does not contain a known reflection spectrum, the AWB algorithm can (and often does) produce a digital image that is visibly inaccurate with respect to color. For example, if red is dominant in the scene, the AWB algorithm may try to correct it by confusing it with a warm illuminant-induced color cast and bringing the average color closer to neural. be. However, this introduces a bluish tint, which can be quite noticeable.

Adding known reflection spectra is not practical in many situations, so a better approach is needed to generate images with high fidelity colors. Presented herein are computer programs and related computer implementation techniques for achieving high fidelity color reproduction in the absence of any known reflection spectrum. That is, color reproduction with high fidelity can be realized without using a portable reference such as a gray card or a color checker. To achieve this, a new reference spectrum, the "reference illuminant spectrum," is introduced into the scene that will be imaged by the image sensor. The reference illuminant spectrum is created by a multi-channel light source with known spectral characteristics.

FIG. 6 shows a flow diagram of process 600 for pre-deployment calibration of an electronic device including a multi-channel image sensor and a multi-channel light source. Process 600 may be initiated by the manufacturer of the electronic device before being packaged for sale.

First, the manufacturer selects a scene that contains at least one known reflection spectrum (step 601). Manufacturers typically achieve this by selecting and / or creating a scene that contains one or more portable references. An example of a portable reference is a flat surface in neutral gray color derived from a flat reflection spectrum, which is referred to as a "gray card". Another example of a portable reference is a series of flat surfaces of different colors that correspond to different reflection spectra and is referred to as a "color checker". In general, the manufacturer selects the scene to include a variety of different reflection spectra. These different reflection spectra can be provided by a single portable reference (eg, color checker) or by multiple portable references (eg, color checker and gray card).

Next, a set of images of the scene is captured one after another. For example, the electronic device captures a first image of the scene over the first exposure interval using the automatic white balance (AWB) setting (step 602) and a second image of the scene. You may capture using the fixed white balance (FWB) setting over the exposure interval of 2 (step 603). Neither the first image nor the second image is taken in combination with a lighting event performed by a multi-channel light source. In other words, the first and second images are captured in combination with the same ambient light but different white balance settings. Normally, AWB settings are applied by electronic devices by default to automatically correct color cast. On the other hand, the FWB setting may be either a custom white balance or one of the preset white balances provided by the electronic device for various color temperatures. Examples of preset white balances include tungsten, fluorescent lights, daytime, flash, cloudy weather, shade and the like. In general, the first exposure interval is different from the second exposure interval. For example, the second exposure interval may be 10, 20, 30 or 50 percent of the first exposure interval.

The electronic device may also capture a series of differentially illuminated images of the scene using the FWB setting (step 604). That is, the electronic device may capture a series of images in combination with a series of different illuminant spectra. As further described below with respect to FIG. 9, the series of different emitter spectra is a multi-channel light source such that all color channels produce a series of flashes that are illuminated with higher intensity by a single color channel. It can be generated by addressing each color channel of. Therefore, the number of differentially illuminated images may correspond to the number of color channels of the multi-channel light source.

The characterization module can then calculate the calibration matrix based on the analysis of a set of images, i.e., a first image, a second image, and a series of differentially illuminated images (step 605). ). As further described below with respect to FIG. 7, each entry in the calibration matrix may contain a vector of coefficients calculated from the corresponding pixels in the set of images. The characterization module can then store the calibration matrix in memory accessible to the electronic device (step 606). The calibration matrix is usually stored in the local memory of the electronic device for quicker callback during later imaging operations. However, the calibration matrix may be stored in remote memory accessible to electronic devices over the network, in place of or in addition to local memory.

FIG. 7 shows a first image captured using the AWB setting, a second image captured using the FWB setting, and a series of differentially illuminated images captured using the FWB setting. The flow diagram of the process 700 to calculate the calibration matrix based on is shown. At a high level, the characterization module, for each pixel, can be used to register for the corresponding entry in the calibration matrix, with the perimeter subtracted chromaticity fingerprint (or simply "chromaticity fingerprint"). Can be generated.

First, the characterization module divides the red, green, and blue values of each pixel in the second image from the red, green, and blue values of the corresponding pixels in a series of differentially illuminated images. This creates a series of modified images (step 701). This results in a modified image in which the red, green and blue channels of each pixel are reduced by that amount in the unlit second image.

The characterization module then transforms a series of modified images into a CIELAB color space so that each pixel is represented by a series of a ^* values and a series of b ^* values (step 702). In the CIELAB color space (also referred to as "CIE L ^* a ^* b ^* color space" or "Lab color space"), the colors are lightness L ^* (black (0) to white (11)), a ^* (green). It is expressed as three values (-) to red (+)) and b ^* (blue (-) to yellow (+)). The chromaticity fingerprint consists of these a ^* and b ^* values. For example, suppose a multi-channel light source contains five color channels. In such a situation, the series of modified images will contain 5 images, and the chromaticity fingerprint (F) for each pixel can be represented in the following vector format:
F = [a ^* ₁ b ^* ₁ a ^* ₂ b ^* ₂ a ^* ₃ b ^* ₃ a ^* ₄ b ^* ₄ a ^* ₅ b ^* ₅ ]
Here, each value pair (a ^* _i b ^* _i ) is associated with the corresponding pixel in one of the modified images. Similarly, the characterization module can transform the first image into the CIELAB color space so that each pixel is represented as a reference a ^* value and a reference b ^* value (step 703). The reference a ^* and b ^* values representing Ground Truth's answer can be expressed in the vector format shown below.
[A ^* _r b ^* _r ]

For each pixel, the characterization module can form a system of linear equations for a ^* and b ^* using the vector of coefficients (C) as shown below.
C ・ [a ^* ₁ b ^* ₁ a ^* ₂ b ^* ₂ a ^* ₃ b ^* ₃ a ^* ₄ b ^* ₄ a ^* ₅ b ^* ₅ ] _xy = a ^* _{r, xy}
C ・ [a ^* ₁ b ^* ₁ a ^* ₂ b ^* ₂ a ^* ₃ b ^* ₃ a ^* ₄ b ^* ₄ a ^* ₅ b ^* ₅ ] _xy = b ^* _{r, xy}
Here, C = [c ₁ c ₂ c ₃ c ₄ c ₅ ], and xy is the coordinate of the pixel. Therefore, the color value of a given pixel in the first image is defined as the inner product of the coefficient vector and the chromaticity fingerprint of that pixel determined from a series of modified images. At a high level, each of the systems of linear equations, with (i) a first linear equation based on (i) a reference a ^* value of a given pixel and a series of a ^* values of the given pixel (step 704). (Ii) Represents a second linear equation based on a reference b ^* value of a given pixel and a series of b ^* values of the given pixel (step 705).

The characterization module can then perform a least squares optimization on the system of linear equations to generate a vector of coefficients for each pixel (step 706). In other words, the characterization module can perform least squares optimization to establish the coefficients. The characterization module can then register a vector of coefficients in the data structure representing the calibration matrix (step 707). Each entry in the calibration matrix may contain a vector of coefficients established for the corresponding pixel.

FIG. 8 shows a flow diagram of process 800 that employs a calibration matrix created for electronic devices during the pre-deployment calibration process (eg, by completing process 600 in FIG. 6). By adopting a calibration matrix, the electronic device can achieve high fidelity color reproduction without the need for a portable reference to be present in the scene.

First, a set of images of the scene is captured one after another. For example, the electronic device captures the first image of the scene over the first exposure interval using the AWB setting (step 801) and captures the second image of the scene over the second exposure interval. , You may capture using the FWB setting (step 802). Neither the first image nor the second image is taken in combination with a lighting event performed by a multi-channel light source. In general, the first exposure interval is different from the second exposure interval. For example, the second exposure interval may be 10, 20, 30 or 50 percent of the first exposure interval. The electronic device may also capture a series of differentially illuminated images of the scene using the FWB setting (step 803). That is, the electronic device may capture a series of images in combination with a series of different illuminant spectra. As further described below with respect to FIG. 9, the series of different emitter spectra is a multi-channel light source such that all color channels produce a series of flashes that are illuminated with higher intensity by a single color channel. It can be generated by assembling each color channel of. Therefore, the number of differentially illuminated images may correspond to the number of color channels of the multi-channel light source.

The characterization module then divides the red, green, and blue values of each pixel in the second image from the red, green, and blue values of the corresponding pixels in a series of differentially illuminated images. By doing so, a series of modified images is created (step 804). This results in a modified image in which the red, green and blue channels of each pixel are reduced by that amount in the unlit second image. The characterization module may then generate a chromaticity fingerprint for each pixel based on a series of modified images (step 805). Step 805 of FIG. 8 is substantially similar to step 702 of FIG. These chromaticity fingerprints can be multiplied by a calibration matrix to obtain calibrated a ^{* and calibrated b * values} for each pixel (step 806). More specifically, each chromaticity fingerprint can be multiplied by the corresponding entry in the calibration matrix to obtain the calibrated a ^* and b ^* values. Generally, the calibration matrix is stored in the local memory of the electronic device. However, the calibration matrix can be drawn from remote memory to which the electronic device is connected over the network.

The characterization module can then transform the first image into the CIELAB color space so that each pixel is represented as an L ^* value, an a ^* value, and a b ^* value (step 807). However, rather than using these a ^* and b ^* values, the characterization module replaces them with the calibrated a ^* and b ^* values obtained in step 806 to calibrate the image. Can be generated. This calibrated image may represent color using the L ^* values of the first image and the calibrated a ^* and b ^* values derived from a series of modified images.

It is envisioned that the above steps may be performed in various orders and combinations, as long as it does not violate physical potential. For example, multiple instances of process 600 in FIG. 6 may be executed for multiple sets of images. Other steps may also be included in some embodiments. For example, the electronic device can display a calibrated image on the interface for confirmation. The electronic device may display a calibrated image near the first image so that the individual can see the difference, or the electronic device alternates between the personally calibrated image and the first image. It may be possible to switch.

FIG. 9 illustrates how a multi-channel light source (or simply a "light source") performs a series of lighting events, thereby illuminating the scene differentially. Differential lighting can be key to the calibration process described for FIG. 6 and the optimization process described for FIG.

For example, suppose a light source contains five different color channels and produces five different illuminant spectra. Each color channel may be individually driven by an electric current. More specifically, each color channel aggregates in some intermediate state between the off state (eg, when the output level is 0) and the maximum intensity state (eg, when the output level is 255). It may be possible. As an example, the maximum intensity state may correspond to an output current of approximately 1 amp, while all intermediate states may correspond to some fractional amp.

The drive currents of the different color channels can be modified to produce different illuminant spectra. FIG. 9 contains examples of five different combinations of channel spectra shown with the corresponding complex illuminant spectra. Here, each composite illuminant spectrum is generated by driving four color channels at an output level of 100 and one color channel at an output level of 150. However, one of ordinary skill in the art will recognize that these numbers are provided solely for explanatory purposes. In general, as long as one output level is higher than the other, the output level itself is not important, but it may be desirable to drive all color channels with sufficient current to produce white light. .. As shown in FIG. 9, each composite illuminant spectrum has a different spectral power distribution (SPD).

Evidence of the Need for High Fidelity Color Reproduction Figures 10-16 illustrate why the reference illuminant white balance (RIWB) approach described herein is needed to reproduce high fidelity colors. .. In each drawing, the first image in the upper left corner is taken with the AWB setting in a scene that does not have a known reflection spectrum, while the second image in the upper right corner (here, the color checker). The same scene with a known reflection spectrum (provided by) is taken with the AWB setting. At a high level, the second image can be thought of as a "ground truth" that shows what the first image actually looks like.

Below these images are two columns. The first column contains a comparison of the average colors of the pixels within the segmented portion of each image, and the second column visually contains a comparison of the average colors of those pixels after the brightness scaling operation. These average color values provide an indicator of how close the pixels of the first image are compared to the second image. It should be noted that these average color values can be quite different. In fact, these average color values are, in most cases, easily distinguishable from each other, as indicated by the delta E (dE) value corresponding to the color error. Many of these delta E values are above 5.0, which means that the visual difference between the first and second images should be very noticeable. By adopting the RIWB approach described herein, a delta E value of less than 1.0 can be achieved. In most cases, color pairs with a delta E value of less than 1.0 are visually indistinguishable from each other.

The second column is similar to the first column, except that the segmented parts are adjusted to have the same lightness. That is, the pixels in the segmented portion are scaled to have the same L ^* value in the CIELAB color space. This is done because the introduction of a color checker can affect the exposure when comparing the known and unknown reflection spectra, which can make the second image brighter or darker. .. As can be seen in the second column, despite equalizing the brightness levels between the segmented parts of the first and second images (ie, brightening the dark parts of these images, or these. The colors are still significantly off (by darkening the bright areas of the image).

Computing System FIG. 17 is a block diagram showing an example of a computing system 1700 capable of implementing at least some of the operations described herein. For example, some components of the computing system 1700 may be part of an electronic device (eg, electronic device 150 in FIG. 1) that includes a multi-channel light source and / or a multi-channel image sensor.

The computing system 1700 includes one or more central processing units (also referred to as "processors") 1702, main memory 1706, non-volatile memory 1710, network adapter 1712 (eg, network interface), and video display 1718. And the input / output device 1720, the control device 1722 (eg, keyboard and pointing device), the drive unit 1724 including the storage medium 1726, and the signal generation device 1730 communicably connected to the bus 1716. good. Bus 1716 is illustrated as an abstraction representing one or more physical buses and / or point-to-point connections connected by appropriate bridges, adapters or controllers. Therefore, bus 1716 includes a system bus, a Peripheral Component Interconnect (PCI) bus or a PCI-Express bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, and a universal serial bus. (USB), IIC (I2C) bus, or standard 1394 bus of the Electrical and Electronic Engineers Association (IEEE) (also referred to as "Firewire") may be included.

The computing system 1700 includes personal computers, tablet computers, mobile phones, game consoles, music players, wearable electronic devices (eg, watches or fitness trackers), networked (“smart”) devices (eg, televisions or home assistant devices). , A virtual / augmented reality system (eg, a head-mounted display), or a set of instructions (sequentially or otherwise) that specify an action (one or more) that will be performed by the computing system 1700. ) It may share a computer processor architecture similar to another viable electronic device.

The main memory 1706, the non-volatile memory 1710 and the storage medium 1726 (also referred to as "machine readable medium") are shown as a single medium, while the terms "machine readable medium" and "storage medium" are 1 It should be understood to include a single medium or multiple media (eg, a centralized or distributed database, and / or associated caches and servers) that store a set of 1728 or more instructions. The terms "machine readable medium" and "storage medium" should be understood to include any medium that can store, encode or carry a set of instructions executed by the computing system 1700.

As a whole, the routine performed to implement the embodiments of the present disclosure is one of an operating system or a particular application, component, program, object, module or instruction sequence (collectively referred to as a "computer program"). It may be implemented as a part. A computer program typically comprises one or more instructions set at various time points (eg, instructions 1704, 1708, 1728) in various memory and storage devices of the computer. When the (one or more) instructions are read and executed by one or more processors 1702, the computing system 1700 is made to perform an operation to execute an element with various aspects of the present disclosure.

Further, although embodiments are described in the context of fully functional computing devices, one of ordinary skill in the art will appreciate that different embodiments can be distributed as different forms of program product. There will be. This disclosure applies regardless of the particular type of machine or computer readable medium actually used for distribution.

Further examples of machine-readable storage media, machine-readable media or computer-readable media include volatile and non-volatile memory devices 1710, floppy and other removable disks, hard disk drives, optical disks (eg, compact disk read-only memory (CD-ROM)). ), Recordable media such as digital multipurpose discs (DVDs)), and transmission media such as digital and analog communication links.

The network adapter 1712 allows the computing system 1700 to mediate data in the network 1714 with an entity external to the computing system 1700 through any communication protocol supported by the computing system 1700 and the external entity. to enable. Network adapter 1712 includes network adapter cards, wireless network interface cards, routers, access points, wireless routers, switches, multilayer switches, protocol converters, gateways, bridges, bridge routers, hubs, digital media receivers and / or repeaters. be able to.

The network adapter 1712 may control and / or manage permissions to access / proxy data in the computer network, and is a firewall that tracks different levels of trust between different machines and / or applications. May include. Firewalls regulate the flow of traffic and resource sharing between a particular set of machines and an application, between machines and / or between an application and an application (eg, between these entities). It can be any number of modules with any combination of hardware and / or software components capable of implementing a given set of access rights. The firewall also manages permissions, including permissions and manipulation of objects by individuals, machines and / or applications, and access control lists detailing when the permissions are in effect, and / or said access controls. You can access the list.

The techniques presented herein are programmable circuits (eg, one or more microprocessors), software and / or firmware, dedicated hard-wired (ie, non-programmable) circuits, or as such. It can be implemented by a combination of various forms. Dedicated circuits can be in the form of one or more application specific integrated circuits (ASICs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and the like.

NOTE The above description of the various embodiments in the claimed subject matter are provided for illustration and explanation. The description is not intended to be exhaustive or to limit the claimed subject matter to the exact form disclosed. Many modifications and variations will be apparent to those skilled in the art. Embodiments have been selected and described to best illustrate the principles of the invention and its practical application, so that those skilled in the art of related art will appreciate the claimed subject matter, various embodiments and considerations. Various modifications suitable for a particular application can be understood from these embodiments.

Although the detailed description describes specific embodiments and the best embodiments to be considered, those embodiments are performed in many ways, regardless of how detailed they are disclosed above. obtain. Although embodiments may vary considerably in the details of their implementation, they are still incorporated herein. The particular term used in describing a particular function or embodiment in various embodiments is herein limited to a particular feature, function, or embodiment of the art in which it relates. It should not be understood as an implicit implication of being redefined. As a whole, the terms used in the following claims are intended to limit the technique to the specific embodiments disclosed herein, unless expressly defined herein. Should not be interpreted. Accordingly, the practical scope of the present invention includes not only the disclosed embodiments but all equal methods of implementing or implementing those embodiments.

The language used herein is selected primarily for readability and teaching purposes and may not be selected to delineate or limit the subject matter of the invention. Accordingly, the scope of the art is intended to be limited not by this detailed description but by any claims published in the application under this specification. Therefore, the disclosure of the various embodiments is intended to illustrate, but not limit, the scope of the art as described in the claims below.

Claims (19)

A method for calibrating electronic devices using image sensors.
A step of capturing a first image of a scene with at least one known reflection spectrum using an automatic white balance (AWB) setting over the first exposure interval.
A step of capturing a second image of the scene with a fixed white balance (FWB) setting over a second exposure interval different from the first exposure interval.
A step of capturing a series of differentially illuminated images of the scene using the FWB settings.
A step of calculating a calibration matrix based on the first image, the second image, and the series of differentially illuminated images.
A method comprising storing the calibration matrix in a memory accessible to the electronic device. Each color channel of the light source further comprises assembling by at least three color channels in such a way as to perform a series of lighting events in which all color channels are illuminated with higher intensity by one color channel.
The method of claim 1, wherein the series of differentially illuminated images is captured in conjunction with the series of lighting events. The method of claim 1, wherein the FWB setting is one of a plurality of preset white balances provided by the electronic device. The method of claim 1, wherein the second exposure interval is 20 percent or less of the first exposure interval. The calculation step is
A series by dividing the red, green, and blue values of each pixel in the second image from the red, green, and blue values of the corresponding pixels in the series of differentially illuminated images. Steps to create a modified image,
A step of converting the set of modified images into a CIELAB color space, such that each pixel is represented by a set of a ^* values and a series of b ^* values.
A step of converting the first image into a CIELAB color space so that each pixel is represented as a reference a ^* value and a reference b ^* value.
For each pixel,
(I) a first linear equation based on the corresponding reference a ^* value and the corresponding series of a ^* values, and (ii) the corresponding reference b ^* value and the corresponding series of b ^* values. Second linear equation based on,
Steps to form a system of linear equations, including
For each pixel, a step of performing least squares optimization on the first and second linear equations to generate a vector of coefficients.
The method of claim 1, comprising registering a vector of the coefficients in a data structure representing the calibration matrix. The method of claim 1, further comprising receiving an input specifying the FWB setting used to capture the second image. The method of claim 1, wherein at least one reference reflection spectrum is provided by a gray card with a flat surface in neutral gray color. The method of claim 1, wherein the at least one reference reflection spectrum is provided by a color checker comprising at least one color evaluation sample in the Lighting Engineering Association (IES) Technical Memorandum (TM) 30-15. A method for using a calibration matrix created for an electronic device with an image sensor.
With the step of capturing the first image of the scene over the first exposure interval using the automatic white balance (AWB) setting,
A step of capturing a second image of the scene with a fixed white balance (FWB) setting over a second exposure interval different from the first exposure interval.
A step of capturing a series of differentially illuminated images of the scene,
A series by dividing the red, green, and blue values of each pixel in the second image from the red, green, and blue values of the corresponding pixels in the series of differentially illuminated images. Steps to create a modified image of
For each pixel, a step of generating a chromaticity fingerprint based on the series of modified images,
A step of multiplying each chromaticity fingerprint by a calibration matrix stored in the memory of the electronic device, thereby obtaining a calibrated a ^* value and a calibrated b ^* value for each pixel. Steps and
The step of converting the first image into the CIELAB color space so that each pixel is represented as an L ^* value, an a ^* value and a b ^* value.
A method comprising the steps of generating a calibrated image representing a color using the L ^* value, the calibrated a ^* value and the calibrated b ^* value. The method of claim 9, wherein each image in the series of differentially illuminated images corresponds to a different illuminant spectrum. 10. The method of claim 10, wherein each illuminant spectrum represents light of a plurality of colors with one color having a higher intensity than the other colors. The series of differentially illuminated images is captured in conjunction with a series of differential lighting events performed by the multi-channel light source, the multi-channel light source having a plurality of corresponding wavelengths within the electromagnetic spectrum. 9. The method of claim 9, comprising the channel of. 12. The method of claim 12, wherein each channel comprises one or more illuminants configured to produce light of substantially the same color. The step to generate is
A step of converting the set of modified images into a CIELAB color space, such that each pixel is represented by a set of a ^* values and a series of b ^* values.
9. The method of claim 9, comprising registering the series of a ^* values and the series of b ^* values for each pixel in a vector representing the chromaticity fingerprint. 9. The method of claim 9, further comprising causing the display of the calibrated image to occur on an interface generated by the electronic device. 15. The method of claim 15, further comprising allowing an individual to alternate between the calibrated image and the first image on the interface. With an image sensor configured to generate an image from the light collected through the lens,
A light source configured to produce light by assembling at least three color channels, each color channel comprising a light source and one or more illuminants that produce substantially similar colors.
A memory configured to store the calibration matrix generated for the image sensor and the light source, and
It ’s a processor,
A step of receiving an input indicating a request to capture an image of the scene that can be seen through the lens.
A step of allowing the first image of the scene to be captured using the automatic white balance (AWB) setting.
A step of allowing the second image of the scene to be captured using a fixed white balance (FWB) setting.
A step of allowing a series of images to be captured in combination with a series of differential lighting events performed by the light source.
For each pixel, a step of generating a chromaticity fingerprint based on comparing each image in the series of images with the second image.
A step of multiplying each chromaticity fingerprint by the calibration matrix, thereby obtaining the calibrated color value of the corresponding pixel.
A step of calibrating the first image by replacing the color values with the calibrated color values on a pixel-by-pixel basis.
The step of displaying the calibrated first image on the interface, and
Electronic devices, including processors, configured to do. 17. The electron of claim 17, wherein the first image is captured over the first exposure interval and the second image is captured over a second exposure interval that is different from the first exposure interval. device. 17. The electronic device of claim 17, wherein the calibrated color value comprises a calibrated a ^* value and a calibrated b ^* value.

Images