CN116112651A

CN116112651A - White balance processing method and device, electronic equipment and storage medium

Info

Publication number: CN116112651A
Application number: CN202310118363.4A
Authority: CN
Inventors: 雷志燊
Original assignee: Beijing Yihang Yuanzhi Technology Co Ltd
Current assignee: Beijing Yihang Yuanzhi Technology Co Ltd
Priority date: 2023-02-02
Filing date: 2023-02-02
Publication date: 2023-05-12

Abstract

The present disclosure provides a white balance processing method, including: responding to the recognition result that the working mode of the sensor is a high dynamic range acquisition mode, and adjusting the digital gain of the sensor by utilizing the channel gain corresponding to the historical image; performing white balance processing on the multi-frame exposure image by using a sensor to obtain an original image, wherein a transition region in the original image presents a target color; and performing full-image color correction on the original image based on the channel gain of the original image and the color temperature of the target environment in which the sensor is positioned so as to obtain a target image matched with the color temperature. The disclosure also provides a white balance processing device, electronic equipment and a storage medium.

Description

White balance processing method and device, electronic equipment and storage medium

Technical Field

The disclosure relates to the technical field of image processing, and in particular relates to a white balance processing method, a white balance processing device, electronic equipment and a storage medium.

Background

In the field of vehicle-mounted image processing, white balance color cast causes color cast of the edge of a highlight object, influences the detection of a machine vision algorithm on the edge, and further influences the identification of the object, in particular the identification of traffic signs, traffic lights and lane lines; for vehicles with automatic driving equipment, the images of the automobile driver recorder are usually processed by directly using the ISP (Image Signal Processor ) of the front camera, and the user can have poor visual experience when watching the images of the automobile driver recorder with white balance color cast.

Based on this, AWB (Automatic White Balance ) technology is generally utilized to perform AWB processing on an original image acquired by a sensor, so as to obtain r_gain Red channel gain (red_gain) and b_gain (blue_gain) in a target environment corresponding to the original image, and adjust gain parameters in the ISP by utilizing the r_gain and the b_gain, so that the ISP can finally correct the color of the original image, so that the output target image accords with the actual color temperature of the target environment, and the situation that the image has an object edge color cast is avoided.

HDR (High Dynamic Range ) sensors have the ability to render pictures with a larger luminance span and clearer detail, which is more suitable for scenes with higher demands on detail capture capability and luminance span, such as on-board, than linear sensors. However, since the original image obtained by the HDR sensor is a splicing result of multi-frame exposure images with different exposure degrees, the highlighted object presented by the HDR sensor has color edges, and the AWB technology only has color calibration capability, and cannot meet the requirements of color edge recognition, color edge removal and the like, so that object recognition cannot be accurately performed, and better visual experience cannot be provided for users.

Disclosure of Invention

In order to solve at least one of the above technical problems, the present disclosure provides a white balance processing method, a white balance processing device, an electronic apparatus, and a storage medium.

One aspect of the present disclosure provides a white balance processing method, which may include: responding to the recognition result that the working mode of the sensor is a high dynamic range acquisition mode, and adjusting the digital gain of the sensor by utilizing the channel gain corresponding to the historical image; performing white balance processing on a multi-frame exposure image by using the sensor to obtain an original image, wherein a transition region in the original image presents a target color; and carrying out full-image color correction on the original image based on the channel gain of the original image and the color temperature of the target environment where the sensor is positioned so as to obtain a target image matched with the color temperature.

In some embodiments, the performing white balance processing on the multi-frame exposure image by using the sensor to obtain an original image may include: respectively carrying out white balance processing on the exposure images of each frame by the sensor to at least obtain a highlight correction image, a dark area correction image and a general correction image, wherein transition areas in the highlight correction image, the dark area correction image and the general correction image all show the target color; performing feature clipping on the highlight correction image, the dark region correction image, and the general correction image, respectively, to obtain a highlight correction section corresponding to a high-luminance region, a dark region correction section corresponding to a low-luminance region, and a general correction section corresponding to a general luminance region; and splicing the highlight correction segment, the dark correction segment and the general correction segment in sequence to obtain an original image.

In some embodiments, before the adjusting the digital gain of the sensor by using the channel gain corresponding to the historical image in response to the identification result that the working mode of the sensor is the high dynamic range acquisition mode, the method may include: and determining the channel gain of each color channel based on the calibration parameters corresponding to the historical image, wherein the color channels at least comprise a red channel and a blue channel.

In some embodiments, the channel gain is expressed as: cali_x_gain=1/cali_x_g=g_avg/x_avg; wherein cali_x_gain represents the channel gain of the target channel, including at least the channel gain cali_r_gain of the red channel and the channel gain cali_b_gain of the blue channel; cali_x_g represents the calibration parameters of the target channel, and at least comprises the calibration parameters cali_r_g of the red channel and the calibration parameters cali_b_g of the blue channel; g_avg represents the average value of the number of green pixels in the history image; x_avg represents a target color pixel number average value in the history image, and at least comprises a red pixel number average value r_avg and a blue pixel number average value b_avg.

In some embodiments, before the determining the channel gain of each color channel based on the calibration parameter corresponding to the historical image, the method further includes: and determining calibration parameters of the sensor under various standard light sources, wherein the calibration parameters at least comprise a blue channel calibration parameter and a red channel calibration parameter, and the inverse of the calibration parameters is used for representing the channel gain of each color channel.

In some embodiments, the determining calibration parameters of the sensor under various standard light sources includes: shooting an original image of a target gray card under the standard light source, counting initial calibration parameters of the original image of the target gray card, and taking the reciprocal of the initial calibration parameters as an initial channel gain; taking the initial channel gain as an initial digital gain of the sensor; acquiring an original image of a gray scale block by using the sensor, and analyzing a deviation value of the original image of the gray scale block, wherein the deviation value is used for representing the color accuracy of the original image acquired by the sensor, and the deviation value is inversely proportional to the color accuracy; responding to the judgment result that the deviation value is larger than the maximum limit value, and utilizing the deviation value to adjust the initial digital gain so as to obtain a process digital gain; and taking the reciprocal of the process digital gain as a calibration parameter of the sensor under the standard light source until the deviation value is smaller than or equal to the maximum limit value, and storing the calibration parameter into a device memory.

In some embodiments, before the adjusting the digital gain of the sensor by using the channel gain corresponding to the historical image in response to the identification result that the working mode of the sensor is the high dynamic range acquisition mode, the method includes: an operating mode of the sensor is determined, wherein the operating mode includes at least the high dynamic range acquisition mode and a linear acquisition mode.

In some embodiments, after said determining the operation mode of the sensor, comprising: responding to the recognition result that the working mode of the sensor is the linear acquisition mode, and acquiring the channel gain of the original image and the color temperature of the target environment where the sensor is positioned; adjusting the digital gain of the image signal processor by utilizing the channel gain of the original image; based on the color temperature, full-image color correction is performed on the original image by the image signal processor to obtain a target image matching the color temperature.

In some embodiments, before the performing white balance processing on the multi-frame exposure image by using the sensor to obtain an original image, the method includes: and acquiring a plurality of frames of exposure images by using the sensor, wherein the exposure images at least comprise a highlight exposure image for showing details of a high-brightness area, a dark area exposure image for showing details of a low-brightness area and a general exposure image for showing general brightness area.

In some embodiments, before the performing full-image color correction on the original image based on the color temperature of the target environment in which the sensor is located to obtain a target image matching the color temperature, the method includes: compensating the white balance state information to shield the weight of the channel gain of the historical image in the white balance state information and obtain target state information used for representing the white balance state of the target environment; and estimating a color temperature of the target environment based on the target state information.

Another aspect of the present disclosure provides a white balance processing apparatus, which may include: the system comprises a sensor parameter adjustment module, an original image generation module and a full-image color correction module. The sensor parameter adjustment module is used for responding to the recognition result that the working mode of the sensor is a high dynamic range acquisition mode, and adjusting the digital gain of the sensor by utilizing the channel gain corresponding to the historical image; the original image generation module is used for carrying out white balance processing on the multi-frame exposure image by utilizing the sensor so as to obtain an original image, wherein a transition area in the original image presents a target color; and the full-image color correction module is used for carrying out full-image color correction on the original image based on the channel gain of the original image and the color temperature of the target environment where the sensor is positioned so as to obtain a target image matched with the color temperature.

Yet another aspect of the present disclosure provides an electronic device, which may include: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the white balance processing method according to any one of the above embodiments.

Yet another aspect of the present disclosure provides a readable storage medium having stored therein execution instructions which, when executed by a processor, are to implement the white balance processing method according to any one of the above embodiments.

Drawings

The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the disclosure and together with the description serve to explain the principles of the disclosure.

FIG. 1 is a schematic diagram of a related art white balance processing method;

fig. 2 is a flow chart of white balance gain generation of an original image of the related art;

FIG. 3 is a flowchart of a white balance processing method according to an exemplary embodiment of the present disclosure;

FIG. 4 is a schematic diagram of a white balance processing method architecture according to an exemplary embodiment of the present disclosure;

FIG. 5 is a calibration parameter generation flow chart of an exemplary embodiment of the present disclosure;

FIG. 6 is a flow chart of raw image white balance gain generation in accordance with an exemplary embodiment of the present disclosure;

FIG. 7 is a compensation flow chart of white balance status information according to an exemplary embodiment of the present disclosure;

FIG. 8 is a state information analysis flow chart of an exemplary embodiment of the present disclosure; and

Fig. 9 is a schematic diagram of a white balance processing apparatus according to an exemplary embodiment of the present disclosure.

Description of the reference numerals

1000. White balance processing device

1002. Sensor parameter adjustment module

1004. Original image generation module

1006. Full-image color correction module

1100. Bus line

1200. Processor and method for controlling the same

1300. Memory device

1400. Other circuits

Detailed Description

The present disclosure is described in further detail below with reference to the drawings and the embodiments. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant content and not limiting of the present disclosure. It should be further noted that, for convenience of description, only a portion relevant to the present disclosure is shown in the drawings.

In addition, embodiments of the present disclosure and features of the embodiments may be combined with each other without conflict. The technical aspects of the present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

Unless otherwise indicated, the exemplary implementations/embodiments shown are to be understood as providing exemplary features of various details of some ways in which the technical concepts of the present disclosure may be practiced. Thus, unless otherwise indicated, features of the various implementations/embodiments may be additionally combined, separated, interchanged, and/or rearranged without departing from the technical concepts of the present disclosure.

The use of cross-hatching and/or shading in the drawings is typically used to clarify the boundaries between adjacent components. As such, the presence or absence of cross-hatching or shading does not convey or represent any preference or requirement for a particular material, material property, dimension, proportion, commonality between illustrated components, and/or any other characteristic, attribute, property, etc. of a component, unless indicated. In addition, in the drawings, the size and relative sizes of elements may be exaggerated for clarity and/or descriptive purposes. While the exemplary embodiments may be variously implemented, the specific process sequences may be performed in a different order than that described. For example, two consecutively described processes may be performed substantially simultaneously or in reverse order from that described. Moreover, like reference numerals designate like parts.

When an element is referred to as being "on" or "over", "connected to" or "coupled to" another element, it can be directly on, connected or coupled to the other element or intervening elements may be present. However, when an element is referred to as being "directly on," "directly connected to," or "directly coupled to" another element, there are no intervening elements present. For this reason, the term "connected" may refer to physical connections, electrical connections, and the like, with or without intermediate components.

The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, when the terms "comprises" and/or "comprising," and variations thereof, are used in the present specification, the presence of stated features, integers, steps, operations, elements, components, and/or groups thereof is described, but the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof is not precluded. It is also noted that, as used herein, the terms "substantially," "about," and other similar terms are used as approximation terms and not as degree terms, and as such, are used to explain the inherent deviations of measured, calculated, and/or provided values that would be recognized by one of ordinary skill in the art.

FIG. 1 is a schematic diagram of a related art white balance processing method; fig. 2 is a flow chart of white balance gain generation of an original image of the related art. Before describing the white balance processing method of the present disclosure, a related art white balance processing method and its drawbacks will be briefly described with reference to fig. 1 to 2.

The white balance processing method of the related art is mainly proposed for solving the color cast problem of an original image acquired by a linear sensor, and the main flow is as follows: acquiring 12-to 16-bit raw images by a sensor; preprocessing the original image with an ISP (Image Signal Processor ) to obtain a preprocessed original image; the preprocessed original image is simultaneously sent to a white balance gain processing module and an AWB (Automatic White Balance ) module of the ISP, the AWB module carries out AWB statistics on the preprocessed original image to obtain AWB state information of the preprocessed original image, and the AWB state information of the original image is analyzed by utilizing an AWB core algorithm of the AWB state information to obtain white balance gain corresponding to the original image; the ISP performs white balance processing on the original image after the preprocessing with the white balance gain to obtain a target image.

The linear sensor only collects one frame of image under a certain illumination environment and takes the frame of image as an original image. Because the original image output by the linear sensor is not a fitting image, at least the transition area presented by the linear sensor has no color edge, and the target image can be obtained by carrying out full-image color correction on the image by using ISP.

The white balance processing belongs to one of the processing procedures of the original image, and before the white balance processing is performed, a series of preprocessing including dark angle correction, dead point correction, exposure control and the like is performed on the original image, so that the related implementation of the preprocessing is not described.

The AWB module has the ability to count AWB state information and also has the ability to determine CCT (Correlated Color Temperature, relative color temperature) and white balance gain using a core algorithm. The following briefly describes the flow of an AWB core algorithm of the AWB module of the related art with reference to fig. 2, where the AWB module performs state information analysis according to the AWB state information of the preprocessed original image, and further performs CCT estimation according to the result of the state information analysis, so as to determine the CCT of the illumination environment corresponding to the original image; further, by combining the result of the state information analysis, the CCT of the illumination environment corresponding to the original image and the calibration parameters of the sensor under different standard light sources, the channel gain (namely, white balance gain) corresponding to the original image can be determined by calculating the white balance gain of the original image, wherein the channel gain can be red channel gain R_gain and blue channel gain B_gain; furthermore, R_gain and B_gain are applied to an R_gain register and a B_gain register in the ISP, and CCT is synchronized to the ISP at the same time, so that the ISP white balance gain adjustment is realized, and finally the adjusted white balance gain of the ISP is obtained.

Obviously, the white balance processing method shown in the related art can only be suitable for the situation that the linear sensor collects one frame of image as the original image without color edges, and cannot deal with the problem of color edges of the original image formed by fitting multiple frames, which is generated by the HDR (High Dynamic Range ) sensor. In addition, the illumination environment in which the sensor is located at any moment has unique CCT, the CCT is used as the input condition of different functional modules on the ISP, the functional modules can call different functional parameters under the influence of the different CCTs, the functional parameters can be a color transformation matrix, and if the CCT is inaccurate, the color accuracy of the image output by the ISP is also deteriorated; the functional parameters may also be, for example, lens vignetting correction parameters, and if CCT is inaccurate, the image output by the ISP may have a color cast vignetting or a center color cast condition. In the related art, the digital gain of the sensor is not adjusted, so that the AWB module does not consider the masking processing of the weight of the historical channel gain when executing the AWB core algorithm; however, for the architecture for adjusting the digital gain of the sensor proposed in the present disclosure, the processing mode is affected by the weight of the historical channel gain, and the obtained CCT cannot represent the light source characteristic of the current illumination environment, so the current AWB core algorithm is not suitable for the overall architecture of the white balance processing method proposed in the present disclosure.

The disclosure provides a white balance processing method to solve the problem that the related art cannot process the color edge of an original image output by an HDR sensor; meanwhile, in order to adapt to the whole architecture of the present disclosure, part of the data processing flow in the related art is also optimized.

FIG. 3 is a flowchart of a white balance processing method according to an exemplary embodiment of the present disclosure; FIG. 4 is a schematic diagram of a white balance processing method architecture according to an exemplary embodiment of the present disclosure; FIG. 5 is a calibration parameter generation flow chart of an exemplary embodiment of the present disclosure; FIG. 6 is a flow chart of raw image white balance gain generation in accordance with an exemplary embodiment of the present disclosure; FIG. 7 is a compensation flow chart of white balance status information according to an exemplary embodiment of the present disclosure; fig. 8 is a state information analysis flowchart of an exemplary embodiment of the present disclosure. The white balance processing method S100 proposed in the present disclosure will be explained below with reference to fig. 3 to 8.

Step S102, in response to the recognition result that the working mode of the sensor is the high dynamic range acquisition mode, the digital gain of the sensor is adjusted by utilizing the channel gain corresponding to the historical image.

Among them, the sensor is a device for capturing an image, which may have different operation modes, such as a high dynamic range capturing mode and a linear capturing mode, according to the characteristics of the sensor itself. Under different working modes, the sensor can be matched with different white balance processing modes, that is, the method is also suitable for white balance processing of the original image acquired by the linear sensor.

The high dynamic range acquisition mode refers to that the sensor is an HDR sensor, which has the capability of simultaneously displaying details of a highlight portion and a shadow portion in one image, and the HDR sensor is more suitable for the recognition of traffic signs, traffic lights, lane lines, and the like in the vehicle-mounted field, and the recording and acquisition of a vehicle recorder, so that the HDR sensor will become a necessary trend as an image acquisition device in the vehicle-mounted field.

The HDR sensor acquires a plurality of exposure images before outputting an original image, wherein the original image is a fitting result of the plurality of exposure images. If color cast exists in each frame of exposure image, color edges can appear in the transition area of two adjacent frames of images after fitting. Therefore, when the working mode of the sensor is identified as a high dynamic range acquisition mode, the digital gain of the sensor is adjusted, so that the color cast problem of each exposure image can be solved before fitting, and the effect that the color edges do not appear in the transition area of the fitting result is achieved.

The exposure image at least comprises a highlight exposure image corresponding to a high-brightness area, a dark area exposure image corresponding to a low-brightness area and a general exposure image corresponding to a general brightness area according to the actual condition of the current illumination environment. The HDR sensor controls the exposure intensity by adjusting the exposure time length so as to realize the presentation of details and contents of objects with various brightness. Specifically, the exposure time length can show the details and the contents of the high-brightness area, the exposure time length can show the details and the contents of the low-brightness area, and the exposure time length is moderate and can show the details and the contents of the general-brightness area.

The history image is a previous output result of the original image output by the sensor, and is adjacent to the original image in output timing. The channel gain corresponding to the historical image is a white balance gain obtained based on the historical image, including a red channel gain and a blue channel gain. Based on the foregoing, in the related art, the current original image is generally used for adjusting the white balance gain of the ISP, and the method uses the channel gain obtained in the previous time to adjust the digital gain of the sensor, so that the balance of three channels of RGB is achieved before the original image is fitted, and therefore, the situation that a transition region (especially the edge of a highlight object) is color-deviated or colored in the fitted original image is avoided, and the problem of the color edge of the original image is fundamentally avoided.

The digital gain of the sensor refers to a parameter of the sensor that performs white balance processing on an image, and is used for characterizing the light source characteristics of the illumination environment.

Step S104, performing white balance processing on the multi-frame exposure image by using the sensor to obtain an original image.

The sensor performs white balance processing on the multi-frame exposure image, so that a transition region in the original image presents a target color.

Specifically, the implementation manner of step S104 may be: respectively carrying out white balance processing on each frame of exposure image by a sensor to at least obtain a highlight correction image, a dark area correction image and a general correction image, wherein the transition areas in the highlight correction image, the dark area correction image and the general correction image all show target colors; performing feature clipping on the highlight correction image, the dark correction image, and the general correction image, respectively, to obtain a highlight correction section corresponding to a high-luminance region, a dark correction section corresponding to a low-luminance region, and a general correction section corresponding to a general luminance region; and splicing the highlight correction segment, the dark correction segment and the general correction segment in sequence to obtain an original image.

In other words, the obtained exposure images of each frame are subjected to white balance correction before fitting to overcome the problem of color cast, then the corresponding parts of the correction images matched with the exposure degree are cut, and finally the cut fragments are spliced into a complete image according to the position of an actual scene. Based on this, the transition region between two adjacent segments does not have a color edge, and details and contents at various brightness can be clearly presented.

The target color represents the color matched with the actual illumination environment and is regulated and controlled by the digital gain of the sensor; then when the white portion of each exposure image can assume a true white state, then no color edges of the transition region will appear when the stitched edges of the corresponding segments of each exposure image fit.

Step S106, based on the channel gain of the original image and the color temperature of the target environment where the sensor is located, performing full-image color correction on the original image to obtain a target image matched with the color temperature.

The channel gain (i.e. white balance gain) of the original image is essentially the correction factor of the corresponding color channel, including the red channel gain and the blue channel gain, and is used to adjust the average value of the pixel numbers of various colors in the original image.

The color temperature CCT refers to the actual light source color of the light source corresponding to the target environment, and when the target environment is fixed, the corresponding CCT should be unique and fixed.

Due to the limitation of equipment, the acquired original image has color cast problem, namely, the CCT of the shot image deviates from the actual CCT seen by human eyes. Based on this, it is necessary to perform color correction on the original image using the channel gain acquired from the original image and the CCT of the target environment (i.e., the actual illumination environment) to restore the actual light source color of the target environment, thereby obtaining a target image matching the CCT of the target environment.

In some embodiments, before step S102, the method may further include: and determining the channel gain of each color channel based on the calibration parameters corresponding to the historical image, wherein the color channels at least comprise a red channel and a blue channel.

The calibration parameters corresponding to the historical images are the number of pixels, the deviation value of which is smaller than or equal to the maximum limit value, between the original images and the illumination environment, and the inverse of the pixel is the channel gain corresponding to the historical images. Since the adjustment of the digital gain of the sensor is performed based on the channel gain of the history image, it is necessary to determine the channel gain corresponding to the history image before step S102. The calibration parameters for the different light sources are different, and some preparation steps exist before this step to determine the calibration parameters of the respective standard light sources (which will be described in the following.

The channel gain can be expressed as:

cali_x_gain＝1/cali_x_g＝g_avg/x_avg；

wherein cali_x_gain represents the channel gain of the target channel, including at least the channel gain cali_r_gain of the red channel and the channel gain cali_b_gain of the blue channel; cali_x_g represents the calibration parameters of the target channel, and at least comprises the calibration parameters cali_r_g of the red channel and the calibration parameters cali_b_g of the blue channel; g_avg represents the average value of the number of green pixels in the history image; x_avg represents a target color pixel number average in the history image, and at least comprises a red pixel number average r_avg and a blue pixel number average b_avg.

In some embodiments, before determining the channel gain of each color channel based on the calibration parameters corresponding to the historical image, the method further includes: and determining calibration parameters of the sensor under various standard light sources, wherein the calibration parameters at least comprise a blue channel calibration parameter and a red channel calibration parameter, and the inverse of the calibration parameters is used for representing the channel gain of each color channel.

Standard light sources are artificial light sources that simulate various ambient light conditions, allowing production plants or off-site environments such as a room to achieve substantially consistent lighting effects with the light sources in a particular environment. The standard light sources may be H, A, TL, D50, D65, D75, U30, etc., all of which are color temperature designations of the standard light sources.

Different calibration parameter determining processes are adopted for different working modes of the sensor, and the calibration parameter determining processes under various working modes are described below with reference to fig. 5.

When the working mode of the sensor is a linear acquisition mode, the process of determining the calibration parameters is as follows: judging whether the working mode of the sensor is an HDR acquisition mode, if the working mode is a non-HDR acquisition mode, setting pre-HDR gain=0 of the sensor, wherein the process is equivalent to a calibration parameter instruction for issuing a linear acquisition mode. Further, 18% gray cards are shot by using a sensor under each standard light source, so that the image cards are positioned in the middle of the image and occupy 50% of the area of the image; the exposure time of the sensor is adjusted to enable the image to be exposed to be in a normal state, and then the original image of any scene under the irradiation of the standard light source is shot; and counting the number of red pixels and the number of blue pixels in the area, accounting for 10% of the original image field, of the center of the original image, and obtaining a calibration parameter cali_r_g of a red channel and a calibration parameter cali_b_g of a blue channel.

Wherein cali_r_gain=r_avg/g_avg; cali_b_gain=b_avg/g_avg.

Where i is the sequence number of the region of interest in the image captured by the sensor, ri represents the number of red pixels in the ith region of interest, N is the number of all pixels in the region of interest (N meets the requirement that the sum of N and 4 is 0, i.e., N mod 4=0), and bi is the number of blue pixels in the ith region of interest; g_avg is the average value of the number of green pixels, gri represents the total amount of red and green pixels in the ith region of interest, gbi represents the total amount of blue and green pixels in the ith region of interest.

And finally, writing the calibration parameters cali_r_g and cali_b_g into the memory of the sensor device.

When the working mode of the sensor is a high dynamic range acquisition mode, the process of determining the calibration parameters is as follows: judging whether the working mode of the sensor is an HDR acquisition mode, if so, setting pre-HDR gain=1 (RGB three channels are all set to be 1) of the sensor, wherein the process is equivalent to a calibration parameter instruction issued to the HDR acquisition mode. Furthermore, each standard light source is set, and an original image of 18% gray card is shot under the irradiation of each standard light source, so that the calibration parameters cali_r_g of the red channel and the calibration parameters cali_b_g of the blue channel are determined according to the acquisition flow of the calibration parameters of the linear acquisition mode, and the description is omitted.

The difference is that, after cali_r_g and cali_b_g are acquired, their inverses are taken: cali_r_gain=1/cali_r_g, cali_b_gain=1/cali_b_g to obtain the channel gain of the corresponding channel and set to the corresponding digital gain pre-HDR gain in the HDR sensor. When the HDR sensor synthesizes the original images, the synthesis algorithm built in the sensor synthesizes the multi-frame exposure images in the sensor, the synthesized original images have nonlinear parts, then the gains obtained through reciprocal are written into the digital gains of the HDR sensor, and the white balance state of the original images may deviate from the corresponding standard light sources. Based on the above, the white balance result of the digital gain needs to be verified, and the digital gain which does not meet the expectations is finely adjusted, so that the accuracy of the calibration parameters is ensured.

The specific process of verification comprises the following steps: under the selected standard light source, the original image of the alice standard 24 color card is shot, so that the image card is positioned in the center of the original image and occupies 75% of the field of view of the original image. The deviation value deltaC of the gray-scale blocks (i.e., 20 to 23 gray-scale blocks of the 24 color chart) in the original image is analyzed, wherein when a plurality of gray-scale blocks are selected (one gray-scale block can be selected), deltaC can be the average value or weighted average value of the deviation values of the gray-scale blocks. Further, a maximum limit value of deltaC is set, the smaller deltaC is, the higher the color accuracy is proved, and therefore, when deltaC is smaller than or equal to the maximum limit value, the white balance result has credibility; otherwise, the digital gain of the sensor needs to be adjusted, namely, the 24 color chart is repeatedly shot, so that the deviation value of the gray scale blocks in the original chart is smaller than or equal to the maximum limit value. deltaC typically takes a value of 0 to 5. Finally, when deltaC is smaller than or equal to the maximum limit value, taking the reciprocal of the optimized digital gain as the white balance calibration parameter of the HDR sensor under the standard light source. And switching other standard light sources, repeating the process to obtain the calibration parameters corresponding to the standard light sources, and storing the calibration parameters into the equipment memory.

Specifically, shooting an original image of a target gray card under a standard light source, counting initial calibration parameters of the original image of the target gray card, and taking the reciprocal of the initial calibration parameters as an initial channel gain; taking the initial channel gain as the initial digital gain of the sensor; collecting an original image of a gray scale block by using a sensor, and analyzing a deviation value of the original image of the gray scale block, wherein the deviation value is used for representing the color accuracy of the original image collected by the sensor, and the deviation value is inversely proportional to the color accuracy; responding to the judgment result that the deviation value is larger than the maximum limit value, and adjusting the initial digital gain by utilizing the deviation value to obtain a process digital gain; and taking the reciprocal of the process digital gain as a calibration parameter of the sensor under a standard light source until the deviation value is smaller than or equal to the maximum limit value, and storing the reciprocal of the process digital gain into a device memory.

In some embodiments, prior to step S102, comprising: an operating mode of the sensor is determined, wherein the operating mode includes at least a high dynamic range acquisition mode and a linear acquisition mode.

As different working modes of the sensor can be matched with different white balance processing modes, the method provides a step of identifying the working modes, and can be seen that the method also supports the white balance processing modes of the linear acquisition modes of the related technology.

In some embodiments, a channel gain of an original image and a color temperature of a target environment in which the sensor is located are obtained in response to an identification result that a working mode of the sensor is a linear acquisition mode; adjusting the digital gain of the image signal processor by using the channel gain of the original image; based on the color temperature, full-image color correction is performed on the original image with an image signal processor to obtain a target image matching the color temperature.

In some embodiments, before step S104, further comprising: a plurality of frames of exposure images are acquired by a sensor, wherein the exposure images at least comprise a highlight exposure image for presenting high brightness area details, a dark area exposure image for presenting low brightness area details and a general exposure image for presenting general brightness areas.

This step is generated based on the characteristics of the HDR sensor and will not be described in detail here.

In some embodiments, after step S104, further comprising: compressing the original image by a sensor to form an original image compression packet with the size of 12 bits to 16 bits; prior to processing with ISP, the compressed original image compression packet needs to be decompressed to obtain the original image using 20-bit or 24-bit to 16-bit look-up table mapping; further, the ISP performs preprocessing, which may include dark angle correction, dead spot correction, exposure control, etc., and will not be described in detail herein. After the preprocessed original image is obtained, AWB statistics of the original image is carried out by utilizing an AWB module, and then AWB state information of the original image is obtained; meanwhile, the original image after the preprocessing is also transmitted to the ISP white balance gain unit for subsequent white balance processing. The foregoing processes are similar to the white balance processing method in the related art, and reference is made to the foregoing, and details are not repeated.

The method is different from the related technology in that: the AWB module performs the flow of the AWB core algorithm. Specifically, as shown in fig. 6, a step of compensating for the white balance status information is newly added, that is, before step S106, the following steps are further provided: compensating the white balance state information to shield the weight of the channel gain of the historical image in the white balance state information and obtain target state information used for representing the white balance state of the target environment; and estimating a color temperature of the target environment based on the target state information.

In particular, from the principle that the AWB module performs AWB core algorithm, this process can be regarded as a linear process. Assuming that I is an image processed by the AWB module, i_raw is an original image, and G is a shorthand for gain output by the AWB module, i=i_raw. From this equation, it can be seen that I arrives at the ISP input, which can be equivalently the product of the original image i_raw reflecting the current lighting environment and the digital gain pre-HDR gain (i.e., G) calculated from the previous frame. The purpose of the white balance state information compensation is therefore to compensate back for the pre-HDR gain of the previous frame, i.e. to reject the interference of the pre-HDR gain of the previous frame on the color temperature presented by the current image. The compensated image can reflect the current illumination environment. Compensation can directly act on white balance statistics without additional hardware support, and only the underlying software needs to be modified, which can be supported by the general ISP architecture.

More specifically, after the compensated white balance state information is obtained, the state information is analyzed to obtain weighted white balance state information, and CCT estimation is performed using the weighted white balance state information; further, white balance gain calculation is performed based on the CCT estimation value and the weighted white balance state information, and the result of the white balance gain calculation (the channel gain of the original image) is synchronized to the white balance state information compensation unit and written into the HDR sensor; and finally, performing ISP white balance gain adjustment on the ISP by using the CCT estimated value and the white balance gain calculation result so as to write the adjusted white balance gain into a register of the ISP.

Specifically, the white balance state information compensation is realized by a white balance state information compensation unit whose main function is to compensate for currently input AWB state information. Because the method controls the digital gain pre-HDR gain of the sensor, the output original image cannot reflect the white balance state of the current illumination environment, and the target state information obtained after the white balance state information compensation can reflect the current white balance state. Specific implementation flow referring to fig. 7, an awb_gain structure is defined, and when the AWB module is executed for the first time, pre_gain and cur_gain are both 1.000. Defining an AWB_proteins (namely white balance state information) structure, and calculating compensated target state information as follows:

compst_awb_stats.r_g＝raw_awb_stats.r_g/pre_gain.r_gain；

compst_awb_stats.b_g＝raw_awb_stats.b_g/pre_gain.b_gain；

sensor_gain_en is the enable bit of the digital gain before HDR synthesis (sensor pre-HDR gain): if 1, sensor pre-HDR gain is enabled, pre_gain=pre_gain is cur_gain, wherein "=" right pre_gain is the previous channel gain, the product of the previous channel gain cur_gain and the current channel gain cur_gain is taken as a new pre_gain (i.e., "=" left pre_gain "), and the new pre_gain is used as the compensation (/ pre_gain) of the next AWB state information; if not 1, the sensor pre-HDR gain is not enabled, at which time the pre_gain and cur_gain remain at default values, that is, when sensor_gain_en=0, the control method is applicable to linear sensors.

Specifically, during initialization, the internal program code of the white balance status information may be expressed as:

after reading the gain pre_gain, the white balance status information is compensated, i.e. the parameters are processed by "/pre_gain" to obtain the target status information, and the internal program code can be expressed as:

judging whether sensor_gain_en=1 (namely the enabling bit of the sensor) is 1, if not, not adjusting pre_gain and cur_gain to keep a default value; if 1, the cur_gain of the write sensor is read. Further, the pre_gain of the next frame is assigned a value using the product of pre_gain and cur_gain to obtain a new pre_gain for use in the next compensation, where pre_gain=pre_gain.

The internal program code of the new pre_gain can be expressed as:

pre_gain.r_gain＝pre_gain.r_gain*cur_gain.r_gain；

pre_gain.b_gain＝pre_gain.b_gain*cur_gain.b_gain；

specifically, the flow of performing state analysis on the compensated target state information may refer to fig. 8, where the generated weighted white balance state information can represent state point information of the current illumination environment, and is used as input of subsequent white balance gain and CCT estimation. When the state analysis is carried out, the output result is a series of state point information which is output after brightness weighting, gray area weighting and CCT weighting and weight values taken by each CCT interval.

More specifically, the step of obtaining the weighted white balance status information of each CCT section is: acquiring compensated white balance state information, and removing pixels in overexposure and overexposure areas in a sample; removing non-gray pixels from the sample according to the gray region boundary parameters; according to the gray area weight parameters, gray area pixels are weighted; counting white balance state information of each CCT interval according to white balance calibration parameters under each light source; generating white balance state information of each CCT section; and carrying out illumination weighting on the white balance state information according to the illumination weight parameters, and finally obtaining the white balance state information of each weighted CCT interval. The implementation manner directly adopts the prior art, and a more specific flow is not repeated.

The white balance processing method is suitable for an HDR sensor and a linear sensor, and solves the problem of color edges of images generated by the HDR sensor; in addition, based on the proposed architecture for the HDR sensor, a step of white balance compensation is added to match the final output target image with the color temperature of the target environment.

As shown in fig. 9, the present disclosure proposes a white balance processing apparatus 1000, which may include: a sensor parameter adjustment module 1002, an original image generation module 1004, and a full-map color correction module 1006. The sensor parameter adjustment module 1002 is configured to adjust a digital gain of the sensor by using a channel gain corresponding to the historical image in response to a recognition result that the working mode of the sensor is a high dynamic range acquisition mode; the original image generating module 1004 is configured to perform white balance processing on the multi-frame exposure image by using a sensor, so as to obtain an original image, where a transition area in the original image presents a target color; and a full-image color correction module 1006 is configured to perform full-image color correction on the original image based on the channel gain of the original image and the color temperature of the target environment in which the sensor is located, so as to obtain a target image matching the color temperature.

The respective modules of the white balance processing apparatus 1000 are provided for performing the respective steps of the white balance processing method, and the implementation principle and the performing steps thereof are referred to the foregoing and will not be described again.

The apparatus may include corresponding modules that perform the steps of the flowcharts described above. Thus, each step or several steps in the flowcharts described above may be performed by respective modules, and the apparatus may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform the respective steps, or be implemented by a processor configured to perform the respective steps, or be stored within a computer-readable medium for implementation by a processor, or be implemented by some combination.

The hardware architecture may be implemented using a bus architecture. The bus architecture may include any number of interconnecting buses and bridges depending on the specific application of the hardware and the overall design constraints. Bus 1100 connects together various circuits including one or more processors 1200, memory 1300, and/or hardware modules. Bus 1100 may also connect various other circuits 1400, such as peripherals, voltage regulators, power management circuits, external antennas, and the like.

Bus 1100 may be an industry standard architecture (ISA, industry Standard Architecture) bus, a peripheral component interconnect (PCI, peripheral Component) bus, or an extended industry standard architecture (EISA, extended Industry Standard Component) bus, among others. The buses may be divided into address buses, data buses, control buses, etc. For ease of illustration, only one connection line is shown in the figure, but not only one bus or one type of bus.

The white balance processing device is suitable for an HDR sensor and a linear sensor, and solves the problem of color edges of images generated by the HDR sensor; in addition, based on the proposed architecture for the HDR sensor, a step of white balance compensation is added to match the final output target image with the color temperature of the target environment.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present disclosure in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the embodiments of the present disclosure. The processor performs the various methods and processes described above. For example, method embodiments in the present disclosure may be implemented as a software program tangibly embodied on a machine-readable medium, such as a memory. In some embodiments, part or all of the software program may be loaded and/or installed via memory and/or a communication interface. One or more of the steps of the methods described above may be performed when a software program is loaded into memory and executed by a processor. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above in any other suitable manner (e.g., by means of firmware).

Logic and/or steps represented in the flowcharts or otherwise described herein may be embodied in any readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.

For the purposes of this description, a "readable storage medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM). In addition, the readable storage medium may even be paper or other suitable medium on which the program can be printed, as the program can be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner if necessary, and then stored in a memory.

It should be understood that portions of the present disclosure may be implemented in hardware, software, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

Those of ordinary skill in the art will appreciate that all or part of the steps implementing the method of the above embodiments may be implemented by a program to instruct related hardware, and the program may be stored in a readable storage medium, where the program when executed includes one or a combination of the steps of the method embodiments.

Furthermore, each functional unit in each embodiment of the present disclosure may be integrated into one processing module, or each unit may exist alone physically, or two or more units may be integrated into one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules may also be stored in a readable storage medium if implemented in the form of software functional modules and sold or used as a stand-alone product. The storage medium may be a read-only memory, a magnetic disk or optical disk, etc.

The present disclosure also provides an electronic device, including: a memory storing execution instructions; and a processor or other hardware module that executes the memory-stored execution instructions such that the processor or other hardware module performs the compensation method of the white balance gain.

The present disclosure also provides a readable storage medium having stored therein execution instructions that when executed by a processor are to implement a compensation method for white balance gain.

In the description of the present specification, a description referring to the terms "one embodiment/mode," "some embodiments/modes," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment/mode or example is included in at least one embodiment/mode or example of the present disclosure. In this specification, the schematic representations of the above terms are not necessarily the same embodiments/modes or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments/modes or examples. Furthermore, the various embodiments/implementations or examples described in this specification and the features of the various embodiments/implementations or examples may be combined and combined by persons skilled in the art without contradiction.

Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present disclosure, the meaning of "a plurality" is at least two, such as two, three, etc., unless explicitly specified otherwise.

It will be appreciated by those skilled in the art that the above-described embodiments are merely for clarity of illustration of the disclosure, and are not intended to limit the scope of the disclosure. Other variations or modifications will be apparent to persons skilled in the art from the foregoing disclosure, and such variations or modifications are intended to be within the scope of the present disclosure.

Claims

1. A white balance processing method, comprising:

responding to the recognition result that the working mode of the sensor is a high dynamic range acquisition mode, and adjusting the digital gain of the sensor by utilizing the channel gain corresponding to the historical image;

performing white balance processing on a multi-frame exposure image by using the sensor to obtain an original image, wherein a transition region in the original image presents a target color; and

And carrying out full-image color correction on the original image based on the channel gain of the original image and the color temperature of the target environment where the sensor is positioned so as to obtain a target image matched with the color temperature.

2. The white balance processing method according to claim 1, wherein the performing white balance processing on the multi-frame exposure image with the sensor to obtain an original image includes:

respectively carrying out white balance processing on the exposure images of each frame by the sensor to at least obtain a highlight correction image, a dark area correction image and a general correction image, wherein transition areas in the highlight correction image, the dark area correction image and the general correction image all show the target color;

performing feature clipping on the highlight correction image, the dark region correction image, and the general correction image, respectively, to obtain a highlight correction section corresponding to a high-luminance region, a dark region correction section corresponding to a low-luminance region, and a general correction section corresponding to a general luminance region;

and splicing the highlight correction segment, the dark correction segment and the general correction segment in sequence to obtain an original image.

3. The white balance processing method according to claim 1, wherein before the step of adjusting the digital gain of the sensor by using the channel gain corresponding to the history image in response to the recognition result that the operation mode of the sensor is the high dynamic range collection mode, the method comprises:

and determining the channel gain of each color channel based on the calibration parameters corresponding to the historical image, wherein the color channels at least comprise a red channel and a blue channel.

4. A white balance processing method according to claim 3, wherein the channel gain is expressed as:

cali_x_gain＝1/cali_x_g＝g_avg/x_avg；

wherein cali_x_gain represents the channel gain of the target channel, including at least the channel gain cali_r_gain of the red channel and the channel gain cali_b_gain of the blue channel; cali_x_g represents the calibration parameters of the target channel, and at least comprises the calibration parameters cali_r_g of the red channel and the calibration parameters cali_b_g of the blue channel; g_avg represents the average value of the number of green pixels in the history image; x_avg represents a target color pixel number average value in the history image, and at least comprises a red pixel number average value r_avg and a blue pixel number average value b_avg.

5. The white balance processing method according to claim 3, further comprising, before the determining the channel gain of each color channel based on the calibration parameter corresponding to the history image:

and determining calibration parameters of the sensor under various standard light sources, wherein the calibration parameters at least comprise a blue channel calibration parameter and a red channel calibration parameter, and the inverse of the calibration parameters is used for representing the channel gain of each color channel.

6. The method of claim 5, wherein determining calibration parameters of the sensor under various standard light sources comprises:

shooting an original image of a target gray card under the standard light source, counting initial calibration parameters of the original image of the target gray card, and taking the reciprocal of the initial calibration parameters as an initial channel gain;

taking the initial channel gain as an initial digital gain of the sensor;

acquiring an original image of a gray scale block by using the sensor, and analyzing a deviation value of the original image of the gray scale block, wherein the deviation value is used for representing the color accuracy of the original image acquired by the sensor, and the deviation value is inversely proportional to the color accuracy;

Responding to the judgment result that the deviation value is larger than the maximum limit value, and utilizing the deviation value to adjust the initial digital gain so as to obtain a process digital gain; and

and taking the reciprocal of the process digital gain as a calibration parameter of the sensor under the standard light source and storing the calibration parameter into a device memory until the deviation value is smaller than or equal to the maximum limit value.

7. The white balance processing method according to claim 1, wherein before the step of adjusting the digital gain of the sensor by using the channel gain corresponding to the history image in response to the recognition result that the operation mode of the sensor is the high dynamic range collection mode, the method comprises:

an operating mode of the sensor is determined, wherein the operating mode includes at least the high dynamic range acquisition mode and a linear acquisition mode.

8. A white balance processing apparatus, comprising:

the sensor parameter adjusting module is used for responding to the recognition result that the working mode of the sensor is a high dynamic range acquisition mode, and adjusting the digital gain of the sensor by utilizing the channel gain corresponding to the historical image;

the original image generation module is used for carrying out white balance processing on the multi-frame exposure image by utilizing the sensor so as to obtain an original image, wherein a transition area in the original image presents a target color; and

And the full-image color correction module is used for carrying out full-image color correction on the original image based on the channel gain of the original image and the color temperature of the target environment where the sensor is positioned so as to obtain a target image matched with the color temperature.

9. An electronic device, comprising:

a memory storing execution instructions; and

a processor that executes the execution instructions stored in the memory, so that the processor executes the white balance processing method according to any one of claims 1 to 7.

10. A readable storage medium, wherein execution instructions are stored in the readable storage medium, which when executed by a processor are for implementing the white balance processing method of any one of claims 1 to 7.