CN115314695A

CN115314695A - Image white balance processing method and device, electronic equipment and storage medium

Info

Publication number: CN115314695A
Application number: CN202210934724.8A
Authority: CN
Inventors: 张科武
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2022-08-04
Filing date: 2022-08-04
Publication date: 2022-11-08

Abstract

The embodiment of the disclosure relates to an image white balance processing method and device, an electronic device and a storage medium, and relates to the technical field of images, wherein the image white balance processing method comprises the following steps: acquiring an image to be processed, and determining the global white balance gain of the image to be processed; partitioning the image to be processed to obtain a plurality of image blocks, and determining local color information of each image block through a color sensor array; and determining a target white balance gain of each image block according to the global white balance gain and the local color information, and performing white balance processing on the image blocks according to the target white balance gain to generate a target image corresponding to the image to be processed. According to the technical scheme in the embodiment of the disclosure, the accuracy of white balance can be improved.

Description

Image white balance processing method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of video technologies, and in particular, to a method and an apparatus for processing white balance of an image, an electronic device, and a computer-readable storage medium.

Background

In the image processing process, image white balance processing is a common way to improve image quality. The image White Balance processing may be realized by an AWB (Auto White Balance) algorithm.

In the related art, white balance processing is generally performed by outputting a white balance gain of global action through global statistics, and an actual color temperature is calculated by performing interpolation calculation using different color temperatures by calibrating a light source in advance. In this way, there is a certain limitation and there is a deviation, so that the white balance effect is poor, and the image quality is reduced.

It is noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure and therefore may include information that does not constitute prior art that is already known to a person of ordinary skill in the art.

Disclosure of Invention

An object of the present disclosure is to provide an image white balance processing method and apparatus, an electronic device, and a computer-readable storage medium, which overcome, at least to some extent, the problem of poor white balance effect due to the limitations and disadvantages of the related art.

Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.

According to a first aspect of the present disclosure, there is provided an image white balance processing method including: acquiring an image to be processed, and determining the global white balance gain of the image to be processed; partitioning the image to be processed to obtain a plurality of image blocks, and determining local color information of each image block through a color sensor array; and determining a target white balance gain of each image block according to the global white balance gain and the local color information, and performing white balance processing on the image blocks according to the target white balance gain to generate a target image corresponding to the image to be processed.

According to a second aspect of the present disclosure, there is provided an image white balance processing apparatus comprising: the global acquisition module is used for acquiring an image to be processed and determining the global white balance gain of the image to be processed; the local color determining module is used for partitioning the image to be processed to obtain a plurality of image blocks and determining local color information of each image block through a color sensor array; and the image processing module is used for determining the target white balance gain of each image block according to the global white balance gain and the local color information, and performing white balance processing on the image blocks according to the target white balance gain to generate a target image corresponding to the image to be processed.

According to a third aspect of the present disclosure, there is provided an electronic device comprising: an image module including a color sensor array; a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the image white balance processing method of the first aspect and possible implementations thereof via execution of the executable instructions.

According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the image white balance processing method of the first aspect described above and possible implementations thereof.

In the technical scheme provided by the embodiment of the disclosure, the global white balance gain of an image to be processed is determined, and the local color information of an image block is determined through a color sensor array; and further determining the target white balance gain of each image block according to the global white balance gain and the local color information so as to perform white balance processing on the image blocks. On one hand, the local color information of each image block can be determined according to the color sensor array, the target white balance gain of each image block is further determined according to the global white balance gain and the local color information, and the global white balance gain can be adjusted through the local color information of the color sensor array, so that the independent control of each image block is realized, the limitation that only the whole process can be carried out in the related technology is avoided, and the accuracy and the comprehensiveness of the target white balance gain are improved. On the other hand, the local color information of the color sensor array is combined, so that the image can be processed from multiple dimensions such as global dimension and local dimension more accurately, and the image quality is improved.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty.

Fig. 1 shows a schematic diagram of an application scenario to which the image white balance processing method of the embodiment of the present disclosure may be applied.

Fig. 2 schematically illustrates a schematic diagram of an image white balance processing method according to an embodiment of the present disclosure.

Fig. 3 schematically illustrates a schematic diagram of an image block in an embodiment of the present disclosure.

Fig. 4 schematically shows a flow chart of determining the target white balance gain in different ways in the embodiment of the present disclosure.

Fig. 5 schematically shows a weight change diagram of the target white balance gain in the embodiment of the present disclosure.

Fig. 6 schematically illustrates a schematic diagram of mask information of an embodiment of the present disclosure.

Fig. 7 schematically illustrates a schematic view of a target area of an embodiment of the present disclosure.

Fig. 8 schematically illustrates a schematic diagram of smooth transition of gain information for different regions in an embodiment of the present disclosure. Schematic of local gain information for a target region.

Fig. 9 schematically illustrates a schematic diagram of a manner of smoothing a transition in an embodiment of the present disclosure.

Fig. 10 schematically shows a schematic configuration diagram of an image signal processor in an embodiment of the present disclosure.

Fig. 11 schematically illustrates a flowchart of acquiring a target image in an embodiment of the present disclosure.

Fig. 12 schematically illustrates a block diagram of an image white balance processing apparatus in an embodiment of the present disclosure.

Fig. 13 schematically illustrates a block diagram of an electronic device in an embodiment of the disclosure.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the embodiments of the present disclosure can be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

In the related art, channel ratio statistics is performed on each block region in a multi-window region of an image. And performing matching calculation by using the actual statistical data and the light source calibration data, and outputting different White Balance Gain (WBG) results. And applying the global gain to red, green and blue (RGB) four-channel data acquired by the system to obtain a color matched with the perception of a real scene. Wherein, a global action gain value is output through global statistics. The gain value of local self-adaptive correction cannot be achieved. Moreover, by calibrating the light source in advance, the interpolation calculation mode of different color temperatures is easy to have deviation under the scene that the actual color temperature calculation is inaccurate, and finally the mismatching of the image color and the actual scene performance is influenced. Also, the related art described above lacks a process for further refining the information content. For example, it is impossible to perform more detailed processing on more important contents when there are more interesting contents such as portrait self-timer, group photo, etc. In some scenarios, more important information areas are needed to be segmented and extracted. And carrying out richer color management on the segmented areas in combination with color information of a multi-window color sensor so as to achieve more targeted expression of the brightness and the color of the target area. Therefore, the method has certain limitations and poor accuracy, and cannot realize targeted processing on the target area.

In order to solve technical problems in the related art, an embodiment of the present disclosure provides an image white balance processing method, which may be applied to an application scenario in which white balance processing is performed on an image in a photographing process. Fig. 1 is a schematic diagram illustrating a system architecture to which the image white balance processing method and apparatus according to the embodiment of the present disclosure can be applied.

As shown in fig. 1, the terminal 101 may be a smart device with an image processing function, for example, a smart device such as a smart phone, a computer, a tablet computer, a smart speaker, a smart watch, an in-vehicle device, a wearable device, and a monitoring device. The terminal can contain a camera, and the type of the camera can be any type as long as the shooting processing can be carried out. The number of cameras may be at least one, for example, one, four, etc., as long as photographing is possible. The image to be processed can be a shot image or an image of each frame in a shot video.

In the disclosed embodiment, the terminal 101 may include a memory 102 and a processor 103. The memory is used for storing the image, and the processor is used for processing the image, such as white balance processing and the like. The memory 102 may store a to-be-processed image 104 therein. The terminal 101 obtains an image 104 to be processed from the memory 102, and sends the image to the processor 103, the image to be processed is processed in the processor 103 in a blocking manner to obtain a plurality of image blocks, and the global white balance gain of the image to be processed is determined. And determining a target white balance gain of each image block according to the global white balance gain and the local color information of each image block acquired by the color sensor array, and further performing white balance processing on the image to be processed according to the target white balance gain, thereby generating a white-balanced target image 105.

It should be noted that the image white balance processing method provided by the embodiment of the present disclosure may be executed by the terminal 101. The image white balance processing method may also be provided in the terminal.

Next, an image white balance processing method in the embodiment of the present disclosure is explained in detail with reference to fig. 2.

In step S210, an image to be processed is acquired, and a global white balance gain of the image to be processed is determined.

In the embodiment of the disclosure, the image to be processed may be an image obtained by shooting an object to be shot through a camera module of the terminal, or may be an image of each frame in a shot video. The image to be processed may also be each frame of image in an image or video taken directly from an album or other storage location. The terminal may be any one of a smartphone, a digital camera, a smart watch, a wearable device, a vehicle-mounted device, or a camera of a monitoring device, as long as it can photograph an object to be photographed and can perform image processing, and the smartphone is taken as an example here for description. The camera module can include at least one camera, for example, any one or combination of a main camera, a long-focus camera, a wide-angle camera and a macro camera. The image to be processed may be various types of images, for example, a dynamic image or a static image, and the like.

The image to be processed may be an RGB image, i.e., an RGB three-channel image. Each pixel point of the RGB image is composed of three colors of RGB. When the terminal is in a photographing mode, a photographed image obtained through the camera module can be a RAW image. The RAW image is the original image data information collected by the camera module. In the embodiment of the disclosure, the shot image shot by the terminal can be converted to obtain the image to be processed. For example, format conversion may be performed on a captured image in the RAW format by using a general conversion algorithm to obtain an image to be processed in the RGB format, so as to improve convenience of subsequent processing.

After the to-be-processed image is acquired, an image white balance algorithm module in the image signal processor can acquire a global white balance gain of the to-be-processed image. Specifically, a statistical analysis may be performed on scene pixels of the image to be processed to obtain a global white balance gain WBG. The global white balance gain refers to a white balance gain with respect to the entire image to be processed, i.e., can be used for the entire white balance of the image to be processed. The statistical analysis of the scene pixels can be realized through hardware multi-window. For example, the image signal processor may perform block statistics on pixels in an ROI (Region of interest) Region in the image sensor to count the number of pixels included in a color channel in each image block Region and a pixel value of each pixel, where the color channel may be r, gr, gb, b, or other types of channels, which is not limited herein. The effective area may be at least a partial area of the whole area of the image to be processed, for example, the effective area may be the whole area or a partial area of the image to be processed, which is determined according to actual requirements, and the effective areas corresponding to different scenes may be the same or different.

In some embodiments, the number of pixels included in each color channel of each image block in the effective area of the image to be processed and the pixel value of each pixel may be counted. Further, the number of pixels included in each image block and the pixel value of each pixel may be statistically processed to obtain a global white balance gain. Illustratively, each pixel may contain a plurality of color sub-pixels, one color channel for each color sub-pixel. The pixel values of each color sub-pixel may be averaged to obtain an average value of each color channel, and the R/G value and the B/G value may be determined according to the average value of each color channel to determine a global AWB gain, i.e., a global white balance gain.

In step S220, the image to be processed is partitioned into a plurality of image blocks, and the local color information of each image block is determined by the color sensor array.

In the embodiment of the present disclosure, the image to be processed may be processed in a blocking manner to obtain a plurality of image blocks. The image block may be a part of an image to be processed, and a plurality of image blocks are not overlapped. The size of each image block may be the same, and the number of image blocks may be determined according to the number of grids. For example, a mesh area may be provided and applied to the image to be processed to divide the image to be processed into a plurality of image blocks according to the mesh area, and the divided image blocks may correspond to the mesh area one to one. Referring to fig. 3, a plurality of image blocks 301 may be included. For example, grid 00 corresponds to image block 00, grid 01 corresponds to image block 01, and so on. The size of the grid area can be set according to actual requirements and hardware structures together, namely, under the condition that the hardware structures can support, the grid area is set according to the actual requirements. For example row col column. Based on this, the color sensor may be associated with an image block and a grid, the image to be processed may be divided into image blocks of row × col, and each image block corresponds to each grid, and under the same field angle, the multi-window spatial range represented by the grid area coincides with the image to be processed, so that the grid area can be overlaid on the image to be processed. Each grid may correspond to a window and thus may be referred to as a multi-window.

In the embodiment of the present disclosure, a color sensor array may be included, and the color sensor array may include a plurality of color sensors 302 arranged in an array. The number of color sensors may be determined according to the number of image blocks. Also, the color sensor array may be combined with multi-window information, so each color sensor may be a multi-window color sensor. Each grid represents a color sensor, each color sensor corresponding to each image block. Since the image to be processed is divided into a plurality of grids, a row × col Color sensor array is spatially formed, which forms a multi-window Color sensor.

The multi-window color sensor array is an independent sensor and can be arranged on one side of any camera in the camera module and close to the camera module. The module of making a video recording can be rear-mounted camera module, and the module of making a video recording can include at least one camera, for example can include main camera, long burnt camera, wide-angle camera, arbitrary one in the macro camera or its combination. The specific arrangement position and arrangement order of the multiple cameras can be determined according to actual requirements, and are not specifically limited herein. The color sensor array may be, for example, the left side of a tele camera, or the right side of a main camera or the underside of the last camera in at least one camera, etc. The camera module and the color sensor array can be arranged adjacently or at intervals. The specific position of the color sensor array may be determined according to a calibration result obtained by calibrating the color sensor array in an actual application process, or may be determined according to an actual requirement, which is not specifically limited herein.

After the color sensor array is introduced, the associated image blocks can be detected by each color sensor in the color sensor array, and local color information of each image block is obtained. The color sensor may be a sensor for detecting relevant information such as a color, a color temperature, a spectrum, and the like of the scene, and may be configured to detect color information of an object corresponding to each image block and color temperature information of the current scene. The object may be an object included in each image block, and may be any type of object, such as an object, a person, and the like. The spectrum is different wavelengths, the color temperature is a response curve of different wavelengths, and the wavelength corresponds to a color, so that the spectrum, the color temperature and the color are correlated. Therefore, the local color information is taken as the color information of the object corresponding to each image block for example.

Next, as shown in fig. 2, in step S230, a target white balance gain of each image block is determined according to the global white balance gain and the local color information, and the image block is subjected to white balance processing according to the target white balance gain to generate a target image corresponding to the image to be processed.

In the embodiment of the present disclosure, the local color information calculated by the color sensor and the global white balance gain calculated by the white balance module may be compared to calculate difference information between the local color information and the global white balance gain. Specifically, the difference information between the two can be calculated according to the absolute value of the difference between the two. It should be noted that, since the white balance gain can be determined by the ratio of each color sub-pixel, it can also be considered to compare the values of each color sub-pixel. For example, the numerical values of the color information R, G, B may be compared, and the R/G value and the B/G value may be compared.

Specifically, the reference color information of each image block can be obtained by multiplying the global white balance gain by the numerical value of each color sub-pixel of each image block; and further comparing the reference color information with the color information acquired by the color temperature sensor to obtain difference information. The difference information may be an absolute value of a difference between the two. In addition, the R/G value and the B/G value may be determined from the color information acquired by the color temperature sensor and compared with the global white balance gain to obtain the difference information.

Further, the difference information may be compared with a threshold parameter to obtain a comparison result. And selecting different modes to determine the white balance gain of each image block as the target white balance gain according to the comparison result between the difference information and the threshold parameter. The threshold parameter may include a first threshold value and a second threshold value, and the first threshold value Th1 is smaller than the second threshold value Th2. The threshold parameters for determining the difference information using the color information and the difference information using the ratio are also different, and the numerical values of R, G, and B are illustrated as an example.

A flow chart for selecting different ways of determining the target white balance gain is schematically shown in fig. 4, and with reference to fig. 4, mainly comprises the following steps:

in step S410, it is determined whether the difference information is smaller than a first threshold; if yes, go to step S420; if not, go to step S430;

in step S420, if the difference information is smaller than the first threshold, determining a target white balance gain according to the global white balance gain;

in step S430, it is determined whether the difference information is smaller than a second threshold; if yes, go to step S440; if not, go to step S450;

in step S440, if the difference information is greater than the first threshold and smaller than the second threshold, the global white balance gain is adjusted by the gain information corresponding to the local color information;

in step S450, if the difference information is greater than a second threshold, the target white balance gain is determined according to the gain information corresponding to the local color information.

In the embodiment of the present disclosure, as the difference information changes, the weight of the target white balance gain in the global white balance gain is also gradually decreased, specifically referring to the weight change diagram shown in fig. 5. When the difference information is greater than the first threshold, the weight of the target white balance gain in the global white balance gain is 1; when the difference information is between the first threshold and the second threshold, the weight of the target white balance gain in the global white balance gain is gradually reduced; when the difference information is greater than the second threshold, the weight of the target white balance gain in the global white balance gain is 0, that is, the weight of the target white balance gain in the gain information corresponding to the local color information acquired by the color sensor array is 1.

On this basis, if the difference information between the two is smaller than the first threshold, the target white balance gain of each image block may be a global white balance gain calculated by a white balance algorithm. In this case, the color sensor does not need to correct the global white balance gain calculated by the white balance algorithm. Therefore, the global result represented by the global white balance gain is directly used as the input for determining the target white balance gain to obtain the target white balance gain of each image block, that is, the global white balance gain obtained by the white balance calculation module is directly used as the target white balance gain of each image block, and the target white balance gain of each image block is the same.

If the difference information between the two is greater than the first threshold and smaller than the second threshold, the target white balance gain of each image block can be determined jointly according to the color sensor and the global white balance gain calculated by the white balance algorithm, that is, the local color information acquired by the color sensor array needs to adjust the global white balance gain calculated by the white balance algorithm, that is, the global white balance gain calculated by the white balance algorithm is adjusted through the gain information corresponding to the local color information acquired by the color sensor array. For example, the target white balance gain of each image block may be obtained by interpolating all gain information calculated in different manners. The interpolation may be a weighted sum operation. The gain information obtained in different manners has different corresponding weight parameters, and the sum of all the weight parameters is 1. Based on this, all gain information can be fused according to the weight parameters. For example, the gain information corresponding to the local color information acquired by the color sensor array and the global white balance gain may be weighted and fused according to the corresponding weighting parameters, for example, as shown in formula (1):

out = weight awb _ algo + (1-weight) color _ sensor formula (1)

Wherein awb _ algo represents the global white balance gain, and color _ sensor represents the gain information corresponding to the local color information acquired by the color sensor array.

And fusing the global white balance gain by using the gain information of the local color information acquired by the color sensor array, wherein the calculated target white balance gain of each image block can be different because the local color information of each image block can be different.

If the difference information between the two is greater than the second threshold, the target white balance gain of each image block may be determined according to the gain information corresponding to the local color information acquired by the color sensor array. Specifically, the determination may be performed according to a ratio of the obtained local color information, and details are not described here. At this time, the target white balance gain of each image block is calculated only according to the local color information acquired by the color sensor array, and thus the target white balance gains of different image blocks may be different.

In the embodiment of the disclosure, the difference information between the local color information acquired by the color sensor array and the reference color information corresponding to the global white balance gain is compared with the first threshold and the second threshold, and the target white balance gain of each image block is acquired in different manners based on the comparison result, so that the accuracy of the target white balance gain corresponding to each image block can be improved. The target white balance gain can be independently determined for each image block, so that the limitation that the target white balance gain of each image block can only be determined according to the global white balance gain is avoided, the flexibility and pertinence can be improved, and the accuracy is improved.

After the target white balance gain of each image block is determined, the target white balance gain of each image block may be smoothed to achieve smooth transition between the target white balance gains of all the image blocks, and to reduce a sudden change effect generated after different target white balance gains are applied between local blocks represented by each image block. Wherein the smoothing process may be implemented by low-pass filtering. The low-pass filtering refers to a filtering mode that low-frequency signals can normally pass through, and high-frequency signals exceeding a set critical value are blocked and weakened. The blocking and attenuating amplitudes may vary depending on the frequency and the filtering procedure or purpose.

And then, performing white balance processing on the image blocks according to the target white balance gain of each image block to generate a target image corresponding to the image to be processed. Specifically, the target white balance gain of each image block can be applied to the image to be processed, and the color parameters of each pixel point in each image block are obtained to obtain the target image. The target image may be an image obtained by white balancing at least a part of image blocks in the image to be processed. For example, the white balance processing may be performed on a partial image in the image to be processed, or may be performed on all image blocks in the image to be processed, which is determined according to actual needs.

Each pixel point refers to a pixel point composed of RGB in the image to be processed, and each pixel point may include a first color sub-pixel R, a second color sub-pixel B, and a third color sub-pixel G. The color parameter refers to a value of each color sub-pixel, which may be a channel component, e.g., R, G, B value, of the color channel represented by each color sub-pixel. Based on the method, multiplication operation can be carried out on the target white balance gain of each image block and the numerical value of each color sub-pixel in each image block, and the color parameter of each pixel point in each image block is obtained.

In the embodiment of the disclosure, a color sensor array is introduced to obtain local color information in each image block, and the global white balance gain can be further adjusted through the local color information to obtain a numerical value of a target white balance gain AWB gain of each image block, so that local adaptive adjustment is realized, accuracy and flexibility are improved, and independent control of each image block can be realized. On the basis, the local self-adaptive white balance processing is carried out on the image to be processed according to the target white balance gain of each image block, the limitation of carrying out global white balance in the related technology can be avoided, the pertinence and the flexibility are improved, the quality of the target image can also be improved, the target image is enabled to better meet the actual requirement, and the application range is enlarged.

It should be noted that, in the embodiment of the present disclosure, besides the color sensor array, the image signal processor may further include a segmentation subsystem module. The information acquired by the image sensor may be correlated with the segmentation subsystem module and sent to the segmentation subsystem module for processing to enable the segmentation subsystem module to obtain desired mask information, such as mask information 600 shown in fig. 6. The mask information can block a partial region and transmit other partial regions. The mask information is used to acquire a partial region, for example, a target region, from the entire image to be processed. Referring to fig. 6, a face region may be obtained according to the mask information 600, and the face region may be used as a target region 601, and a remaining region in the image to be processed except the target region may be used as a reference region 602.

In order to solve the technical problems in the related art, a target region may be determined from an image to be processed according to mask information, that is, a region corresponding to the mask information is used as the target region, and independent color control and color processing are performed on color information in the target region. The target area may be an area that needs to be finely processed, and is determined according to actual needs, for example, a partial area in a face image, an area of content that is more interesting in the image, and the like.

As can be seen from fig. 7, the target area 701 may include a plurality of image blocks, and thus color information of the target area may be acquired in combination with local color information of the plurality of image blocks. Specifically, the color information of the target area may be obtained by performing averaging processing according to the local color information of each image block included in the target area; the maximum value or the minimum value in the local color image of each image block may be directly used as the color information of the target area, and the maximum value or the minimum value is not specifically limited herein, and may be obtained by performing a combination process based on the local color information of a plurality of image blocks.

Further, since the target region may include a plurality of image blocks, the color information of the target region may be determined based on the local color information of the plurality of image blocks, and then the local gain information of the target region may be determined according to the color information of the target region. Specifically, the local gain information of the target region may be determined according to the R/G value and the B/G value corresponding to the color information of the target region. In addition, the target white balance gain of each image block contained in the target area can be averaged to obtain local gain information of the target area; the maximum or minimum of the target white balance gains of each image block may also be directly used as the local gain information of the target region, which is not limited herein.

In some embodiments, gain information of a reference region outside the target region may be determined as the first gain information, and local gain information within the target region may be determined as the second gain information. That is, the processed red-green-blue (rgb) color gain information WBG2 is output in the target area. The reference area outside the target area outputs the original red, green and blue (rgb) color gain value WBG1, and the first gain information WBG1 is the result of the fusion (weighted summation) of the global white balance gain and the local color information of each image block. Referring to fig. 8, the gain information of the reference area 802 outside the target area is the first gain information WBG1, and the target area 801 gain information is the second gain information WBG2. The first gain information may be greater than the second gain information or may be smaller than the second gain information, as long as the first gain information is different from the second gain information, and is not particularly limited herein.

After determining the second gain information in the target region and the first gain information in the reference region, the second gain information in the target region may be adjusted while keeping the first gain information in the reference region unchanged. The second gain information may be specifically adjusted according to a preset adjustment parameter, and the preset adjustment parameter may be set according to an actual requirement, which is not specifically limited herein. The adjustment parameters may include the image blocks to be adjusted and the adjustment amplitude. When the adjustment is performed, the image blocks and the adjustment amplitude which are adjusted as required can be fully adjusted or partially adjusted, so that the second gain information of the target area is adjusted, the second gain information of the target area is finely and independently controlled, and the flexibility of white balance processing is improved.

In order to avoid the difference between the gain information of different regions, the second gain information in the target region and the first gain information in the reference region outside the target region may be smoothly transitioned. The smooth transition may be a radial transition, or may be in other transition manners, and the radial transition is taken as an example for description here.

The radial transition is defined by its center. The center, shape and size of the transition can also be specified during the radial transition. The shape may be circular or elliptical and the gradual size may be indicated to the farthest corner. In the embodiment of the present disclosure, with the central area of the first gain information of the reference area outside the target area as the radial origin, the second gain information of the target area is radially transitioned to the first gain information of the reference area, as shown in fig. 8. The transition manner between the two can be a series of weight transition manners such as a smooth curve weight radial distribution or a linear distribution, which is shown in fig. 9 and is not limited in detail here. The smooth curve weight radial distribution means that the weight of the transition mode can be in a smooth curve distribution, and the linear distribution means that the distribution mode of the weight of the transition mode is in a linear distribution.

After the local gain information of the target region and the gain information of the reference region are determined, white balance processing may be performed on the image to be processed based on the white balance gain of each position region, specifically, multiplication operation may be performed on the local gain information of the target region and the numerical value of the color sub-pixel of each pixel point of the target region, and multiplication operation may be performed on the gain information of the reference region and the numerical value of the color sub-pixel of each pixel point of the reference region, so as to obtain the color parameter of each pixel point in the image to be processed. The target image may be an image obtained by performing a targeted process on the image content of the target region.

In the embodiment of the disclosure, the target area is obtained through the mask information, and then the local white balance processing can be performed on the image content of the local area represented by the target area in the image to be processed, so that the limitation that only the whole processing can be performed in the related technology is avoided, the flexibility and pertinence of the image white balance processing are improved, and the image quality is also improved.

In the embodiment of the present disclosure, by introducing the color sensor array, the local color information of each image block can be obtained according to the multi-window information of the color sensor array. And the gain in the global white balance can be further adjusted according to the local color information of each image block to obtain the numerical value of the target white balance gain of each image block. The method can realize local self-adaptive adjustment, thereby obtaining the target white balance gain of a local area and improving the accuracy. The segmentation subsystem module can carry out further color gain correction on the image content in a more targeted manner. By combining the information of the color sensor array, the image to be processed can be processed with more optimal color more accurately in multiple dimensions such as global dimension, local dimension, image content and the like.

Fig. 10 schematically shows a block diagram of an image signal processor, and referring to fig. 10, the image signal processor may include an image white balance algorithm module 1000, and may BE further divided into front-end processing 1001 (FE) and back-end processing 1002 (BE). The front-end processing (FE) includes a white balance sub-system module 1003, and the back-end processing (BE) includes sub-system modules such as a back tone mapping (tone) 1004 and a color (color) 1005. A segmentation subsystem module 1006 may also be included in the image signal processor. In addition, a color sensor array 1007 and a sensor 1008 may be included.

Referring to the illustration in fig. 10, the grid information acquired by the color sensor array is associated with the grid acquired in the white balance algorithm subsystem module. The sensor is associated with the segmentation subsystem module, and the acquired information is sent to the segmentation subsystem module to be processed to acquire the required mask information. And associating the color sensor array with the mask information output by the segmentation subsystem module, and outputting the color information in the target area.

Based on the hardware structure, the color sensor array can acquire local color information of each image block, acquire a target white balance gain of each image block in different modes through the image white balance algorithm module according to difference information of reference color information corresponding to the local color information and the global white balance gain and a comparison result between the reference color information and the threshold parameter, and perform image white balance processing through the front-end white balance subsystem module. In addition, after the color sensor array acquires the target white balance gain of each image block, the mask information can be acquired according to the segmentation subsystem module, so that the color information of the target area corresponding to the mask information is acquired, the gain information of the target area is calculated through the image white balance algorithm module, and the image white balance processing is performed according to the front white balance subsystem module. In the embodiment of the disclosure, the capability of the white balance subsystem module in front-end processing can be improved by introducing the color sensor array.

Fig. 11 schematically shows a flowchart for acquiring a target image, and referring to fig. 11, mainly includes the following steps:

in step S1101, a to-be-processed image 1110 is acquired.

In step S1102, a global white balance gain 1120 is acquired.

In step S1103, the image to be processed is divided into a plurality of image blocks 1130.

In step S1104, local color information 1150 is acquired based on the color sensor array 1040.

In step S1105, a target white balance gain 1160 is generated from the global white balance gain 1120 and the local color information 1150.

In step S1106, the image to be processed 1110 is subjected to white balance processing in accordance with the target white balance gain 1060.

In step S1107, a target image 1170 is generated.

In the embodiment of the disclosure, by introducing the color sensor array, the local color information of each image block can be obtained according to the multi-window information of the color sensor array. And further adjusting the gain in the global white balance according to the local color information of each image block to obtain the value of the target white balance gain of each image block. The method can realize local self-adaptive adjustment, thereby obtaining the target white balance gain of a local area and improving the accuracy.

It should be added that, in the process of image processing, the smoothing process of the time domain space can be performed for the real-time processing mode such as video or preview. At each moment t, one image can be sent into the image signal processing system for processing to obtain an image corresponding to Frame t-2, frame t-1, frame t +1, and Frame t + 2. In this process, data smoothing filtering may be performed on a time domain (image sequences direction) on each image. Smoothing the temporal domain of each image may be understood as smoothing in the direction of the image time series. Specifically, the smoothing filtering may use an IIR filtering method performed in a time domain space, where the output result may be calculated as I = a × w + B (1-w). Wherein, I represents the current Frame, i.e. the output result of Frame t, a is the data or parameter of the current Frame, and w is the weight of the current Frame. B is the data or parameter of the last Frame-1 adjacent to the current Frame on the time axis, and 1-w is the weight of Frame-1. By the method, the smoothing of the time domain (image sequence) can be carried out to reduce the difference between different times and avoid the situation of being more abrupt.

In summary, according to the technical solution in the embodiment of the present disclosure, the local color information of each image block can be determined according to the color sensor array, and then the target white balance gain of each image block is determined according to the global white balance gain and the local color information, and the global white balance gain can be adjusted according to the local color information of the color sensor array, so that each image block is independently controlled, the limitation that only the whole process can be performed in the related art is avoided, and the accuracy and the comprehensiveness of the target white balance gain are improved. The image can be processed from multiple dimensions such as global dimension and local dimension more accurately by combining the local color information of the color sensor array, and the image quality is improved. In addition, smooth transition is carried out on the target white balance gain among the image blocks, so that a very sharp area cannot appear on the whole image, and the smoothness and the image quality are improved. After the segmentation subsystem module is introduced, the target area can be obtained in the image to be processed through the mask information, and then the local white balance processing can be carried out on the image content of the local area represented by the target area in the image to be processed, so that the limitation that only the whole processing can be carried out in the related technology is avoided, the flexibility and pertinence of the image white balance processing are improved, and the image quality is also improved. The target white balance gain of each image block calculated by combining the acquired local color information of the color sensor array can more accurately perform color processing on the image to be processed from multiple dimensions such as global, local and image contents, the effect and reality of block white balance can be improved, smooth transition among the image blocks is realized, the accuracy of image processing is improved, and the image quality is improved.

An image white balance processing apparatus is provided in the embodiments of the present disclosure, and referring to fig. 11, the image white balance processing apparatus 1200 may include:

a global obtaining module 1201, configured to obtain an image to be processed, and determine a global white balance gain of the image to be processed;

the local color determining module 1202 is configured to block the image to be processed to obtain a plurality of image blocks, and determine local color information of each image block through a color sensor array;

the image processing module 1203 is configured to determine a target white balance gain of each image block according to the global white balance gain and the local color information, and perform white balance processing on the image block according to the target white balance gain to generate a target image corresponding to the image to be processed.

In an exemplary embodiment of the present disclosure, the global acquisition module includes: and the pixel counting module is used for counting the pixel number of each color channel and the pixel value of each pixel of each image block in the effective area of the image to be processed and determining the global white balance gain.

In an exemplary embodiment of the present disclosure, an image processing module includes: the comparison module is used for comparing the reference color information corresponding to the global white balance gain with the local color information to determine difference information; and the target white balance gain acquisition module is used for selecting different modes to acquire the target white balance gain according to the comparison result of the difference information and the threshold parameter.

In an exemplary embodiment of the present disclosure, the target white balance gain acquiring module includes: a first obtaining module, configured to determine a target white balance gain according to a global white balance gain if the difference information is smaller than a first threshold; a second obtaining module, configured to adjust the global white balance gain according to a gain corresponding to local color information to determine the target white balance gain if the difference information is greater than a first threshold and smaller than a second threshold; and the third acquisition module is used for determining the target white balance gain according to the gain corresponding to the local color information if the difference information is greater than a second threshold value.

In an exemplary embodiment of the present disclosure, the third obtaining module includes: and the interpolation module is used for interpolating the gain corresponding to the local color information and the global white balance gain to obtain the target white balance gain.

In an exemplary embodiment of the present disclosure, the apparatus further includes: the color information acquisition module is used for acquiring the color information of a target area in the image to be processed; the gain information determining module is used for determining local gain information according to the color information of the target area and determining gain information of a reference area except the target area; and the local white balance processing module is used for carrying out white balance processing on the image to be processed according to the local gain information and the gain information of the reference region except the target region.

In an exemplary embodiment of the present disclosure, the color information acquisition module includes: and the acquisition control module is used for acquiring a target area according to mask information in an image to be processed and determining the color information according to local color information of the image block contained in the target area.

In an exemplary embodiment of the present disclosure, the gain information determination module includes: a first determining module, configured to determine first gain information as gain information of a reference region outside the target region, and keep the gain information unchanged; and the second determining module is used for determining second gain information as the local gain information in the target region and adjusting the local gain information.

It should be noted that, the specific details of each part in the image white balance processing apparatus have been described in detail in the embodiment of the image white balance processing method part, and details that are not disclosed may refer to the embodiment of the method part, and thus are not described again.

Exemplary embodiments of the present disclosure also provide an electronic device. The electronic device may be the terminal 101 described above. In general, the electronic device may include a processor and a memory for storing executable instructions of the processor, the processor being configured to perform the above-described image white balance processing method via execution of the executable instructions.

The structure of the electronic device will be exemplarily described below by taking the mobile terminal 1300 in fig. 13 as an example. It will be appreciated by those skilled in the art that the configuration of figure 13 can also be applied to fixed type devices, in addition to components specifically intended for mobile purposes.

As shown in fig. 13, the mobile terminal 1300 may specifically include: a processor 1301, a memory 1302, a bus 1303, a mobile communication module 1304, an antenna 1, a wireless communication module 1305, an antenna 2, a display screen 1306, a camera module 1307, an audio module 1308, a power module 1309, and a sensor module 1310.

Processor 1301 may include one or more processing units, such as: the Processor 1301 may include an AP (Application Processor), a modem Processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband Processor, and/or an NPU (Neural-Network Processing Unit), etc. The image denoising processing method in the exemplary embodiment may be performed by an AP, a GPU, or a DSP, and when the method involves neural network related processing, may be performed by an NPU, for example, the NPU may load neural network parameters and execute neural network related algorithm instructions. Illustratively, an image to be processed may be acquired, and a global white balance gain of the image to be processed may be determined; partitioning the image to be processed to obtain a plurality of image blocks, and determining local color information of each image block through a color sensor array; and determining a target white balance gain of each image block according to the global white balance gain and the local color information, and performing white balance processing on the image blocks according to the target white balance gain to generate a target image corresponding to the image to be processed.

An encoder may encode (i.e., compress) an image or video to reduce the data size for storage or transmission. The decoder may decode (i.e., decompress) the encoded data for the image or video to recover the image or video data. Mobile terminal 1300 may support one or more encoders and decoders, such as: image formats such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap), and Video formats such as MPEG (Moving Picture Experts Group) 1, MPEG10, h.1063, h.1064, and HEVC (High Efficiency Video Coding).

Processor 1301 may form a connection with memory 1302 or other components via bus 1303.

The memory 1302 may be used to store computer-executable program code, which includes instructions. The processor 1301 executes various functional applications of the mobile terminal 1300 and data processing by executing instructions stored in the memory 1302. The memory 1302 may also store application data, such as files for storing images, videos, and the like.

The communication function of the mobile terminal 1300 may be implemented by the mobile communication module 1304, the antenna 1, the wireless communication module 1305, the antenna 2, the modem processor, the baseband processor, and the like. The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. The mobile communication module 1304 may provide mobile communication solutions such as 3G, 4G, 5G, etc. applied to the mobile terminal 1300. The wireless communication module 1305 may provide a wireless communication solution such as wireless lan, bluetooth, near field communication, etc. applied to the mobile terminal 1300.

The display 1306 is used to implement display functions such as displaying user interfaces, images, videos, and the like. The camera module 1307 is used for performing a photographing function, such as photographing an image, video, etc., and may include a color sensor array therein. Audio module 1308 is used to implement audio functions, such as playing audio, collecting speech, etc. The power module 1309 is used to implement power management functions, such as charging a battery, supplying power to a device, monitoring a battery status, and the like. The sensor module 1310 may include one or more sensors for implementing corresponding sensing functions. For example, the sensor module 1310 may include an inertial sensor for detecting a motion pose of the mobile terminal 1300 and outputting inertial sensing data.

It should be noted that, in the embodiments of the present disclosure, a computer-readable storage medium is also provided, and the computer-readable storage medium may be included in the electronic device described in the foregoing embodiments; or may be separate and not incorporated into the electronic device.

A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable storage medium may transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The computer-readable storage medium carries one or more programs which, when executed by an electronic device, cause the electronic device to implement the method as described in the embodiments below.

Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, and may also be implemented by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

Furthermore, the above-described drawings are merely schematic illustrations of processes involved in methods according to exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily appreciated that the processes illustrated in the above figures are not intended to indicate or limit the temporal order of the processes. In addition, it is also readily understood that these processes may be performed, for example, synchronously or asynchronously in multiple modules.

It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims. It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is to be limited only by the terms of the appended claims.

Claims

1. An image white balance processing method, comprising:

acquiring an image to be processed, and determining the global white balance gain of the image to be processed;

partitioning the image to be processed to obtain a plurality of image blocks, and determining local color information of each image block through a color sensor array;

and determining a target white balance gain of each image block according to the global white balance gain and the local color information, and performing white balance processing on the image blocks according to the target white balance gain to generate a target image corresponding to the image to be processed.

2. The image white balance processing method according to claim 1, wherein the determining the global white balance gain of the image to be processed includes:

and counting the number of pixels of each color channel of each image block and the pixel value of each pixel in the effective area of the image to be processed, and determining the global white balance gain.

3. The image white balance processing method according to claim 1, wherein the determining the target white balance gain of each image block according to the global white balance gain and the local color information includes:

comparing reference color information corresponding to the global white balance gain with the local color information to determine difference information;

and selecting different modes to obtain the target white balance gain according to the comparison result of the difference information and the threshold parameter.

4. The image white balance processing method according to claim 3, wherein the selecting different ways to obtain the target white balance gain according to the comparison result between the difference information and a threshold parameter includes:

if the difference information is smaller than a first threshold value, determining a target white balance gain according to the global white balance gain;

if the difference information is larger than a first threshold and smaller than a second threshold, adjusting the global white balance gain through gain information corresponding to local color information to determine the target white balance gain;

and if the difference information is larger than a second threshold value, determining the target white balance gain according to the gain information corresponding to the local color information.

5. The image white balance processing method according to claim 4, wherein the adjusting the global white balance gain by the gain information corresponding to the local color information to determine the target white balance gain comprises:

and interpolating the gain information corresponding to the local color information and the global white balance gain to obtain the target white balance gain.

6. The image white balance processing method according to claim 1, characterized by further comprising:

acquiring color information of a target area in the image to be processed;

determining local gain information according to the color information of the target area, and determining gain information of a reference area except the target area;

and carrying out white balance processing on the image to be processed according to the local gain information and gain information of a reference region except the target region.

7. The image white balance processing method according to claim 6, wherein the acquiring color information of the target area in the image to be processed includes:

and acquiring a target area according to mask information in an image to be processed, and determining the color information according to local color information of an image block contained in the target area.

8. The image white balance processing method according to claim 1, wherein the determining local gain information from the color information of the target region and determining gain information of a reference region other than the target region includes:

determining first gain information as gain information of a reference region outside the target region, and keeping the gain information unchanged;

and determining second gain information as local gain information in the target region, and adjusting the local gain information.

9. An image white balance processing apparatus, comprising:

the global acquisition module is used for acquiring an image to be processed and determining the global white balance gain of the image to be processed;

the local color determining module is used for partitioning the image to be processed to obtain a plurality of image blocks and determining local color information of each image block through a color sensor array;

and the image processing module is used for determining the target white balance gain of each image block according to the global white balance gain and the local color information, and performing white balance processing on the image blocks according to the target white balance gain to generate a target image corresponding to the image to be processed.

10. An electronic device, comprising:

an image module including a color sensor array;

a processor; and

a memory for storing executable instructions of the processor;

wherein the processor is configured to perform the image white balance processing method of any one of claims 1-8 via execution of the executable instructions.

11. A computer-readable storage medium on which a computer program is stored, the computer program, when being executed by a processor, implementing the image white balance processing method according to any one of claims 1 to 8.