CN117726564A

CN117726564A - Image processing method, apparatus, electronic device, and computer-readable storage medium

Info

Publication number: CN117726564A
Application number: CN202311750153.3A
Authority: CN
Inventors: 戴文; 李鑫鹏
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2023-12-18
Filing date: 2023-12-18
Publication date: 2024-03-19

Abstract

The present application relates to an image processing method, an apparatus, an electronic device, a storage medium, and a computer program product. The method comprises the following steps: acquiring a first image, a second image and a moving region mask; the brightness of the first image is greater than the brightness of the second image; the motion region mask corresponds to the first image and the second image; synthesizing the first image, the second image and the moving region mask in a RAW domain to obtain a third image; determining a target correction gain map according to the first image, the third image and the motion area mask; the pixel value in the target correction gain map represents a correction gain coefficient corresponding to each pixel in the third image; and correcting the third image according to the target correction gain map to obtain a target image. Imaging anomalies such as artifacts and gray blocks which are common in a motion highlight region during high dynamic range synthesis can be accurately reduced.

Description

Image processing method, apparatus, electronic device, and computer-readable storage medium

Technical Field

The present invention relates to the field of image processing, and in particular, to an image processing method, an image processing apparatus, an electronic device, and a computer readable storage medium.

Background

With the development of electronic equipment shooting technology, people utilize electronic equipment to shoot images anytime and anywhere, and the quality requirements on the shot images are higher and higher.

When the electronic equipment is used for fusing multiple frames in unequal exposure, the problems of artifact, gray block and the like are easy to occur in a highlight movement area, and detection of the artifact or the gray block by the traditional image processing method is easy to occur detection omission and false detection, so that imaging anomalies of the artifact or the gray block area and the like cannot be accurately reduced.

Disclosure of Invention

The embodiment of the application provides an image processing method, an image processing device, electronic equipment and a computer readable storage medium, which can accurately reduce imaging anomalies such as artifacts or gray blocks.

In a first aspect, the present application provides an image processing method. The method comprises the following steps:

acquiring a first image, a second image and a moving region mask; the brightness of the first image is greater than the brightness of the second image; the motion region mask corresponds to the first image and the second image;

synthesizing the first image, the second image and the moving region mask in a RAW domain to obtain a third image;

determining a target correction gain map according to the first image, the third image and the motion area mask; the pixel value in the target correction gain map represents a correction gain coefficient corresponding to each pixel in the third image;

And correcting the third image according to the target correction gain map to obtain a target image.

In a second aspect, the present application also provides an image processing apparatus. The device comprises:

the acquisition module is used for acquiring the first image, the second image and the moving area mask; the brightness of the first image is greater than the brightness of the second image; the motion region mask corresponds to the first image and the second image;

the synthesizing module is used for synthesizing the first image, the second image and the motion area mask to obtain a third image;

a determining module, configured to determine a target correction gain map according to the first image, the third image, and the motion region mask; the pixel value in the target correction gain map represents a correction gain coefficient corresponding to each pixel in the third image;

and the correction module is used for correcting the third image according to the target correction gain map to obtain a target image.

In a third aspect, the present application also provides an electronic device. The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to realize the following steps:

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:

In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of:

According to the image processing method, the device, the electronic equipment, the computer readable storage medium and the computer program product, the first image, the second image and the moving area mask are obtained, then the first image, the second image and the moving area mask are synthesized in the RAW domain to obtain the third image, then the target correction gain map is determined according to the first image, the third image and the moving area mask, then the third image is corrected according to the target correction gain map to obtain the target image, the moving area and the unequal exposure image are synthesized in the RAW domain to obtain the third image, more information is ensured not to be lost, the correction gain coefficient corresponding to each pixel in the third image can be accurately determined according to the first image, the third image and the moving area mask, the correction gain coefficient corresponding to each pixel in the third image is used for correcting the third image, and imaging anomalies such as artifacts and gray blocks which are common in the moving highlight area during high dynamic range synthesis can be accurately reduced.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of an image processing method in one embodiment.

FIG. 2 is a flow chart of determining a target correction gain map based on the first image, the third image, and the motion field mask, in one embodiment.

FIG. 3 is a flow chart of determining a target correction gain map based on the first image, the third image, and the motion field mask, in one embodiment.

Fig. 4 is a flowchart of an image processing method according to another embodiment.

FIG. 5 is a flow chart of obtaining correction gains from a composite RGB map, a bright frame RGB map, and a motion field mask, in one embodiment.

Fig. 6A is an illustration of an image before artifact processing in one embodiment.

Fig. 6B is an illustration of an image after artifact processing in one embodiment.

Fig. 7A is an illustration of an image before ash block processing in one embodiment.

Fig. 7B is an illustration of an image after ash block processing in one embodiment.

Fig. 8 is a block diagram of the structure of an image processing apparatus in one embodiment.

Fig. 9 is an internal structural diagram of an electronic device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.

In the related art, aiming at the problems that artifact, gray blocks and the like are easy to occur in a highlight movement area during unequal exposure and fusion, the electronic equipment eliminates the artifact area by detecting the artifact area in a YUV (YUV) domain and filling or correcting the color in the YUV domain. However, detection of the artifact region in the YUV domain is prone to missed detection and false detection, the artifact or gray block can be reserved in the missed detection region, and unknown anomalies can occur in the false detection region.

To this end, an embodiment of the present application provides an image processing method, by acquiring a first image, a second image and a moving area mask, then synthesizing the first image, the second image and the moving area mask in a RAW domain to obtain a third image, determining a target correction gain map according to the first image, the third image and the moving area mask, and correcting the third image according to the target correction gain map to obtain a target image. Therefore, the third image is obtained by combining the motion area and the unequal exposure image in the RAW domain, so that more information is ensured not to be lost, the correction gain coefficient corresponding to each pixel in the third image can be accurately determined according to the first image, the third image and the motion area mask, the correction gain coefficient corresponding to each pixel in the third image is utilized to correct the third image, and imaging anomalies such as artifacts, gray blocks and the like which are common in the motion highlight area during the high dynamic range combination can be accurately reduced.

In one embodiment, as shown in fig. 1, an image processing method is provided, and the method is applied to the electronic device in fig. 1 for illustration, where the electronic device may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices and portable wearable devices, and the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, smart automobiles, and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The image processing method includes the following steps 102 to 108. Wherein:

step 102, acquiring a first image, a second image and a motion area mask.

Wherein, the brightness of the first image is greater than the brightness of the second image, i.e. the first image may be a bright frame and the second image may be a dark frame. The motion field mask corresponds to the first image and the second image.

The first image and the second image can be two frames of images shot by adopting different exposure time lengths aiming at the same scene, and the exposure time length of the first image is longer than that of the second image. Since the exposure time of the first image is longer than the exposure time of the second image, the brightness of the first image is greater than the brightness of the second image. The first image and the second image may be images obtained by performing one or more of black level correction, denoising, demosaicing and automatic white balance on the acquired original RAW domain data. The first image and the second image may be RGB (red, green, blue) images. The RAW field is a graph output from the image sensor, and the RAW field data is generally RAW data without any processing.

The motion region mask may be a mask map obtained by performing pixel matching or optical flow method calculation or deep neural network model calculation on the first image and the second image. The motion region mask may have a first mark representing a motion region and a second mark representing a non-motion region, the first mark may be 1 and the second mark may be 0.

Optionally, the electronic device may obtain the first image and the second image after performing one or more of black level correction, denoising, demosaicing, automatic white balancing, and the like on the collected RAW data, and obtain the moving region mask according to the first image and the second image.

And 104, synthesizing the first image, the second image and the moving area mask in the RAW domain to obtain a third image.

Optionally, the electronic device performs alignment processing on the second image to the first image in the RAW domain, and then performs HDR (High Dynamic Range ) synthesis on the first image, the second image aligned with the first image, and the operation region mask to obtain a third image. In the HDR synthesis process, a non-overexposed region of the first image can be reserved, the overexposed region in the first image is removed, a region corresponding to the overexposed region of the first image in the second image is reserved, and a third image is obtained through mask synthesis in combination with a moving region. The overexposed region may be a region formed by pixels whose luminance exceeds the overexposed threshold, or may be a region formed by pixels whose three-channel maximum value is greater than the channel luminance threshold. The three-channel maximum value refers to the luminance maximum value in RGB three channels of a pixel. The aligning of the second image to the first image includes: an exposure ratio between the first image and the second image is obtained, and then the brightness value of each pixel in the second image is multiplied by the exposure ratio to obtain an aligned second image.

Alternatively, the first image, the second image, and the third image may be linear RAW domain data. Because the first image, the second image and the third image are all linear RAW domain data, the data information loss can be reduced.

And step 106, determining a target correction gain map according to the first image, the third image and the motion area mask, wherein the pixel value in the target correction gain map represents the correction gain coefficient corresponding to each pixel in the third image.

Optionally, the electronic device may screen the overexposed region of the first image, compare the overexposed region of the first image with the moving region mask, detect a region meeting a preset condition, determine an initial correction gain map by using the brightness values of the region meeting the preset condition and the third image, and perform smoothing processing on the initial correction gain, so as to obtain the target correction gain map. The pixel value of each pixel in the target correction gain map characterizes the correction gain coefficient corresponding to each pixel in the third image.

And step 108, correcting the third image according to the target correction gain map to obtain a target image.

Optionally, the electronic device may multiply each pixel value in the target correction gain map by a luminance value of a corresponding pixel in the third image to obtain an updated luminance value, thereby implementing correction of the third image and obtaining the target image.

According to the image processing method, a first image, a second image and a moving area mask are obtained, then the first image, the second image and the moving area mask are synthesized in a RAW domain to obtain a third image, then a target correction gain map is determined according to the first image, the third image and the moving area mask, then the third image is corrected according to the target correction gain map to obtain a target image, the moving area and the unequal exposure image are synthesized in the RAW domain to obtain the third image, more information is ensured not to be lost, then a correction gain coefficient corresponding to each pixel in the third image can be accurately determined according to the first image, the third image and the moving area mask, the correction gain coefficient corresponding to each pixel in the third image is utilized to correct the third image, and imaging anomalies such as artifacts and gray blocks which are common in a moving high-light area during high dynamic range synthesis can be accurately reduced. In addition, the image processing method also realizes the task of pixel-to-pixel operation of pixels such as image brightness, color and the like, and the processing is more refined.

In an exemplary embodiment, the area meeting the preset condition may be an artifact area, as shown in fig. 2, and the determining the target correction gain map according to the first image, the third image, and the moving area mask includes:

Step 202, determining an artifact region from the first image and the motion region mask.

Since artifacts and gray blocks generally appear in areas where there is motion and bright frames are overexposed, areas where artifact removal is required can be initially detected based on these two indicators. The motion field mask may be a binary image, the motion field being represented by a first identifier and the non-motion field being represented by a second identifier. The first identification may be 1 and the second identification may be 0.

Optionally, the electronic device may determine an overexposed region of the first image in the RAW domain, obtain, for each pixel in the first image, a three-channel maximum value of the pixel, determine whether the three-channel maximum value is greater than a channel brightness threshold, if so, determine that the pixel is an overexposed pixel, obtain the overexposed region according to the overexposed pixel, and determine an overexposed region in the first image and a region representing motion in a motion region mask as an artifact region. The channel brightness threshold can be obtained according to a large amount of data statistics.

Step 204, determining an initial correction gain map according to the brightness of the artifact region and the third image.

The brightness of the third image may be a maximum brightness value or a second maximum brightness value in the third image. An initial correction gain map is determined from the luminance maximum or luminance sub-maximum in the third image in combination with the three channel maximum for the pixels in the artifact region, each pixel value in the initial correction gain map characterizing an initial gain coefficient for each pixel in the third image.

Step 206, filtering the initial correction gain map to obtain a target correction gain map.

Since there may be very sharp jumps in the correction gain coefficients in the initial correction gain map, it is necessary to smooth the initial correction gain. The filtering process is to smooth the correction gain coefficient. The filtering process can select different filtering methods, such as mean filtering mode, guided filtering and the like, according to the requirements.

Optionally, the initial correction gain map is subjected to mean filtering or guided filtering to obtain a target correction gain map.

In the method, the artifact area is detected more accurately by utilizing the integrity of the data information of the RAW area according to the first image and the moving area mask in the RAW area, omission is reduced, the initial correction gain can be determined according to the brightness of the third image and the artifact area, the initial correction gain is filtered, the target correction gain is obtained, the target correction gain is more accurate, imaging anomalies such as the artifact area or gray block can be corrected, and the artifact area or gray block can be reduced.

In an exemplary embodiment, determining an artifact region from the first image and the motion region mask comprises: determining pixels with three channel maxima greater than a channel brightness threshold according to the three channel maxima of each pixel in the first image; and determining an artifact region according to the pixels with the three channel maximum value larger than the channel brightness threshold and the pixels representing the motion region in the motion region mask.

The three channels of the pixel may be RGB three channels. For each pixel in the first image, acquiring the RGB three-channel value of the pixel, comparing the RGB three-channel values to obtain the RGB three-channel maximum value, comparing the three-channel maximum value with the channel brightness threshold value, and determining that the pixel is overexposed if the three-channel maximum value of the pixel is larger than the channel brightness threshold value. And screening to obtain pixels with the three channel maximum value larger than the channel brightness threshold, obtaining an overexposure region according to the screened pixels, and determining an artifact region according to the overexposure region and the pixels representing the motion region in the motion region mask. In an exemplary embodiment, taking the range of channel values for each pixel in the image as [0,1], the channel brightness threshold may be 0.90, 0.912, 0.92, 0.93, 0.95, 0.96, etc. In this embodiment, taking the channel brightness threshold value as 0.95, the pixel representing the motion region in the motion region mask is represented by 1 as an example, and for each pixel, the artifact region is represented by the following formula: gain_mask= (motion_mask= 1) and (max (R, G, B) >0.95, i.e. the overexposed region in the first image and the region with pixel value 1 in the motion region mask are determined as artifact regions.

The artifact region can be accurately determined by combining the channel brightness threshold with the motion region mask, so that omission is avoided.

In an exemplary embodiment, the determining an initial correction gain map based on the artifact region and the brightness of the third image comprises: acquiring the brightness maximum value of the third image; an initial correction gain map is determined based on the ratio of the luminance maximum to the three channel maximum for each pixel in the artifact region.

The third image may be an HDR image. The brightness maximum value of the third image is obtained, the ratio of the brightness maximum value to the three-channel maximum value of each pixel in the artifact area is calculated, the ratio corresponding to each pixel can be used as the initial correction gain of the corresponding pixel in the third image, the initial correction gain map is further obtained according to the initial correction gain of each pixel, and the product obtained by multiplying the ratio by a certain value can be used as the initial correction gain map.

In one exemplary embodiment, the artifact region is represented using a gain mask; the determining an initial correction gain map based on the ratio of the luminance maximum to the three channel maximum for each pixel in the artifact region, comprising: if the pixel corresponding value in the gain mask is a specified value, obtaining the initial correction gain of the pixel according to the ratio of the brightness maximum value to the three-channel maximum value of each pixel in the artifact region; if the corresponding value of the pixel in the gain mask is not specified value, the initial correction gain of the pixel is determined to be 1.

The gain mask may be a binary map, the specified value in the gain mask may be 1, and the initial correction gain calculation may be expressed as:

gain_map=max (third image)/max (R, G, B), if gain_mask= 1,

=1, other

In an exemplary embodiment, the filtering the initial correction gain map to obtain a target correction gain map includes: filtering the initial correction gain map by adopting an average filtering mode to obtain a target correction gain map; or, taking the first image as a guide image, and performing guide filtering processing on the initial correction gain map to obtain a target correction gain map.

The mean filtering (mean filter or box filter) is a linear filtering algorithm, which refers to replacing the current pixel value with a mean value of n×n pixel values around the current pixel point. The method is used for traversing each pixel in the processed image, and average filtering of the whole image is completed. For example, the pixel value of the point a is replaced by taking the average value of the pixel values of all the pixels in the 5×5 area around the point a.

And taking the first image as a guide image, and respectively guiding and filtering (guided filter) each channel of the initial correction gain image to obtain a target correction gain image.

By means of average filtering, the calculation speed can be improved, and noise can be better filtered by guiding filtering.

In an exemplary embodiment, the process of downsampling the first image and the motion region mask and the process of upsampling the filtered correction gain map are added on the basis of fig. 2 to save the calculation amount. As shown in fig. 3, determining a target correction gain map from the first image, the third image, and the motion region mask includes:

step 302, downsampling the first image and the motion field mask to obtain a downsampled first image and the downsampled motion field mask.

Downsampling is to reduce the image. The downsampling magnification can be set according to the requirement, such as 4 times, namely the width and the height of the image are respectively changed into one fourth of the original image. The downsampled interpolation may be in the nearest neighbor interpolation (nearest neighbor interpolation) or bilinear interpolation (bi-linear interpolation).

The first image and the moving region mask may be downsampled at the same magnification, or downsampled at different magnifications.

Step 304, determining an artifact region according to the downsampled first image and the downsampled motion region mask.

Optionally, the electronic device may determine, in the RAW domain, an overexposed region of the first image, obtain, for each pixel in the downsampled first image, a three-channel maximum value of the pixel, determine whether the three-channel maximum value is greater than a channel brightness threshold, if so, determine that the pixel is an overexposed pixel, obtain, according to the overexposed pixel, the overexposed region in the first image and a region representing motion in a motion region mask are determined as the artifact region. The channel brightness threshold can be obtained according to a large amount of data statistics.

If the first image downsampling is 2 times and the moving area mask downsampling is 4 times, sampling the first image at intervals of 2 points when determining the artifact area, and determining the overexposure area.

Step 306, determining an initial correction gain map according to the brightness of the artifact region and the third image.

The process of step 306 is the same as step 204.

And step 308, filtering the initial correction gain map to obtain a filtered initial correction gain map.

The process of step 308 is the same as that of step 206.

Step 310, upsampling the filtered initial correction gain map to obtain a target correction gain map.

Wherein the downsampling and upsampling are the same multiplying power. Upsampling may employ bilinear interpolation or nearest neighbor interpolation. Since a downsampling operation was previously employed, upsampling is employed here back to the original size.

In this embodiment, downsampling is performed on the first image and the motion region mask, then an artifact region is determined according to the downsampled first image and the downsampled motion region mask, an initial correction gain map is determined according to the brightness of the artifact region and the third image, then filtering processing and upsampling processing are performed on the initial correction gain map to obtain a target correction gain map, and by the downsampling processing, the calculation amount is saved by determining the artifact region, the initial correction gain map, the filtering processing and the like subsequently, and the image processing efficiency can be improved subsequently.

In an exemplary embodiment, after the determining a target correction gain map from the first image, the third image, and the motion region mask, the method further comprises: determining a weight of each pixel in the target correction gain map according to a three-channel maximum value of each pixel in the first image, wherein the weight is positively correlated with the three-channel maximum value; and adjusting the target correction gain map according to the weight of each pixel in the target correction gain map to obtain the adjusted target correction gain map.

Correspondingly, the correcting the third image according to the target correction gain map to obtain a target image comprises the following steps: and correcting the third image according to the adjusted target correction gain graph to obtain a target image.

A weight lookup table may be pre-established, where the abscissa in the weight lookup table is the image brightness and the ordinate is the weight, and each channel maximum value in the first image with the original size is used to search from the weight lookup table, where the larger the three channel maximum value of each pixel in the first image is, the closer the calculated weight is to 1, and otherwise, the closer is to 0. The weight is positively correlated with the three channel maximum, i.e., the larger the three channel maximum, the larger the weight.

The target correction gain map is adjusted through the weight, so that the highlight region is smoothly additivated, and artifacts, gray blocks and the like can be removed more accurately.

In an exemplary embodiment, the adjusting the target correction gain map according to the weight of each pixel in the target correction gain map, to obtain the adjusted target correction gain map includes: for each pixel in the target correction gain map, acquiring a first product of a pixel value of the pixel and a weight of the pixel, and acquiring a second product of 1 and a corresponding weight, and determining a pixel value after the pixel adjustment according to the sum of the first product and the second product; and obtaining an adjusted target correction gain graph according to the adjusted pixel value of each pixel in the target correction gain graph.

The found weights are applied to the upsampled gain, as can be expressed as: weight=lut (max (R, G, B)), gain_map_refined=gain_map×weight+1×1-weight, where weight is the weight of a pixel. 1-weight is the weight corresponding to 1, and introducing 1 indicates that 1 does not need to be lightened.

The image processing method is described below by taking the first image as a bright frame RGB image, the second image as a dark frame RGB image, the third image as a synthesized RGB image, and the RGB images as linear RAW domain data as an example, as shown in fig. 4, the bright frame RGB and dark frame RGB images and the motion area mask obtained after the front-end image is restored are output as synthesized high dynamic range linear images, namely, the synthesized RGB images through the RAW domain HDR synthesis module. The artifact correction module takes the synthesized RGB image, the bright frame RGB image and the motion area mask as inputs, detects the artifact area in the algorithm module, calculates correction gain, corrects the synthesized RGB image by adopting the correction gain, and outputs a corrected RGB image.

As shown in fig. 5, obtaining correction gains according to the synthesized RGB map, the bright frame RGB map, and the motion area mask includes:

(1) Mask downsampling bright frame RGB image and motion region

The downsampling operation may take into account factors that affect the detection and correction effects and the power of the electronic device processor, and the downsampling magnification may be set to 2 times, 3 times, 4 times, 6 times, etc. The interpolation mode used for downsampling may be nearest neighbor interpolation or bilinear interpolation.

(2) Artifact region detection

According to the algorithm principle of HDR synthesis, artifacts and gray blocks generally appear in a region with motion and overexposure of a bright frame, and based on the two indexes, the region needing to remove the artifacts can be detected preliminarily. The motion area mask is a binary image, and is 1 in the motion area and 0 in the non-motion area. And judging whether each pixel in the bright frame RGB image is overexposed or not, firstly taking the three-channel maximum value of the pixel, then judging whether the three-channel maximum value is larger than 0.95, if so, determining the pixel as an overexposed pixel, obtaining an overexposed region according to the overexposed pixel, and determining an artifact region according to the overexposed region and a region with 1 in a motion region mask. For each pixel, the detection formula can be expressed as: gain_mask= (motion_mask= 1) and (max (R, G, B) > 0.95)

(3) Initial correction gain calculation

An initial gain coefficient is calculated for each detected pixel based on the result of the gain_mask detection. Since the artifact region is originally overexposed in the bright frame RGB image, the dynamic range of the synthesized full image is higher, and then the artifact and gray blocks are caused, so that the artifact region is brightened to be overexposed, and an obvious optimization effect can be achieved, wherein the overexposure is defined as the global maximum value in the synthesized RGB image, namely the brightness maximum value in the synthesized RGB image, and then the initial correction gain calculation can be expressed as:

gain_map=max (composite HDR map)/(max (R, G, B)), if gain_mask= 1,

= 1，otherwise

(4) Filtering the initial correction gain to obtain a filtered initial correction gain

The gain_mask detection result is a binary image, the binary image is used for calculating an initial correction gain, the calculated initial correction gain may have a very severe jump, the initial correction gain needs to be smoothed, if the calculation speed has a very high requirement, mean filtering can be used, if the calculation speed has a very high requirement, guided filtering can be used for better effect expression, a bright frame RGB image is used as a guided image, and guided smoothing filtering is performed on the gain_mask.

(5) And up-sampling the filtered initial correction gain to obtain a target correction gain.

Since (1) employs downsampling operations, steps (2) through (4) are performed on small-scale images, and since correction gains need to be applied to the synthesized RGB map (i.e., HDR map), upsampling is employed back to the original size. The up-sampling magnification is the same as the down-sampling magnification.

(6) Correction gain refinement

The smooth filtering and interpolation up-sampling of the correction gain in steps (4) and (5) at small sizes may result in a large overall smooth area of the up-sampled correction gain, and if applied directly to the composite RGB diagram, may result in slight blooming and halation. A weight lookup table may be pre-established, where the weight lookup table is used to correct smooth adduction or refinement of the highlight region of the gain map, and the abscissa is the image brightness, and the ordinate is the gain refinement weight, and the three-channel maximum value of the pixel in the original size bright frame RGB image is used for searching, where when the three-channel maximum value of the pixel in the bright frame RGB image is greater, the weight is greater, i.e. is closer to 1, and otherwise is closer to 0, and the searched weight acts on the target gain after upsampling.

(7) Gain map application

The step is that the original size gain map calculated in advance is applied to the synthesized HDR image by pixel-to-pixel multiplication, so as to generate a corrected HDR image, as shown in FIG. 6A and FIG. 6B, wherein FIG. 6A is an image before artifact processing, and FIG. 6B is an image after artifact processing; as shown in fig. 7A and 7B, fig. 7A is an image before the gray block processing, and fig. 7B is an image after the gray block processing.

According to the image processing method, due to the artifacts and gray blocks which are frequently generated in the moving highlight region in the HDR unequal exposure synthesis, the characteristics of linear RGB images can be utilized, the related information of the unequal exposure HDR synthesis algorithm is effectively utilized, the image processing method can be embedded into an HDR synthesis algorithm frame, and meanwhile, due to small-scale detection and smoothness, the overall calculated amount and memory occupation are controllable, and the deployment of a mobile terminal is facilitated; by means of input and output of an RAW domain HDR synthesis algorithm, small-scale detection of an image, calculation of an initial correction gain map, filtering processing of the initial correction gain map and application of the gain map to a large-scale image, imaging anomalies such as artifacts, gray blocks and the like in the synthesized HDR image can be reduced, transition between a processed area and an unprocessed area is smooth and natural, abnormal image areas of the image are reduced, customization according to different application scenes is supported aiming at different question scenes, and meanwhile processing speed, power consumption, calculation power, resolution and the like of a mobile terminal are met. In addition, the image processing method not only supports the artifact removal task, but also can be equivalent to the task of pixel-to-pixel operation for any image brightness and color processing.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts referred to in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the execution of the steps or stages is not necessarily sequential, but may be performed in turn or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiments of the present application also provide an image processing apparatus for implementing the above-mentioned image processing method. The implementation of the solution provided by the apparatus is similar to the implementation described in the above method, so the specific limitation of one or more embodiments of the image processing apparatus provided below may refer to the limitation of the image processing method hereinabove, and will not be repeated herein.

As shown in fig. 8, an image processing apparatus includes an acquisition module 810, a synthesis module 820, a determination module 830, and a correction module 840. Wherein:

the acquiring module 810 is configured to acquire a first image, a second image, and a motion area mask; the brightness of the first image is greater than the brightness of the second image; the motion field mask corresponds to the first image and the second image.

The synthesizing module 820 is configured to synthesize the first image, the second image, and the motion area mask to obtain a third image.

The determining module 830 is configured to determine a target correction gain map according to the first image, the third image, and the motion area mask; the pixel values in the target correction gain map characterize the correction gain coefficients corresponding to each pixel in the third image.

The correction module 840 is configured to correct the third image according to the target correction gain map, so as to obtain a target image.

The image processing device in this embodiment obtains the first image, the second image and the moving area mask, synthesizes the first image, the second image and the moving area mask in the RAW domain to obtain a third image, determines a target correction gain map according to the first image, the third image and the moving area mask, corrects the third image according to the target correction gain map to obtain a target image, synthesizes the moving area and the unequal exposure image in the RAW domain to obtain the third image, ensures that more information is not lost, can accurately determine a correction gain coefficient corresponding to each pixel in the third image according to the first image, the third image and the moving area mask, corrects the third image by using the correction gain coefficient corresponding to each pixel in the third image, and can accurately reduce imaging anomalies such as artifacts, gray blocks and the like which are common in the moving highlight area during the high dynamic range synthesis.

In an exemplary embodiment, the determining module 830 is further configured to determine an artifact region according to the first image and the motion region mask; determining an initial correction gain map according to the brightness of the artifact region and the third image; and filtering the initial correction gain map to obtain a target correction gain map.

In an exemplary embodiment, the determining module 830 is further configured to determine, according to a three channel maximum value of each pixel in the first image, a pixel having a three channel maximum value greater than a channel brightness threshold; and determining an artifact region according to the pixels with the three channel maximum value larger than the channel brightness threshold and the pixels representing the motion region in the motion region mask.

In an exemplary embodiment, the determining module 830 is further configured to obtain a brightness maximum of the third image; and determining an initial correction gain map according to the ratio of the brightness maximum value to the three-channel maximum value of each pixel in the artifact region.

In an exemplary embodiment, the artifact region is represented using a gain mask; the determining module 830 is further configured to obtain an initial correction gain of the pixel according to a ratio of the brightness maximum value to a three-way maximum value of each pixel in the artifact region if the pixel corresponding value in the gain mask is a specified value; if the corresponding value of the pixel in the gain mask is not specified value, the initial correction gain of the pixel is determined to be 1.

In an exemplary embodiment, the determining module 830 is further configured to perform a filtering process on the initial correction gain map by using a mean filtering manner to obtain a target correction gain map; or, taking the first image as a guide image, and performing guide filtering processing on the initial correction gain map to obtain a target correction gain map.

In an exemplary embodiment, the determining module 830 is further configured to downsample the first image and the motion area mask, to obtain a downsampled first image and a downsampled motion area mask; determining an artifact region according to the downsampled first image and the downsampled motion region mask; filtering the initial correction gain map to obtain a filtered initial correction gain map; up-sampling the filtered initial correction gain map to obtain a target correction gain map; wherein the downsampling and upsampling have the same multiplying power.

The image processing device further comprises a weight acquisition module and a gain adjustment module. The weight acquisition module is used for determining the weight of each pixel in the target correction gain graph according to the three-channel maximum value of each pixel in the first image, and the weight is positively correlated with the three-channel maximum value. The gain adjustment module is used for adjusting the target correction gain map according to the weight of each pixel in the target correction gain map, and obtaining the adjusted target correction gain map.

The correction module 840 is further configured to correct the third image according to the adjusted target correction gain map, so as to obtain a target image.

In an exemplary embodiment, the gain adjustment module is further configured to, for each pixel in the target correction gain map, obtain a first product of a pixel value of the pixel and a weight of the pixel, and obtain a second product of 1 and a corresponding weight, and determine the pixel value after the pixel adjustment according to a sum of the first product and the second product; and obtaining the adjusted target correction gain map according to the adjusted pixel value of each pixel in the target correction gain map.

In an exemplary embodiment, the first image, the second image, and the third image are all linear RAW domain data.

The respective modules in the above-described image processing apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, an electronic device is provided, the internal structure of which may be as shown in FIG. 9. The electronic device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input device. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the electronic device includes a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the electronic device is used to exchange information between the processor and the external device. The communication interface of the electronic device is used for conducting wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement an image processing method. The display unit of the electronic device is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the electronic equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structure shown in fig. 9 is merely a block diagram of a portion of the structure associated with the present application and is not limiting of the computer device to which the present application applies, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

Embodiments of the present application also provide a computer-readable storage medium. One or more non-transitory computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform steps of an image processing method.

Embodiments of the present application also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform an image processing method.

It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party, and the collection, use, and processing of the related data need to comply with the related laws and regulations.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.

Claims

1. An image processing method, comprising:

2. The method of claim 1, wherein determining a target correction gain map from the first image, the third image, and the motion region mask comprises:

determining an artifact region from the first image and the motion region mask;

determining an initial correction gain map according to the brightness of the artifact region and the third image;

and filtering the initial correction gain map to obtain a target correction gain map.

3. The method of claim 2, wherein determining an artifact region from the first image and the motion region mask comprises:

determining pixels with three channel maxima greater than a channel brightness threshold according to the three channel maxima of each pixel in the first image;

and determining an artifact region according to the pixels with the three channel maximum value larger than the channel brightness threshold and the pixels representing the motion region in the motion region mask.

4. The method of claim 2, wherein determining an initial correction gain map based on the artifact region and the brightness of the third image comprises:

acquiring the brightness maximum value of the third image;

and determining an initial correction gain map according to the ratio of the brightness maximum value to the three-channel maximum value of each pixel in the artifact region.

5. The method of claim 4, wherein the artifact region is represented using a gain mask; said determining an initial correction gain map based on the ratio of said luminance maximum to the three channel maximum for each pixel in said artifact region, comprising:

if the pixel corresponding value in the gain mask is a specified value, obtaining the initial correction gain of the pixel according to the ratio of the brightness maximum value to the three-channel maximum value of each pixel in the artifact region;

if the corresponding value of the pixel in the gain mask is not specified value, the initial correction gain of the pixel is determined to be 1.

6. The method of claim 2, wherein filtering the initial correction gain map to obtain a target correction gain map comprises:

Filtering the initial correction gain map by adopting an average filtering mode to obtain a target correction gain map;

or, taking the first image as a guide image, and performing guide filtering processing on the initial correction gain map to obtain a target correction gain map.

7. The method according to claim 2, wherein the method further comprises:

downsampling the first image and the motion region mask to obtain a downsampled first image and the downsampled motion region mask;

said determining an artifact region from said first image and said motion region mask comprising:

determining an artifact region according to the downsampled first image and the downsampled motion region mask;

the filtering processing is performed on the initial correction gain map to obtain a target correction gain map, including:

filtering the initial correction gain map to obtain a filtered initial correction gain map;

up-sampling the filtered initial correction gain map to obtain a target correction gain map;

wherein the downsampling and upsampling have the same multiplying power.

8. The method according to any one of claims 1 to 7, wherein after said determining a target correction gain map from said first image, said third image and said motion region mask, said method further comprises:

Determining the weight of each pixel in the target correction gain map according to the three-way maximum value of each pixel in the first image, wherein the weight is positively correlated with the three-way maximum value;

adjusting the target correction gain map according to the weight of each pixel in the target correction gain map to obtain an adjusted target correction gain map;

correcting the third image according to the target correction gain map to obtain a target image, including:

and correcting the third image according to the adjusted target correction gain map to obtain a target image.

9. The method of claim 8, wherein said adjusting the target correction gain map according to the weight of each pixel in the target correction gain map, resulting in the adjusted target correction gain map, comprises:

for each pixel in the target correction gain map, acquiring a first product of a pixel value of the pixel and a weight of the pixel, and acquiring a second product of 1 and a corresponding weight, and determining a pixel value after the pixel adjustment according to the sum of the first product and the second product;

and obtaining the adjusted target correction gain map according to the adjusted pixel value of each pixel in the target correction gain map.

10. The method of claim 1, wherein the first image, the second image, and the third image are each linear RAW domain data.

11. An image processing apparatus, comprising:

12. An electronic device comprising a memory and a processor, the memory having stored therein a computer program which, when executed by the processor, causes the processor to perform the steps of the method of any of claims 1 to 10.

13. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 10.