CN118115375A - Image processing method, apparatus, electronic device, and computer-readable storage medium - Google Patents

Abstract

The present application relates to an image processing method, an apparatus, an electronic device, a storage medium, and a computer program product. The method comprises the following steps: denoising and demosaicing the first image features extracted from at least two original images based on noise information of reference frames in the at least two original images to obtain second image features; the frequency of the second image feature is lower than a preset frequency threshold; performing super-resolution processing on the second image features to obtain third image features; the frequency of the third image feature is higher than or equal to a preset frequency threshold; and generating a target image according to the second image characteristic and the third image characteristic. By adopting the method, the accuracy of image processing can be improved.

Description

Image processing method, device, electronic device and computer-readable storage medium

Technical Field

The present application relates to the field of imaging technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium.

Background technique

With the development of imaging technology, multi-frame image fusion technology has emerged. During the multi-frame image fusion process, ISP (Image Signal Processing) processing is usually required, including demosaicing, denoising, white balance, tone mapping, contrast enhancement, etc., so as to fuse the final image.

However, traditional image processing methods have the problem of low image processing accuracy.

Summary of the invention

Embodiments of the present application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of image processing.

In a first aspect, the present application provides an image processing method. The method comprises:

Based on noise information of a reference frame in at least two original images, performing denoising and demosaicing processing on a first image feature extracted from the at least two original images to obtain a second image feature; the frequency of the second image feature is lower than a preset frequency threshold;

Performing super-resolution processing on the second image feature to obtain a third image feature; the frequency of the third image feature is higher than or equal to a preset frequency threshold;

A target image is generated according to the second image feature and the third image feature.

In a second aspect, the present application also provides an image processing device. The device comprises:

A first processing module is used to perform denoising and demosaicing processing on a first image feature extracted from the at least two original images based on noise information of a reference frame in the at least two original images to obtain a second image feature; the frequency of the second image feature is lower than a preset frequency threshold;

A second processing module, configured to perform super-resolution processing on the second image feature to obtain a third image feature; the frequency of the third image feature is higher than or equal to a preset frequency threshold;

An image generation module is used to generate a target image according to the second image feature and the third image feature.

In a third aspect, the present application further provides an electronic device. The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Based on noise information of a reference frame in at least two original images, performing denoising and demosaicing processing on a first image feature extracted from the at least two original images to obtain a second image feature; the frequency of the second image feature is lower than a preset frequency threshold;

Performing super-resolution processing on the second image feature to obtain a third image feature; the frequency of the third image feature is higher than or equal to a preset frequency threshold;

A target image is generated according to the second image feature and the third image feature.

In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

Based on noise information of a reference frame in at least two original images, performing denoising and demosaicing processing on a first image feature extracted from the at least two original images to obtain a second image feature; the frequency of the second image feature is lower than a preset frequency threshold;

Performing super-resolution processing on the second image feature to obtain a third image feature; the frequency of the third image feature is higher than or equal to a preset frequency threshold;

A target image is generated according to the second image feature and the third image feature.

In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

Based on noise information of a reference frame in at least two original images, performing denoising and demosaicing processing on a first image feature extracted from the at least two original images to obtain a second image feature; the frequency of the second image feature is lower than a preset frequency threshold;

Performing super-resolution processing on the second image feature to obtain a third image feature; the frequency of the third image feature is higher than or equal to a preset frequency threshold;

A target image is generated according to the second image feature and the third image feature.

The above-mentioned image processing method, device, electronic device, computer-readable storage medium and computer program product, based on the noise information of the reference frame in the at least two original images, perform denoising and demosaicing on the first image feature extracted from the at least two original images to obtain a second image feature with a frequency lower than a preset frequency threshold; perform super-resolution processing on the second image feature to obtain a third image feature with a frequency higher than or equal to the preset frequency threshold; that is, through denoising and demosaicing, information such as the low-frequency color contour of the image can be more accurately obtained, and through super-resolution processing, information such as the high-frequency texture details of the image can be more accurately obtained, thereby generating a denoised, demosaiced and super-resolution target image based on the second image feature and the third image feature, thereby improving the accuracy of image processing.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a flow chart of an image processing method according to an embodiment;

FIG2 is a flow chart of an image processing method in another embodiment;

FIG3 is a flow chart of an image processing method in another embodiment;

FIG4 is a flow chart of an image processing method in another embodiment;

FIG5 is a flow chart of an image processing method in another embodiment;

FIG6 is a flow chart of an image processing method in another embodiment;

FIG7 is a flow chart of generating multiple frames of images to be input into a network in another embodiment;

FIG8 is a flow chart of an image processing method in another embodiment;

FIG9 is a block diagram of an image processing device according to an embodiment;

FIG. 10 is a diagram showing the internal structure of an electronic device in one embodiment.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

In one embodiment, as shown in FIG1 , an image processing method is provided. This embodiment is illustrated by applying the method to an electronic device, which may be a terminal or a server. It is understandable that the method may also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. Among them, the terminal may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, smart cars, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server may be implemented as an independent server or a server cluster consisting of multiple servers.

In this embodiment, the image processing method includes the following steps:

Step S102, based on the noise information of the reference frame in the at least two original images, denoising and demosaicing are performed on the first image feature extracted from the at least two original images to obtain a second image feature; the frequency of the second image feature is lower than a preset frequency threshold.

It can be understood that the original image may be a RAW domain image, that is, original, unprocessed image data.

The reference frame is an image used for reference in the registration process. Optionally, the reference frame can be the image with the highest definition among at least two original images, or the image with the highest brightness among at least two original images, which is not limited here. The noise information of the reference frame can be a noise map of the reference frame.

De-noising is the process of reducing noise in digital images. De-mosaicing (also known as de-mosaicing, demosaicking or debayering) is a digital image processing algorithm that aims to reconstruct a full-color image from the incomplete color samples output by a photosensitive element covered with a color filter array (CFA). This method is also called color filter array interpolation (CFA interpolation) or color reconstruction.

The preset frequency threshold can be set as needed. The frequency of the second image feature is lower than the preset frequency threshold, that is, the second image feature includes a low-frequency image feature. Low frequency means that the color changes slowly, that is, the grayscale changes slowly, which represents an area of continuous gradient, and this part is low frequency. For an image, the content within the edge is low frequency, and the content within the edge is most of the information of the image, that is, the general outline and contour of the image, which is the approximate information of the image. Exemplarily, the second image feature may include contour, color and other information.

Optionally, the electronic device obtains original images in at least two RAW domains through an image sensor; inputs the original images in at least two RAW domains into a joint demosaicing, denoising and super-resolution network; in the joint demosaicing, denoising and super-resolution network, performs demosaicing, denoising and super-resolution processing on the original images in at least two RAW domains to obtain a target image.

Optionally, in the joint demosaicing, denoising and super-resolution network, a first image feature is extracted from at least two original images, and a reference frame is determined from at least two original images; noise information of the reference frame is obtained; and based on the noise information of the reference frame, the first image feature is denoised and demosaiced to obtain a second image feature.

Step S104, super-resolution processing is performed on the second image feature to obtain a third image feature; the frequency of the third image feature is higher than or equal to a preset frequency threshold.

Super-resolution processing is to improve the resolution of the original image by hardware or software methods. The process of obtaining a high-resolution image through a series of low-resolution images is super-resolution reconstruction. The super-resolution ratio can be set as needed. For example, super-resolution processing can be 2x super-resolution.

The frequency of the third image feature is higher than or equal to the preset frequency threshold, that is, the third image feature includes a high-frequency image feature. High frequency means that the frequency changes quickly. At the edge of the image where the grayscale difference between adjacent areas is large, the corresponding grayscale changes quickly; and the details of the image are also areas where the grayscale value changes sharply. It is precisely because of the sharp change in the grayscale value that details appear. Exemplarily, the third image feature includes information such as edges and details.

Optionally, the electronic device inputs the second image feature into a super-resolution module (Super-resolution imaging, SR), and performs super-resolution processing on the second image feature through the super-resolution module to obtain a third image feature after super-resolution processing.

Step S106: Generate a target image according to the second image feature and the third image feature.

Optionally, the electronic device adds the second image feature and the third image feature to generate a target image.

Optionally, the electronic device multiplies the second image feature and the third image feature by corresponding weight factors respectively, and then adds them together to generate the target image. The weight factors can be set as needed.

Optionally, the electronic device selects a first portion of features from the second image features, selects a second portion of features from the third image features, adds the first portion of features and the second portion of features, and generates a target image.

It should be noted that the method of generating the target image according to the second image feature and the third image feature can be set as needed and is not limited here.

The above-mentioned image processing method performs denoising and demosaicing on the first image feature extracted from at least two original images based on the noise information of the reference frame in at least two original images, and obtains a second image feature with a frequency lower than a preset frequency threshold; performs super-resolution processing on the second image feature, and obtains a third image feature with a frequency higher than or equal to the preset frequency threshold; that is, through denoising and demosaicing, information such as the low-frequency color contour of the image can be obtained more accurately, and through super-resolution processing, information such as the high-frequency texture details of the image can be obtained more accurately, so as to generate a denoised, detail-rich, realistic and super-resolution target image based on the second image feature and the third image feature, thereby improving the accuracy of image processing. At the same time, it can meet the requirements of image processing speed, power consumption, computing power and resolution of electronic equipment.

Moreover, by jointly de-mosaicing, denoising and super-resolution network architecture, the second image features of low-frequency components and the third image features of high-frequency components are output, which can improve the utilization rate of each part in the joint de-mosaicing, denoising and super-resolution network architecture, so that the super-resolution processing part can focus on restoring the high-frequency texture detail information of the image, while the joint de-mosaicing and denoising part can focus on restoring the low-frequency information of the image. In addition, the strategy of separating high and low frequencies allows greater freedom in terms of denoising strength, super-resolution multiples, etc., and supports customization according to different application scenarios.

In one embodiment, based on the noise information of the reference frame in at least two original images, a first image feature extracted from at least two original images is denoised and demosaiced to obtain a second image feature, including: based on the noise information of the reference frame in at least two original images, a first image feature extracted from at least two original images is denoised to obtain a denoised image feature; and a denoised image feature is demosaiced to obtain a second image feature.

It can be understood that the electronic device performs denoising processing on the first image features extracted from at least two original images based on the noise information of the reference frame in the at least two original images, so as to obtain the denoised image features after denoising, and then performs demosaicing processing on the denoised image features, so as to accurately obtain the denoised and demosaiced second image features.

In another embodiment, the electronic device may also first perform a demosaicing process on the first image feature, and then perform a denoising process to obtain the second image feature.

In one embodiment, based on the noise information of the reference frame in at least two original images, a first image feature extracted from at least two original images is denoised to obtain a denoised image feature, including: performing feature mapping on the noise information of the reference frame in at least two original images and the first image feature extracted from at least two original images to obtain a mapping feature; and denoising the mapping feature to obtain a denoised image feature.

Optionally, performing denoising on the mapping features to obtain denoised image features includes: sequentially downsampling and upsampling the mapping features to obtain upsampled features; the resolution of the upsampled features is the same as the resolution of the mapping features; and obtaining denoised image features based on the upsampled features and the mapping features. The noise information may be a noise map.

Downsampling, also known as subsampling, is a multi-rate digital signal processing technology or the process of reducing the signal sampling rate, usually used to reduce the data transmission rate or data size. Upsampling is also called upsampling or interpolating.

The electronic device uses a convolution layer and an activation function layer to perform feature mapping on the noise map of the reference frame and the first image feature to obtain mapping features that can be processed by the denoising module; the mapping features are input into a UResNet module that includes 4 times of 2x downsampling and 4 times of 2x upsampling, and the mapping features are downsampled and upsampled in turn to obtain upsampled features; the upsampled features and the mapping features are spliced in the channel dimension to obtain denoised image features. Among them, the UResNet module is a UNet with the residual module as the basic processing unit. The output after each upsampling is spliced with the same resolution features before downsampling in the channel dimension and then input into the subsequent upsampling module. The number and ratio of downsampling and upsampling in the UResNet module can be set as needed and are not limited here.

In this embodiment, the electronic device performs feature mapping on the noise information of the reference frame in at least two original images and the first image features extracted from the at least two original images to obtain mapping features, and then performs denoising processing on the mapping features to obtain denoised denoised image features, thereby improving the denoising effect of the image features.

In one embodiment, performing demosaicing processing on the denoised image feature to obtain the second image feature includes: performing upsampling and residual processing on the denoised image feature to obtain the second image feature.

Optionally, the electronic device performs a 2-fold upsampling on the denoised image features, and then inputs the obtained image features into a network including two deep residual modules for residual processing, so as to obtain a second image feature after joint demosaicing and denoising. The number of deep residual modules in the network including the deep residual module can be set as needed.

Optionally, the electronic device may also process the second image feature through two convolution layers to obtain a first image component in the RGB domain.

In one embodiment, super-resolution processing is performed on the second image feature to obtain the third image feature, including: performing residual processing on the second image feature to obtain a residual processing feature; and up-sampling the residual processing feature to obtain the third image feature.

Optionally, the electronic device inputs the second image feature into a network including a deep residual module, performs residual processing through the network including the deep residual module to obtain a residual processing feature; and upsamples the residual processing feature to obtain a third image feature.

The number of the deep residual modules in the network including the deep residual modules can be set as needed. Exemplarily, in order to balance the processing efficiency and the processing effect, the network may include 8 deep residual modules.

Optionally, the electronic device may upsample the residual processing feature by using a nearest neighbor interpolation method to obtain a third image feature.

Optionally, taking 2x super-resolution processing as an example, the electronic device performs residual processing on the second image feature to obtain a residual processing feature; and performs 2x upsampling on the residual processing feature to obtain a third image feature.

For a super-resolution ratio greater than 1 and less than 2, the electronic device upsamples the residual processing feature by a preset ratio to obtain a third image feature, and upsamples the second image feature by a preset ratio to obtain an upsampled second image feature. The preset ratio is greater than 1 and less than 2.

For super-resolution ratio requirements greater than 2 times, the electronic device connects a super-resolution processing branch with a preset ratio in parallel to obtain a third image feature with the preset ratio; at the same time, the second image feature is multiplied by 2 times (or other multiples of an integer power of 2 that are closest to the preset ratio and less than the preset ratio) by the corresponding upsampling multiple to obtain the upsampled second image feature.

For example, for an 8x super-resolution ratio requirement, the electronic device increases the number of upsampling times to 3 during super-resolution processing, as well as joint denoising and demosaicing processing, that is, the first 2x upsampling results in 2x, the second 2x upsampling results in 4x, and the third 2x upsampling results in 8x.

Optionally, the electronic device may also process the third image feature through two convolutional layers to obtain a second image component in the RGB domain; and add the first image component in the RGB domain and the second image component in the RGB domain to obtain a target image in the RGB domain.

In this embodiment, residual processing is performed on the second image feature to obtain a residual processing feature, and the residual processing feature is upsampled to accurately obtain a third image feature with super resolution.

In one embodiment, as shown in FIG2 , another image processing method is provided, comprising the following steps:

Step S202 , performing feature mapping on noise information of a reference frame in at least two original images and first image features extracted from at least two original images to obtain mapping features.

Step S204, downsampling and upsampling the mapping features in sequence to obtain upsampled features; the resolution of the upsampled features is the same as the resolution of the mapping features.

It should be noted that the electronic device can achieve a denoising effect by sequentially downsampling and upsampling the mapping features.

Step S206: obtaining denoised image features based on the up-sampling features and the mapping features.

Step S208, upsampling and residual processing are performed on the denoised image feature to obtain a second image feature; the frequency of the second image feature is lower than a preset frequency threshold.

Step S210, performing residual processing on the second image feature to obtain a residual processing feature; performing upsampling on the residual processing feature at a preset ratio to obtain a third image feature; the frequency of the third image feature is higher than or equal to a preset frequency threshold.

The preset magnification can be set as needed. For example, the preset magnification can be 2 times.

Step S212, upsampling the second image feature at a preset ratio to obtain an upsampled second image feature.

It is understandable that the third image feature is based on the second image feature and also includes upsampling at a preset magnification, so the resolution of the third image feature is higher than that of the second image feature. In order to obtain the synthesized target image more accurately, the electronic device also upsamples the second image feature at a preset magnification to obtain the upsampled second image feature; the resolution of the upsampled second image feature is consistent with the resolution of the third image feature.

Optionally, the electronic device upsamples the second image feature by a preset ratio in a bicubic interpolation manner to obtain the upsampled second image feature.

Step S214: Generate a target image according to the upsampled second image features and the third image features.

In this embodiment, the electronic device performs residual processing on the second image feature to obtain a residual processing feature, then upsamples the residual processing feature at a preset ratio to obtain a third image feature, and also upsamples the second image feature at a preset ratio to obtain an upsampled second image feature. The upsampled second image feature and the third image feature have consistent resolutions, thereby more accurately generating a target image based on the upsampled second image feature and the third image feature with consistent resolutions.

In one embodiment, extracting the first image feature from at least two original images includes: extracting a sub-image feature of each original image from the at least two original images; and fusing the at least two sub-image features to obtain the first image feature.

Optionally, the electronic device extracts sub-image features of each original image from at least two original images through multiple convolutional layers, and fuses at least two sub-image features through a convolutional layer or a self-attention mechanism to obtain a first image feature.

In this embodiment, the electronic device extracts sub-image features of each original image from at least two original images, and can fuse the at least two sub-image features using complementary information between the at least two sub-image features to obtain a first image feature with more image information.

In one embodiment, as shown in FIG3 , another image processing method is provided, comprising the following steps:

Step S302: Obtain a motion region mask of each non-reference frame except the reference frame in at least two original images.

Optionally, the electronic device determines a reference frame and a non-reference frame other than the reference frame from at least two original images; detects motion regions of each non-reference frame, and generates a motion region mask for each non-reference frame.

Step S304: extracting sub-image features of each original image based on the reference frame, the non-reference frame and the corresponding motion region mask.

Optionally, the electronic device does not need to perform motion detection on the reference frame, that is, the motion area mask of the reference frame is a completely black image; the electronic device concatenates the reference frame and the corresponding motion area mask in the channel dimension, and concatenates the non-reference frame and the corresponding motion area mask in the channel dimension to obtain a concatenated reference frame and a concatenated non-reference frame; and extracts sub-image features of each original image from the concatenated reference frame and the concatenated non-reference frame through a feature extraction network. Among them, the feature extraction network may include multiple convolutional layers. Exemplarily, the original image in the RAW domain includes 4 channels, and the motion area mask and the original image in the RAW domain are concatenated in the channel dimension, then the new original image in the RAW domain is generated to include 5 channels, including the original 4 channels and the channel of the motion area mask.

Optionally, the electronic device may also input the reference frame and the concatenated non-reference frame into a feature extraction network for feature extraction.

It can be understood that the image and the motion area mask are connected in series in the channel dimension, that is, the motion area mask serves as a new channel added to the image.

Step S306: fuse at least two sub-image features to obtain a first image feature.

Step S308: performing feature mapping on the noise information of the reference frame and the first image features in at least two original images to obtain mapping features.

Step S310, down-sampling and up-sampling the mapping features in sequence to obtain up-sampled features; the resolution of the up-sampled features is the same as the resolution of the mapping features.

Step S312: obtaining denoised image features based on the up-sampling features and the mapping features.

Step S314, upsampling and residual processing are performed on the denoised image feature to obtain a second image feature; the frequency of the second image feature is lower than a preset frequency threshold.

Step S316, performing residual processing on the second image feature to obtain a residual processing feature.

Step S318, up-sampling the residual processing feature at a preset rate to obtain a third image feature; the frequency of the third image feature is higher than or equal to a preset frequency threshold.

Step S320, upsampling the second image feature at a preset ratio to obtain an upsampled second image feature.

Step S322: Generate a target image according to the upsampled second image features and the third image features.

In this embodiment, the electronic device can obtain the motion area mask of each non-reference frame except the reference frame in at least two original images, and determine the motion area position in each image based on the reference frame, the non-reference frame and the corresponding motion area mask, so as to more accurately extract the sub-image features required for each original image with respect to the motion area position and the non-motion area position, thereby improving the accuracy of image processing.

In addition, the motion region mask is used to improve the degree of noise reduction in the motion region. It is understandable that the joint denoising, demosaicing and super-resolution network tends to use multi-frame information for denoising, because noise is generally zero-mean, and the more frames there are, the stronger the denoising ability after averaging. However, the motion region in each non-reference frame is the motion region of the reference frame, that is, the motion region uses one frame of information. Therefore, the joint denoising, demosaicing and super-resolution network will implicitly improve the degree of noise reduction in the motion region to avoid splicing artifacts.

In one embodiment, a method for determining noise information of a reference frame includes: determining shot noise and readout noise corresponding to the reference frame based on shooting parameters of the reference frame; generating a target noise map of the reference frame based on the shot noise and readout noise corresponding to the reference frame; the target noise map includes noise information of the reference frame.

Shot noise is the noise caused by the non-uniformity of electron emission in active devices (such as vacuum tubes) in communication equipment. Read noise is the noise generated by the electronics in the process of transferring the charge in the pixel out of the camera. It is the combination of all the noise generated by the system components when converting the charge of each pixel into a signal and converting it into a digital value, such as charge transfer noise, read amplifier reset noise, analog-to-digital conversion quantization noise, and noise caused by crosstalk between the line transfer clock and the horizontal register drive clock.

The shooting parameters of the reference frame may include sensitivity (ISO), and may also include information such as shutter time or aperture value.

Optionally, the electronic device inputs the shooting parameters of the reference frame into the noise model, and outputs the shot noise and readout noise corresponding to the shooting parameters of the reference frame through the noise model. The noise model may be a Gaussian-Poisson noise model. It is understandable that since different digital gain settings during the shooting process will affect the noise intensity of the shooting, the image sensor is calibrated in advance to obtain the noise model.

Optionally, based on the shot noise and readout noise corresponding to the reference frame, a target noise map of the reference frame is generated, including: multiplying each pixel in the reference frame by the shot noise to obtain an intermediate noise map; and adding the readout noise to the intermediate noise map to generate a target noise map of the reference frame.

The electronic device multiplies each pixel in the reference frame by the noise value of the shot noise to obtain an intermediate noise map, and adds the noise value of the readout noise to each pixel in the intermediate noise map to generate a target noise map of the reference frame. Each pixel value in the target noise map represents the noise information of the pixel at the corresponding position of the reference frame.

In this embodiment, the electronic device determines the shot noise and readout noise corresponding to the reference frame based on the shooting parameters of the reference frame, and can accurately generate a target noise map of the reference frame based on the shot noise and readout noise corresponding to the reference frame, thereby obtaining the noise information of the reference frame from the target noise map.

In one embodiment, as shown in FIG. 4 , the electronic device obtains at least two original images (Low Quality Image, low-quality input images) and a motion region mask of each original image, concatenates each original image and the respective motion region mask in the channel dimension to obtain a new original image; extracts sub-image features of each new original image in the at least two new original images; fuses at least two sub-image features through a convolutional layer or a self-attention mechanism to obtain a first image feature; determines shot noise and readout noise corresponding to the reference frame based on shooting parameters of the reference frame; generates a target noise map of the reference frame based on the shot noise and readout noise corresponding to the reference frame in the at least two original images; inputs the target noise map and the first image feature into a JDD (Joint Demosaic and In a joint demosaicing and denoising) module, a denoising process and a demosaicing process are performed on a first image feature based on a target noise map of a reference frame through a JDD module to obtain a second image feature; wherein the JDD module includes a UResNet network structure and 2x upsampling; the second image feature is upsampled at a preset ratio to obtain an upsampled second image feature, and the upsampled second image feature is a low-frequency component; the second image feature is input into an SR (Super-Resolution) module, and a super-resolution process is performed on the second image feature through the SR module to obtain a third image feature, and the third image feature is a high-frequency component; wherein the super-resolution process includes upsampling at a preset ratio; the upsampled second image feature and the third image feature are added to generate a target image.

In one embodiment, as shown in FIG5 , another image processing method is provided, comprising the following steps:

Step S502 : performing motion detection on non-reference frames other than the reference frame in at least two original images to determine a motion region of each non-reference frame.

Optionally, the electronic device uses a motion detection algorithm to perform motion detection on non-reference frames other than the reference frame in at least two original images to determine a motion region of each non-reference frame.

Optionally, the electronic device performs difference processing on non-reference frames other than the reference frame in at least two original images and the reference frame to obtain a difference image corresponding to each non-reference frame; and determines the motion area of each non-reference frame from the difference image corresponding to each non-reference frame.

For the difference image corresponding to each non-reference frame, the electronic device can compare each pixel value in the difference image with a preset threshold, take pixels with pixel values greater than the preset threshold as motion pixels, and form the motion region of the non-reference frame with each motion pixel. The preset threshold can be set as needed.

Optionally, in order to improve image processing efficiency, the electronic device may downsample at least two original images, perform motion detection on non-reference frames other than the reference frame in the at least two original images after downsampling, and determine the motion region of each non-reference frame.

Step S504: based on the image information of the reference frame, update the motion region of each non-reference frame to obtain an updated non-reference frame.

Optionally, the electronic device may update the motion region of each non-reference frame based on image information of the motion region position corresponding to the non-reference frame in the reference frame to obtain an updated non-reference frame.

In an optional implementation, for each non-reference frame, the motion region of the non-reference frame is replaced with image information of the reference frame corresponding to the position of the motion region of the non-reference frame to obtain an updated non-reference frame.

In another optional implementation, for each non-reference frame, the image information of the motion region position corresponding to the non-reference frame in the reference frame is used to cover the motion region of the non-reference frame to obtain an updated non-reference frame.

Step S506, using the reference frame and the updated non-reference frame as at least two new original images, based on the noise information of the reference frame in the at least two original images, denoising and demosaicing the first image features extracted from the at least two original images to obtain a second image feature; the frequency of the second image feature is lower than a preset frequency threshold.

The electronic device performs denoising and demosaicing processing on the first image features extracted from the new at least two original images based on the noise information of the reference frame in the new at least two original images to obtain the second image features.

Step S508: perform super-resolution processing on the second image feature to obtain a third image feature; the frequency of the third image feature is higher than or equal to a preset frequency threshold.

Step S510: Generate a target image according to the second image feature and the third image feature.

It is understandable that since there is a gap between the shooting times of at least two original images, the photographed object will inevitably have autonomous movement. When the movement is large, multi-frame registration will fail in the moving area, and problems such as motion ghosting will appear after the multi-frame images are fused.

In the present embodiment, the electronic device performs motion detection on non-reference frames other than the reference frame in at least two original images, and determines the motion area of each non-reference frame; based on the image information of the reference frame, the motion area of each non-reference frame is updated to obtain an updated non-reference frame, which can avoid problems such as ghosting after image fusion, eliminate large displacements between multiple frames, and improve the accuracy of image processing.

In one embodiment, as shown in FIG6 , another image processing method is provided, comprising the following steps:

Step S602: registering non-reference frames other than the reference frame in at least two original images with the reference frame to obtain a registered non-reference frame.

Optionally, the electronic device aligns the non-reference frame with the reference frame, which may specifically include operations such as displacement and interpolation, and may also include operations such as brightness alignment and image block alignment.

Optionally, non-reference frames other than the reference frame in at least two original images are aligned with the reference frame to obtain aligned non-reference frames, including: determining a motion transformation relationship between the reference frame and each non-reference frame based on the reference frame and the non-reference frames other than the reference frame in at least two original images; and aligning each non-reference frame with the reference frame based on the motion transformation relationship corresponding to the non-reference frame to obtain aligned non-reference frames.

The motion transformation relationship includes at least one of an affine transformation matrix and a feature point optical flow.

Optionally, in order to improve image processing efficiency, the electronic device cuts the reference frame and the non-reference frames other than the reference frame in at least two original images into blocks to obtain individual image blocks of the reference frame and individual image blocks of the non-reference frames other than the reference frame in at least two original images; performs corner point detection on each image block of the reference frame and each image block of the non-reference frame to obtain reference feature points of each image block of the reference frame and non-reference feature points of each image block of the non-reference frame; for each image block in each non-reference frame, the electronic device determines the motion transformation relationship between the non-reference feature points in the image block of the non-reference frame and the reference feature points of the image block at the corresponding position of the reference frame, and aligns the image block of the non-reference frame to the image block at the corresponding position of the reference frame according to the motion transformation relationship; based on each aligned image block in the non-reference frame, an aligned non-reference frame is obtained.

Optionally, for each image block in each non-reference frame, the electronic device determines an affine transformation matrix or feature point optical flow between non-reference feature points in the image block of the non-reference frame and reference feature points in the image block at a corresponding position in the reference frame, multiplies the image block of the non-reference frame by the affine transformation matrix or feature point optical flow, obtains the registered image block in the non-reference frame, and accurately aligns it to the image block at a corresponding position in the reference frame.

Optionally, the electronic device may perform smoothing filtering on adjacent image blocks, and then splice the image blocks after the smoothing filtering to obtain a registered non-reference frame, so that the transition area between adjacent image blocks in the registered non-reference frame is more natural.

Optionally, adjacent image blocks obtained by the electronic device during slicing have overlapping pixels, and the overlapping pixels of the adjacent image blocks are fused to obtain a registered non-reference frame, so that the transition area between adjacent image blocks in the registered non-reference frame is more natural.

Step S604: performing motion detection on non-reference frames other than the reference frame in at least two original images to determine a motion region of each non-reference frame.

Step S606: based on the image information of the reference frame, update the motion region of each registered non-reference frame to obtain an updated non-reference frame.

For each non-reference frame, the electronic device replaces the motion region of the registered non-reference frame with the image information of the reference frame corresponding to the position of the motion region of the non-reference frame to obtain an updated non-reference frame.

Step S608, using the reference frame and the updated non-reference frame as at least two new original images, based on the noise information of the reference frame in the at least two original images, denoising and demosaicing the first image features extracted from the at least two original images to obtain a second image feature; the frequency of the second image feature is lower than a preset frequency threshold.

Step S610, super-resolution processing is performed on the second image feature to obtain a third image feature; the frequency of the third image feature is higher than or equal to a preset frequency threshold.

Step S612: Generate a target image according to the second image feature and the third image feature.

It is understandable that, in the process of the electronic device capturing at least two original images, there is an overall displacement or a partial displacement between the original images, which may specifically include the displacement of the electronic device itself or the partial displacement of the captured object, etc., which may cause the overall displacement of the image in a short period of time. Therefore, the electronic device aligns the non-reference frames other than the reference frame in the at least two original images with the reference frame, and can subsequently fuse multiple image features more accurately, thereby improving the accuracy of image processing.

Electronic devices use high-precision and high-efficiency registration and alignment modules to restore details that are difficult to obtain with traditional single-frame algorithms through sub-pixel complementary information between multiple frames. At the same time, the effective use of multi-frame information can greatly improve image denoising capabilities. In addition, for multi-frame input, the high-precision motion region detection module and the replacement of non-reference frame motion regions with reference frame motion regions can effectively avoid ghosting problems caused by the motion of the captured object.

In one embodiment, determining a reference frame from at least two original images includes: determining the clarity of each of the at least two original images; and determining the reference frame from the at least two original images based on the clarity of each original image.

Optionally, the electronic device determines an original image with the highest definition from at least two original images as a reference frame.

Optionally, the electronic device determines an original image with the second highest definition from at least two original images as a reference frame.

It should be noted that the method of determining the reference frame from at least two original images is not limited.

Optionally, determining the clarity of each of the at least two original images includes: for each original image in the RAW domain, averaging the green channel of the original image in the RAW domain to generate a grayscale image; extracting a Gaussian difference operator from the grayscale image; and determining the clarity of the original image based on the Gaussian difference operator.

Optionally, for each original image in the RAW domain, the electronic device averages two green channels in the original image in the RAW domain to generate a grayscale image; extracts a Difference of Gaussian (DoG) operator from the grayscale image; and determines the clarity of the original image based on the Difference of Gaussian operator.

Optionally, determining the clarity of the original image based on a Gaussian difference operator includes: averaging the elements included in the Gaussian difference operator to obtain the clarity of the original image. The elements in the Gaussian difference operator include the size and variance of the Gaussian kernel, which can be obtained by analyzing and statistically analyzing the image data.

In the Gaussian difference operator, parameters such as the size and variance of the Gaussian kernel are fixed after being adjusted according to the data distribution. That is to say, the overall distribution of parameters such as the size and variance of the Gaussian kernel is statistically analyzed. Based on the overall distribution of the parameters, a set of sizes and variances of Gaussian kernels are determined to make them conform to the overall distribution of the data of the parameters. The parameters such as the size and variance of the Gaussian kernel are then fixed.

Optionally, the electronic device can use the average value of the Gaussian difference operator as the clarity of the original image, or use the average value of the Gaussian difference operator as the clarity score of the original image, and determine the clarity ranking of each original image according to the clarity score of each original image. The Gaussian difference operator is a matrix of image length*image width, and all element parameters of the matrix are averaged to obtain the clarity score. The highest clarity score means that the corresponding original image has the highest clarity.

It is understandable that, since in the original image in the RAW domain, the green channel has a high sampling rate and the human eye is more sensitive to it, the green channel carries more information. Therefore, averaging the two green channels in the original image in the RAW domain can obtain a grayscale image with more information, thereby more accurately determining the clarity of the original image.

Electronic devices acquire original images in the Raw domain, which can retain the image signal received by the image sensor during shooting to the maximum extent, avoiding the destruction of image detail information, structural information, color brightness information, etc. caused by previous traditional algorithms such as traditional demosaicing, denoising, tone mapping, etc. At the same time, it can avoid information loss caused by image dynamic range compression, etc. Compared with traditional YUV domain or RGB domain images, it has greater tolerance and better visual effects.

In one embodiment, as shown in FIG7 , the electronic device shoots through the image sensor of the lens module and dumps at least two original images; calculates the clarity of each original image, sorts them according to the clarity, removes the original images with low clarity, and obtains the remaining original images; determines the reference frame and the non-reference frame from the remaining original images; cuts and extracts feature points from the reference frame and the non-reference frame respectively, and calculates the optical flow or affine transformation matrix between the reference frame and the non-reference frame; aligns the non-reference frame to the reference frame based on the optical flow or affine transformation matrix; performs motion region detection on each non-reference frame to determine the motion region mask of each non-reference frame; based on the motion region mask of each non-reference frame, replaces the motion region of the non-reference frame after alignment with the reference frame pixel, and obtains multiple frames of images to be input into the network. Among them, the lens module can be a telephoto module, and the network refers to a joint demosaicing, denoising, and super-resolution network.

In one embodiment, the above method also includes: mapping the second image feature to the RGB domain, and mapping the third image feature to the RGB domain; generating a target image based on the second image feature and the third image feature, including: adding the second image feature in the RGB domain and the third image feature in the RGB domain to generate the target image.

Optionally, the electronic device inputs the second image feature into two convolutional layers, maps the second image feature to the RGB domain through the two convolutional layers, and obtains the second image feature in the RGB domain; inputs the third image feature into two convolutional layers, maps the third image feature to the RGB domain through the two convolutional layers, and obtains the third image feature in the RGB domain; adds the second image feature in the RGB domain and the third image feature in the RGB domain to generate a target image in the RGB domain.

In this embodiment, the electronic device maps both the second image feature and the third image feature to the RGB domain, and can accurately generate a target image in the RGB domain.

In one embodiment, the method further includes: capturing and obtaining at least two original images while locking the shooting parameters.

Optionally, the shooting parameters include automatic exposure parameters (AE, Automatic Exposure), automatic focus parameters (AF, Auto Focus) or automatic white balance parameters (AWB, Automatic White Balance), etc. The shooting parameters also include sensitivity, shutter time, etc., but are not limited thereto.

Optionally, the electronic device continuously captures the same shooting scene to obtain at least two original images while locking the shooting parameters.

It is understandable that the electronic device stores each frame of the original image in a queue during the continuous shooting process, and when the user triggers the shutter, at least two original images are obtained from the register.

Optionally, to avoid large displacement caused by the movement of the electronic device during shooting or the movement of the object being shot, the electronic device can control the shutter duration to be less than or equal to a preset shutter duration. The preset shutter duration can be set as needed, and the preset shutter duration is also the maximum shutter duration of the electronic device during shooting.

In this embodiment, the electronic device captures at least two original images while locking the shooting parameters, so that the at least two original images captured can maintain consistency in image features such as brightness and color, thereby improving the accuracy of subsequent image processing.

In one embodiment, as shown in FIG8 , an electronic device shoots and dumps multiple RAW images, the number of which may be N; selects frames from the multiple RAW images, i.e., selects reference frames and non-reference frames, and the electronic device may calculate the clarity of each RAW image, sorts them according to the clarity, and selects the reference frame and other non-reference frames (the total number is less than N) with the highest clarity; aligns the non-reference frame to the reference frame to obtain an aligned RAW image, the aligned RAW image includes the reference frame and each aligned non-reference frame; calculates the motion region mask of the non-reference frame relative to the reference frame; replaces the motion region of the non-reference frame with the image information of the corresponding position of the reference frame according to the motion region mask, connects each image with the motion region mask in series, and sends them to a feature extraction network to extract the features of each image, and obtains a first image feature; obtains a target noise image of the reference frame according to a pre-calibrated sensor noise model, and inputs it into a joint denoising, demosaicing and super-resolution network after being connected in series with the first image feature, and finally obtains a target image. The target image may be input into a subsequent image processing engine for processing.

In one embodiment, according to the optimization requirements for image details, smearing, etc. in different scenarios, the electronic device can also support sharpening of the reference frame and adjusting the noise map when the texture is adjusted, so as to control the denoising intensity, superimpose grayscale noise on the output result of the JDD module, and other operations.

Optionally, the electronic device sharpens the reference frame in the input Raw domain original image, so that the joint denoising, demosaicing and super-resolution network can retain more weak textures during processing, reduce the smearing effect, and avoid artifacts such as black and white edges caused by sharpening in the RGB domain or YUV domain.

Optionally, the denoising strength of the joint denoising, demosaicing and super-resolution network is affected by the target noise map of the reference frame. By adjusting the target noise map input to the joint denoising, demosaicing and super-resolution network during inference, the denoising strength of the joint denoising, demosaicing and super-resolution network can be controlled to achieve a balance between smearing and noise.

In addition, the joint denoising, demosaicing and super-resolution network can adjust the denoising strength of the image globally or locally. For example, for images taken at night, the electronic device can control the joint denoising, demosaicing and super-resolution network to improve the denoising strength as a whole; for images taken during the day, the electronic device can control the joint denoising, demosaicing and super-resolution network to determine the dark area of the image and improve the denoising strength of the dark area.

Optionally, due to the particularity of human eye perception, granular grayscale noise can improve visual quality in texture areas. The electronic device can decouple the JDD module from the SR module to achieve the overlap of the original image noise on the JDD module output to reduce the smearing effect.

In one embodiment, an image processing method is also provided, comprising the following steps:

Step A1: Under the condition of locking the shooting parameters, at least two original images are captured.

Step A2: for each original image in the RAW domain, average the green channel in the original image in the RAW domain to generate a grayscale image; extract a Gaussian difference operator from the grayscale image; average each element included in the Gaussian difference operator to obtain the clarity of the original image.

Step A3, determining a reference frame from at least two original images based on the definition of each original image.

Step A4, based on the reference frame and non-reference frames other than the reference frame in at least two original images, determine the motion transformation relationship between the reference frame and each non-reference frame; based on the motion transformation relationship corresponding to each non-reference frame, align it with the reference frame to obtain the aligned non-reference frame; the motion transformation relationship includes at least one of an affine transformation matrix and a feature point optical flow.

Step A5: performing motion detection on non-reference frames other than the reference frame in at least two original images to determine a motion region of each non-reference frame.

Step A6: for each non-reference frame, replace the motion region of the registered non-reference frame with the image information of the reference frame corresponding to the position of the motion region of the non-reference frame to obtain an updated non-reference frame.

Step A7, taking the reference frame and the updated non-reference frame as the new at least two original images, obtaining the motion area mask of each non-reference frame except the reference frame in the new at least two original images; extracting the sub-image features of each original image based on the reference frame, the non-reference frame and the corresponding motion area mask.

Step A8: Fusing at least two sub-image features to obtain a first image feature.

Step A9, based on the shooting parameters of the reference frame, determine the shot noise and readout noise corresponding to the reference frame; multiply each pixel in the reference frame by the shot noise to obtain an intermediate noise map; add the readout noise to the intermediate noise map to generate a target noise map of the reference frame; the target noise map contains the noise information of the reference frame.

Step A10, performing feature mapping on the target noise map and the first image feature of the reference frame to obtain mapping features.

Step A11, downsample and upsample the mapping features in sequence to obtain upsampled features; the resolution of the upsampled features is the same as the resolution of the mapping features; based on the upsampled features and the mapping features, a denoised image feature is obtained; the denoised image feature is upsampled and residual processed to obtain a second image feature; the frequency of the second image feature is lower than a preset frequency threshold; the second image feature is mapped to the RGB domain.

Step A12, performing residual processing on the second image feature to obtain a residual processing feature; performing upsampling on the residual processing feature at a preset ratio to obtain a third image feature, and mapping the third image feature to the RGB domain; the frequency of the third image feature is higher than or equal to a preset frequency threshold.

Step A13, upsampling the second image feature in the RGB domain at a preset ratio to obtain the upsampled second image feature in the RGB domain; adding the upsampled second image feature in the RGB domain and the third image feature in the RGB domain to generate a target image.

It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

In one embodiment, as shown in FIG9 , an image processing device is provided, including: a first processing module 902, a second processing module 904 and an image generating module 906, wherein:

The first processing module 902 is used to perform denoising and demosaicing on the first image features extracted from the at least two original images based on the noise information of the reference frame in the at least two original images to obtain the second image features; the frequency of the second image features is lower than the preset frequency threshold.

The second processing module 904 is used to perform super-resolution processing on the second image feature to obtain a third image feature; the frequency of the third image feature is higher than or equal to a preset frequency threshold.

The image generation module 906 is used to generate a target image according to the second image feature and the third image feature.

The above-mentioned image processing device, based on the noise information of the reference frame in the at least two original images, performs denoising and demosaicing on the first image feature extracted from the at least two original images to obtain a second image feature with a frequency lower than a preset frequency threshold; and performs super-resolution processing on the second image feature to obtain a third image feature with a frequency higher than or equal to the preset frequency threshold; that is, through the denoising and demosaicing processing, the low-frequency color contour and other information of the image can be more accurately obtained, and through the super-resolution processing, the high-frequency texture details and other information of the image can be more accurately obtained, thereby generating a denoised, demosaiced and super-resolution target image based on the second image feature and the third image feature, thereby improving the accuracy of image processing.

In one embodiment, the first processing module 902 is also used to perform denoising processing on the first image features extracted from at least two original images based on the noise information of the reference frame in the at least two original images to obtain denoised image features; and perform demosaicing processing on the denoised image features to obtain second image features.

In one embodiment, the first processing module 902 is further used to perform feature mapping on the noise information of the reference frame in at least two original images and the first image features extracted from at least two original images to obtain mapping features; and to perform denoising on the mapping features to obtain denoised image features.

In one embodiment, the first processing module 902 is also used to sequentially downsample and upsample the mapping features to obtain upsampled features; the resolution of the upsampled features is the same as the resolution of the mapping features; and based on the upsampled features and the mapping features, denoised image features are obtained.

In one embodiment, the first processing module 902 is further used to perform up-sampling and residual processing on the denoised image features to obtain second image features.

In one embodiment, the second processing module 904 is further configured to perform residual processing on the second image feature to obtain a residual processing feature; and upsample the residual processing feature to obtain a third image feature.

In one embodiment, the second processing module 904 is also used to upsample the residual processing features at a preset ratio to obtain a third image feature; the first processing module 902 is also used to upsample the second image features at a preset ratio to obtain the upsampled second image features; the image generation module 906 is also used to generate a target image based on the upsampled second image features and the third image features.

In one embodiment, the first processing module 902 is further configured to extract sub-image features of each original image from at least two original images; and fuse the at least two sub-image features to obtain the first image features.

In one embodiment, the first processing module 902 is further used to obtain a motion region mask of each non-reference frame except a reference frame in at least two original images; and extract sub-image features of each original image based on the reference frame, the non-reference frame and the corresponding motion region mask.

In one embodiment, the first processing module 902 is further used to determine the shot noise and readout noise corresponding to the reference frame based on the shooting parameters of the reference frame; based on the shot noise and readout noise corresponding to the reference frame, generate a target noise map of the reference frame; the target noise map contains the noise information of the reference frame.

In one embodiment, the first processing module 902 is further configured to multiply each pixel in the reference frame by the shot noise to obtain an intermediate noise map; and add the readout noise to the intermediate noise map to generate a target noise map of the reference frame.

In one embodiment, the above-mentioned device also includes a motion detection module; the motion detection module is used to perform motion detection on non-reference frames other than the reference frame in at least two original images, and determine the motion area of each non-reference frame; based on the image information of the reference frame, the motion area of each non-reference frame is updated to obtain an updated non-reference frame, and the reference frame and the updated non-reference frame are used as the new at least two original images. The above-mentioned first processing module 902 is used to perform denoising and demosaicing on the first image features extracted from the at least two original images.

In one embodiment, the motion detection module is further used to replace the motion region of each non-reference frame with image information of the reference frame corresponding to the position of the motion region of the non-reference frame to obtain an updated non-reference frame.

In one embodiment, the above-mentioned device also includes a registration module; the registration module is used to align the non-reference frames other than the reference frame in at least two original images with the reference frame to obtain the registered non-reference frames; the above-mentioned motion detection module is also used to update the motion area of each registered non-reference frame based on the image information of the reference frame to obtain the updated non-reference frame.

In one embodiment, the above-mentioned registration module is also used to determine the motion transformation relationship between the reference frame and each non-reference frame in the reference frame and at least two original images other than the reference frame; based on the motion transformation relationship corresponding to each non-reference frame, register it with the reference frame to obtain the registered non-reference frame.

In one embodiment, the upper motion transformation relationship includes at least one of an affine transformation matrix and a feature point optical flow.

In one embodiment, the above-mentioned device also includes a reference frame determination module; the reference frame determination module is used to determine the clarity of each original image in at least two original images; based on the clarity of each original image, determine the reference frame from the at least two original images.

In one embodiment, the reference frame determination module is also used to average the green channel of each original image in the RAW domain to generate a grayscale image; extract a Gaussian difference operator from the grayscale image; and determine the clarity of the original image based on the Gaussian difference operator.

In one embodiment, the reference frame determination module is further used to average the elements included in the Gaussian difference operator to obtain the clarity of the original image.

In one embodiment, the first processing module 902 is further used to map the second image feature to the RGB domain, and the second processing module 904 is further used to map the third image feature to the RGB domain; the image generation module 906 is further used to add the second image feature in the RGB domain and the third image feature in the RGB domain to generate a target image.

In one embodiment, the above device is also used in a shooting module; the shooting module is used to capture at least two original images under the condition of locking shooting parameters.

Each module in the above image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so that the processor can call and execute operations corresponding to each module.

In one embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG10. The electronic device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input/output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input/output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input/output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the electronic device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device housing, or an external keyboard, touchpad or mouse.

Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

The embodiment of the present application further provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, enable the processors to perform the steps of the image processing method.

The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the image processing method.

It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims (25)

1. An image processing method, comprising:

Denoising and demosaicing the first image features extracted from at least two original images based on noise information of reference frames in the at least two original images to obtain second image features; the frequency of the second image feature is lower than a preset frequency threshold;

performing super-resolution processing on the second image features to obtain third image features; the frequency of the third image feature is higher than or equal to a preset frequency threshold;

and generating a target image according to the second image characteristic and the third image characteristic.

2. The method according to claim 1, wherein the performing a de-noising process and a demosaicing process on the first image feature extracted from the at least two original images based on noise information of reference frames in the at least two original images to obtain the second image feature includes:

denoising the first image features extracted from at least two original images based on noise information of reference frames in the at least two original images to obtain denoised image features;

And performing demosaicing processing on the denoised image characteristics to obtain second image characteristics.

3. The method according to claim 2, wherein denoising the first image feature extracted from the at least two original images based on noise information of reference frames in the at least two original images, to obtain a denoised image feature, comprises:

Performing feature mapping on noise information of reference frames in at least two original images and first image features extracted from the at least two original images to obtain mapping features;

and denoising the mapping characteristics to obtain denoised image characteristics.

4. A method according to claim 3, wherein denoising the mapped feature results in a denoised image feature, comprising:

Sequentially performing downsampling and upsampling on the mapping features to obtain upsampling features; the resolution of the upsampled features is the same as the resolution of the mapped features;

And obtaining the denoising image characteristic based on the upsampling characteristic and the mapping characteristic.

5. The method of claim 2, wherein said demosaicing the de-noised image features to obtain second image features comprises:

And carrying out up-sampling and residual error processing on the denoising image features to obtain second image features.

6. The method of claim 1, wherein performing super-resolution processing on the second image feature to obtain a third image feature comprises:

Residual processing is carried out on the second image feature to obtain a residual processing feature;

and upsampling the residual processing feature to obtain a third image feature.

7. The method of claim 6, wherein upsampling the residual processing feature to obtain a third image feature comprises:

Up-sampling the residual processing features at a preset multiplying power to obtain third image features;

The method further comprises the steps of:

upsampling the second image feature by the preset multiplying power to obtain an upsampled second image feature;

the generating a target image according to the second image feature and the third image feature comprises:

and generating a target image according to the second image characteristic after upsampling and the third image characteristic.

8. The method of claim 1, wherein extracting a first image feature from the at least two original images comprises:

extracting sub-image features of each original image from the at least two original images;

And fusing at least two sub-image features to obtain a first image feature.

9. The method of claim 8, wherein extracting sub-image features of each original image from the at least two original images comprises:

Acquiring a motion area mask of each non-reference frame except the reference frame in the at least two original images;

sub-image features of each original image are extracted based on the reference frame, the non-reference frame, and a corresponding motion region mask.

10. The method according to claim 1, wherein the determining the noise information of the reference frame includes:

determining shot noise and readout noise corresponding to the reference frame based on shooting parameters of the reference frame;

Generating a target noise map of the reference frame based on shot noise and readout noise corresponding to the reference frame; the target noise map includes noise information for the reference frame.

11. The method of claim 10, wherein generating the target noise map for the reference frame based on shot noise and readout noise corresponding to the reference frame comprises:

multiplying each pixel in the reference frame by the shot noise respectively to obtain an intermediate noise diagram;

And adding the intermediate noise map to the readout noise to generate a target noise map of the reference frame.

12. The method according to claim 1, wherein the method further comprises:

Performing motion detection on non-reference frames except for reference frames in the at least two original images, and determining a motion area of each non-reference frame;

and updating a motion area of each non-reference frame based on the image information of the reference frame to obtain an updated non-reference frame, taking the reference frame and the updated non-reference frame as at least two new original images, and executing the steps of denoising and demosaicing the first image features extracted from the at least two original images.

13. The method of claim 12, wherein updating the motion region of each non-reference frame based on the image information of the reference frame to obtain updated non-reference frames comprises:

And for each non-reference frame, replacing the motion area of the non-reference frame with the image information of the motion area position corresponding to the non-reference frame in the reference frame to obtain an updated non-reference frame.

14. The method according to claim 12, wherein the method further comprises:

registering non-reference frames except for the reference frames in the at least two original images to the reference frames to obtain registered non-reference frames;

Updating the motion area of each non-reference frame based on the image information of the reference frame to obtain updated non-reference frames, including:

And updating the motion area of each registered non-reference frame based on the image information of the reference frame to obtain updated non-reference frames.

15. The method of claim 14, wherein registering non-reference frames other than the reference frame in the at least two original images to the reference frame results in a registered non-reference frame, comprising:

Determining a motion transformation relationship between the reference frame and each non-reference frame based on the reference frame and non-reference frames other than the reference frame in the at least two original images;

Registering the reference frames based on the motion transformation relation corresponding to each non-reference frame to obtain registered non-reference frames.

16. The method of claim 15, wherein the motion transformation relationship comprises at least one of an affine transformation matrix and a feature point optical flow.

17. The method according to any one of claims 1 to 16, wherein determining a reference frame from at least two original images comprises:

Determining the definition of each of at least two original images;

a reference frame is determined from at least two original images based on the sharpness of each original image.

18. The method of claim 17, wherein determining sharpness of each of the at least two original images comprises:

For each RAW domain original image, carrying out average processing on a green channel in the RAW domain original image to generate a gray level image;

extracting a Gaussian difference operator from the gray level map;

and determining the definition of the original image based on the Gaussian difference operator.

19. The method of claim 18, wherein the determining the sharpness of the original image based on the gaussian difference operator comprises:

and averaging all elements included in the Gaussian difference operator to obtain the definition of the original image.

20. The method according to any one of claims 1 to 16, further comprising:

mapping the second image feature to an RGB domain and mapping the third image feature to an RGB domain;

the generating a target image according to the second image feature and the third image feature comprises:

And adding the second image features of the RGB domain and the third image features of the RGB domain to generate a target image.

21. The method according to any one of claims 1 to 16, further comprising:

In the case of locking the shooting parameters, at least two original images are shot.

22. An image processing apparatus, comprising:

The first processing module is used for carrying out denoising processing and demosaicing processing on first image features extracted from at least two original images based on noise information of reference frames in the at least two original images to obtain second image features; the frequency of the second image feature is lower than a preset frequency threshold;

The second processing module is used for performing super-resolution processing on the second image features to obtain third image features; the frequency of the third image feature is higher than or equal to a preset frequency threshold;

And the image generation module is used for generating a target image according to the second image characteristic and the third image characteristic.

23. An electronic device comprising a memory and a processor, the memory having stored therein a computer program which, when executed by the processor, causes the processor to perform the steps of the image processing method according to any of claims 1 to 21.

24. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method according to any one of claims 1 to 21.

25. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 21.