CN111429371A - Image processing method and device and terminal equipment - Google Patents

Abstract

The present application is applicable to the technical field of image processing, and provides an image processing method, device and terminal device, including: acquiring an image to be reconstructed; extracting high-frequency components and low-frequency components of the to-be-reconstructed image to obtain a The first high-frequency image composed of the first high-frequency image and the first low-frequency image composed of the low-frequency components; the first high-frequency image is input into the trained high-frequency image generation network, and the second output of the high-frequency image generation network is obtained. high-frequency image; inputting the first low-frequency image into the trained low-frequency image generation network, to obtain a second low-frequency image output by the low-frequency image generation network; generating according to the second high-frequency image and the second low-frequency image The reconstructed image, the resolution of the reconstructed image is higher than the resolution of the image to be reconstructed. By the above method, a better image can be reconstructed.

Description

Image processing method, device and terminal device

technical field

The present application belongs to the technical field of image processing, and in particular, relates to an image processing method, apparatus, terminal device, and computer-readable storage medium.

Background technique

Image resolution refers to the ability of a sensor to see or measure the smallest objects, and it depends on pixel size. And digital images, recorded as two-dimensional signals, always require higher resolution in most applications. Imaging technology has advanced rapidly over the past few decades, and image resolution has reached a new level, but there are still many applications where better image resolution is desired. For example, digital surveillance products often choose to sacrifice image resolution to some extent to ensure long-term stable operation of recording equipment and appropriate frame rates for dynamic scenes; a similar situation exists in remote sensing: spatial, spectral and image resolution There is always a trade-off; and for medical imaging, how to extract a 3D model of the human body structure with the first image while reducing radiation remains a challenge.

Currently, the resolution of images can be increased by super-resolution methods.

Existing super-resolution methods can be roughly divided into two categories: traditional interpolation methods and deep learning-based methods. Traditional interpolation algorithms have been developed for decades, but in the field of Single Image Super-Resolution (SISR), their effects are far inferior to deep learning methods. Therefore, many new algorithms use data-driven deep learning models to reconstruct the required details of images for accurate super-resolution.

At present, most of the SISR improvements based on deep learning methods are aimed at the model structure or training methods. These methods work well for super-resolution of low-resolution (LR) images obtained by simulation, but when dealing with real acquired images When the second image is used, the effect will be significantly reduced, such as the current SOTA (state of the art, the most advanced level) super-resolution model ESRGAN (X.Wang et.al., "ESRGAN: Enhanced Super-ResolutionGenerative Adversarial Networks" in ECCV 2018workshop) when processing the image obtained by the smartphone, there is obvious over-fitting and blurring of details after reconstruction, and the effect is not even as good as the low-resolution image itself (Fig. 1(a) is the original image, and Fig. 1(b) is the original image. image after over-score). This is because, when processing the second image obtained by a similar smartphone, it contains a lot of noise and artifacts generated in the process of image processing, and the model structure based on deep learning will treat a lot of noise and artifacts as details. Therefore, the image quality after super-resolution will be reduced, so we need to distinguish the details that need to be enhanced from image noise and artifacts in the post-processing process at the source of super-resolution processing, so as to improve super-resolution image quality after reconstruction.

Therefore, it is necessary to propose a new technical solution to solve the above-mentioned technical problems.

SUMMARY OF THE INVENTION

The embodiments of the present application provide an image processing method, which can solve the problem of excessive pseudo details generated by the existing method after reconstructing an image.

In a first aspect, an embodiment of the present application provides an image processing method, including:

Get the image to be reconstructed;

Extracting high-frequency components and low-frequency components of the image to be reconstructed, to obtain a first high-frequency image composed of the high-frequency components and a first low-frequency image composed of the low-frequency components, wherein the high-frequency components are Refers to a frequency component greater than or equal to a preset frequency threshold, and the low-frequency component refers to a frequency component less than the preset frequency threshold;

Inputting the first high-frequency image into the trained high-frequency image generation network to obtain a second high-frequency image output by the high-frequency image generation network;

Inputting the first low-frequency image into the trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;

A reconstructed image is generated according to the second high frequency image and the second low frequency image, and the resolution of the reconstructed image is higher than the resolution of the image to be reconstructed.

In a second aspect, an embodiment of the present application provides an image processing apparatus, including:

an image acquisition unit to be reconstructed, used for acquiring the image to be reconstructed;

a high- and low-frequency image extraction unit, configured to extract high-frequency components and low-frequency components of the image to be reconstructed, to obtain a first high-frequency image composed of the high-frequency components and a first low-frequency image composed of the low-frequency components, Wherein, the high-frequency component refers to a frequency component greater than or equal to a preset frequency threshold, and the low-frequency component refers to a frequency component less than the preset frequency threshold;

The second high-frequency image generation unit is used to input the first high-frequency image into a trained high-frequency image generation network to obtain a second high-frequency image output by the high-frequency image generation network;

A second low-frequency image generation unit, configured to input the first low-frequency image into a trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;

An image reconstruction unit, configured to generate a reconstructed image according to the second high-frequency image and the second low-frequency image, where the resolution of the reconstructed image is higher than the resolution of the image to be reconstructed.

In a third aspect, an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, when the processor executes the computer program A method as described in the first aspect is implemented.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to the first aspect is implemented.

In a fifth aspect, an embodiment of the present application provides a computer program product that, when the computer program product runs on a terminal device, enables the terminal device to execute the method described in the first aspect.

The beneficial effects that the embodiments of the present application have compared with the prior art are:

Since the high-frequency components and low-frequency components of the image are processed separately, noise and artifacts can be suppressed while details are enhanced, so that a super-resolution image with less noise and artifacts and clearer details can be reconstructed.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings that are required to be used in the description of the embodiments or the prior art.

Figure 1(b) and Figure 1(a) are schematic diagrams comparing the reconstructed image provided by the prior art with the original image;

2 is a flowchart of an image processing method provided in Embodiment 1 of the present application;

3 is a flowchart of another image processing method provided in Embodiment 1 of the present application;

4 is a schematic flowchart of a method for generating network training provided in Embodiment 1 of the present application;

5 is a schematic structural diagram of a high-frequency image generation network provided in Embodiment 1 of the present application;

6 is a schematic structural diagram of a discriminant network to be trained provided by Embodiment 1 of the present application;

7 is a schematic flowchart of another method for generating network training provided in Embodiment 1 of the present application;

8 is a structural block diagram of an image processing apparatus provided in Embodiment 2 of the present application;

FIG. 9 is a schematic structural diagram of a terminal device according to Embodiment 3 of the present application.

Detailed ways

In the following description, for the purpose of illustration rather than limitation, specific details such as a specific system structure and technology are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It is to be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described feature, integer, step, operation, element and/or component, but does not exclude one or more other The presence or addition of features, integers, steps, operations, elements, components and/or sets thereof.

It will also be understood that, as used in this specification and the appended claims, the term "and/or" refers to and including any and all possible combinations of one or more of the associated listed items.

As used in the specification of this application and the appended claims, the term "if" may be contextually interpreted as "when" or "once" or "in response to determining" or "in response to detecting ". Similarly, the phrases "if it is determined" or "if the [described condition or event] is detected" may be interpreted, depending on the context, to mean "once it is determined" or "in response to the determination" or "once the [described condition or event] is detected. ]" or "in response to detection of the [described condition or event]".

In addition, in the description of the specification of the present application and the appended claims, the terms "first", "second", "third", "fourth", etc. are only used to distinguish the description, and should not be construed as indicating or implying relative importance.

References in this specification to "one embodiment" or "some embodiments" and the like mean that a particular feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless specifically emphasized otherwise. The terms "including", "including", "having" and their variants mean "including but not limited to" unless specifically emphasized otherwise.

Example 1:

At present, the improvement of image super-resolution based on deep learning methods is usually aimed at the model structure or training method. When these methods perform super-resolution processing on the second image obtained by simulation, the effect is very good, but when processing the second image obtained in real , the effect will be significantly reduced. In order to solve this technical problem, when training the high-frequency image generation network and the low-frequency image generation network, the training samples used in this application are all the real second images and real first images collected by the terminal device directly to ensure the training. The robustness of the post-high-frequency image generation network and the trained low-frequency image generation network when performing superdivision processing on the real second image. In addition, when training the high-frequency image generation network and the low-frequency image generation network, the present application extracts the high-frequency components and low-frequency components of the second image and the first image respectively, and then compares the high-frequency image composed of high-frequency components and the It is realized by processing the low-frequency image composed of low-frequency components. Due to the real second image collected by the terminal device (such as a smart phone), most of the artifacts generated by post-processing and the noise left by image compression are concentrated in the low-frequency area of the image. , the details that really need to be enhanced are concentrated in the high-frequency region of the image. Therefore, by processing the high-frequency and low-frequency components of the image separately, noise and artifacts can be suppressed while the details are enhanced. That is, the high-frequency image generated by the trained high-frequency image generation network and the low-frequency image generated by the trained low-frequency image generation network can reconstruct a super-segmented image with less noise and artifacts and clearer details.

The following describes in detail how to use the trained generation network to reconstruct the image to be reconstructed, wherein the generation network includes a high-frequency image generation network and a low-frequency image generation network.

FIG. 2 shows a flowchart of an image processing method provided by an embodiment of the present application, and details are as follows:

Step S21, acquiring the image to be reconstructed;

In this embodiment, the image to be reconstructed collected by the terminal device is acquired, and the resolution of the image to be reconstructed is relatively low. Specifically, the image to be reconstructed may be taken as the image to be reconstructed by taking or screenshot of the terminal device (such as a mobile phone), or the image to be reconstructed may be taken and transmitted by other terminal devices.

Step S22, extracting high-frequency components and low-frequency components of the image to be reconstructed, and obtaining a first high-frequency image composed of the high-frequency components and a first low-frequency image composed of the low-frequency components, wherein the high-frequency components are The frequency component refers to a frequency component greater than or equal to a preset frequency threshold, and the low-frequency component refers to a frequency component less than the preset frequency threshold;

Specifically, high-frequency components and low-frequency components of the image to be reconstructed are extracted through filters. For example, a low-pass filter (such as a Gaussian low-pass filter) is used to extract the low-frequency component of the image to be reconstructed, and then the low-frequency component of the image to be reconstructed is subtracted to obtain the high-frequency component of the image to be reconstructed. In some embodiments, an averaging filter can also be used to extract high frequency components and low frequency components of the image to be reconstructed.

If a Gaussian low-pass filter is used to extract the low-frequency components, it can be extracted in the following ways:

X _D = w _L *X (1)

Among them, "*" represents the convolution operation, w _L represents the Gaussian low-pass kernel, X represents the image to be reconstructed, and X _D represents the low-frequency component corresponding to X. The core of w _L can be 3x3 or 5x5. As an example, when w _L is 5x5,

After the low-frequency components are extracted, the corresponding high-frequency components are obtained by subtracting the original image from the low-frequency components:

X _H =XX _D (3)

Among them, X _H represents the high frequency component of X.

Step S23, inputting the first high-frequency image into a trained high-frequency image generation network to obtain a second high-frequency image output by the high-frequency image generation network;

Step S24, inputting the first low-frequency image into a trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;

Step S25, generating a reconstructed image according to the second high-frequency image and the second low-frequency image, where the resolution of the reconstructed image is higher than the resolution of the image to be reconstructed.

Specifically, the data corresponding to the second high-frequency image is used as the high-frequency data of the reconstructed image, and the data corresponding to the second low-frequency image is used as the low-frequency data of the reconstructed image, so as to obtain the reconstructed image.

In the embodiment of the present application, since the high-frequency components and low-frequency components of the image are processed separately, noise and artifacts can be suppressed while details are enhanced, so that an ultra-high-frequency image with less noise and artifacts and clearer details can be reconstructed. split image.

FIG. 3 shows a flowchart of another image processing method provided by an embodiment of the present application. In this embodiment, in order to reduce the running consumption of the network and improve the processing speed, only the data of the Y channel of the image to be reconstructed is processed. , detailed as follows:

Step S31, acquiring the image to be reconstructed;

Step S32, convert the color space of the image to be reconstructed into the YUV color space, extract the high-frequency component and the low-frequency component of the Y channel of the image to be reconstructed, and obtain the first high-frequency component composed of the high-frequency components of the Y channel. frequency image, and a first low-frequency image composed of low-frequency components of the Y channel, wherein the high-frequency components refer to frequency components greater than or equal to a preset frequency threshold, and the low-frequency components refer to frequencies less than the preset frequency threshold frequency components.

In this embodiment, the color space of the image to be reconstructed is usually a red, green and blue (RGB) color space. In order to reduce the influence of the processing process on the image color, the RGB color space is converted into a luminance and chrominance (YUV) color space. After converting to YUV color space, extract the Y channel of the YUV color space. If the NV21 standard parameters are used, the Y channel is extracted according to the following formula (4):

In this step, the Y channel of the image to be reconstructed is extracted according to formula (4) to obtain a grayscale image corresponding to the data of the Y channel, and then the low frequency corresponding to the grayscale image is extracted according to the above formulas (1) to (3). components and high-frequency components, and then obtain a first low-frequency image composed of the extracted low-frequency components and a first high-frequency image composed of the corresponding high-frequency components.

Step S33, inputting the first high-frequency image into a trained high-frequency image generation network to obtain a second high-frequency image output by the high-frequency image generation network;

Step S34, inputting the first low-frequency image into a trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;

Step S35, generating a reconstructed image according to the second high-frequency image and the second low-frequency image, where the resolution of the reconstructed image is higher than the resolution of the image to be reconstructed.

In the embodiment of the present application, since only the data of the Y channel of the image to be reconstructed is extracted, the amount of data that needs to be processed by the high-frequency image generation network and the low-frequency image generation network is reduced, so the operation consumption of the network can be effectively reduced, Improve processing speed. In addition, since the RGB color space is converted into the YUV color space, the problem of color cast caused by directly processing the data of three RGB channels can be avoided.

The generation network training method provided by the embodiments of the present application will be introduced below, wherein the generation network here includes a high-frequency image generation network and a low-frequency image generation network, as shown in Figure 4:

Step S41, acquiring a training sample set, where the training sample set stores: a first image collected by a terminal device and a second image corresponding to the first image one-to-one, wherein the first image has a high resolution at the resolution of the second image;

Among them, the terminal equipment includes: mobile phone, camera, tablet computer and other equipment.

In this embodiment, both the first image and the second image stored in the training sample set are real images collected by the terminal device, that is, it is ensured that the samples used for training have real noise. The corresponding relationship between the first image and the second image can be stored in the mapping table after being determined, and then the second image corresponding to the first image can be determined by looking up the mapping table, or the first image corresponding to the second image can be determined. an image. In order to make the trained high-frequency image generation network and low-frequency image generation network more universal, it is necessary to enrich the diversity of samples as much as possible. For example, the samples include both images with rich backgrounds and images with single backgrounds; The images that are exposed include images with insufficient light; both images with normal exposure, images with abnormal exposure, and so on.

In addition, in order to make the difference between the image content contained in the first image and the content contained in the corresponding second image smaller, it is necessary to ensure that the time difference between the time when the terminal device acquires the first image and the time when the corresponding second image is acquired is sufficient. Small.

Step S42, obtaining a first image from the training sample set, and extracting a third high-frequency image composed of high-frequency components of the first image and a third low-frequency image composed of low-frequency components of the first image;

The high frequency components refer to frequency components greater than or equal to a preset frequency threshold, and the low frequency components refer to frequency components less than the preset frequency threshold.

Specifically, high frequency components and low frequency components of the first image are extracted through a filter. For example, a low-pass filter (such as a Gaussian low-pass filter) is used to extract the low-frequency component of the first image, and then the first image and the low-frequency component are subtracted to obtain the high-frequency component of the first image.

Step S43: Obtain a second image corresponding to the first image from the training sample set, and extract a fourth high-frequency image composed of high-frequency components of the second image and a low-frequency component of the second image. the fourth low frequency image;

The extraction of the low-frequency component and the high-frequency component of the second image is similar to the extraction of the low-frequency component and the high-frequency component of the first image in the above steps, and details are not described herein again.

Step S44, inputting the fourth high-frequency image into the high-frequency image generation network to be trained, to obtain the fifth high-frequency image output by the high-frequency image generation network to be trained;

Step S45, if the change of the output value of the loss function of the high-frequency image generation network to be trained is within a preset range, stop the training of the high-frequency image generation network to be trained, and obtain the high-frequency image after training. image generation network, otherwise, adjust the parameters of the high-frequency image generation network to be trained, and return to perform the step of inputting the fourth high-frequency image into the high-frequency image generation network to be trained and subsequent steps, wherein the The loss function of the high-frequency image generation network to be trained includes the discriminant network to be trained, the discriminant network to be trained is trained by the loss function of the discriminant network to be trained, and the loss function of the discriminant network to be trained The input includes the third high-frequency image and the fifth high-frequency image;

Among them, the high-frequency image generation network and the discriminant network constitute a generative adversarial network system, that is, the high-frequency image generation network and the discriminant network are mutually trained.

In this embodiment, if the absolute value difference between the adjacent output values of the loss function of the high-frequency image generation network to be trained is within a preset range (the preset range is a small range, for example, it is [0, 5 ]), it indicates that the loss function has reached a state of convergence. At this time, the training of the high-frequency image generation network to be trained is stopped, that is, the last trained high-frequency image generation network is used as the trained high-frequency image generation network. Of course, if the absolute value difference between the adjacent output values of the loss function is not within the preset range, return to step S42, step S43 and steps after step S43.

In some embodiments, the loss function of the high-frequency image generation network to be trained includes three parts: L _content , which represents the content loss, L _perceptual , which represents the visual difference, and L _texture , which represents the semantic loss, as shown in formulas (5)-(7) shown. Of course, if the loss function of the high-frequency image generation network to be trained includes the three parts listed above, then the change of the output value of the loss function within the preset range may refer to: the sum of the changes of the three parts is within the preset range; or It can mean that the change of any part is within the preset range, such as the change of the output value of L _content is within the preset range, the change of the output value of L _perceptual is within the preset range, and the change of L _texture is within the preset range .

表示第i个第五高频图像，m表示一批第四高频图像的个数，GT_H ⁽ⁱ⁾表示第i个第三高频图像，“||||”为L1范数。需要指出的是，在一些实施例中，该L1范数也可以换为L2范数。where G _H is the high-frequency image generation network to be trained, x _H ⁽ⁱ⁾ represents the i-th fourth high-frequency image,

represents the ith fifth high-frequency image, m represents the number of fourth high-frequency images in a batch, GT _H ⁽ⁱ⁾ represents the ith third high-frequency image, and “||||” is the L1 norm. It should be noted that, in some embodiments, the L1 norm may also be replaced by the L2 norm.

L _perceptual represents the visual difference between the input image and the output image of the high-frequency image generation network, where φ() represents the image passing through the VGG network (Justin Johnson et.al., 'Perceptual Losses for Real-TimeStyle Transfer and Super-Resolution' in ECCV 2016) (such as the features of the relu3_3 layer, or the features of the relu4_3 layer, or the features of the relu2_2 layer, or the features of the relu1_2 layer, etc.), C, H, W represent x _H ⁽ⁱ⁾ number of channels, height and width.

In some embodiments, the features obtained by the above-mentioned VGG network can also be obtained by learning Perceptual Image Patch Similarity (Learned Perceptual Image Patch Similarity, LPIPS).

L _texture represents the score of the fifth high-frequency image generated by the discriminant network (D _H ) to be trained. The closer it is to the image features of the real image GT _H , the higher the obtained score. Because the discriminant network to be trained obtains a feature map, the loss function (loss) of the semantic loss of a single image is the result of the average of the feature maps, that is, mean() in the formula means to average the feature maps.

The loss function (loss)LD used to train the discriminative network to be _trained :

Step S46, inputting the fourth low-frequency image into a low-frequency image generation network to be trained, to obtain a fifth low-frequency image output by the low-frequency image generation network to be trained;

Step S47, if the change of the output value of the loss function of the low-frequency image generation network to be trained is within a preset range, stop the training of the low-frequency image generation network to be trained, and obtain the low-frequency image generation network after training , otherwise, adjust the parameters of the low-frequency image generation network to be trained, and return to step S46 and step S47, wherein the input of the loss function of the low-frequency image generation network to be trained includes the third low-frequency image and the The fifth low-frequency image is described.

Wherein, the loss function of the low-frequency image generation network to be trained includes: L' _content representing the content loss and L' _perceptual representing the visual difference, wherein the definitions of L' _content and L' _perceptual refer to the above formula (5) and formula (6).

In the embodiment of the present application, since the samples for training the high-frequency image generation network and the low-frequency image generation network are real images collected by the terminal device, the trained high-frequency image generation network and the trained low-frequency image generation network are all real images. The output image is more accurate and more in line with the requirements. In addition, when training the high-frequency image generation network and the low-frequency image generation network, the high-frequency components and low-frequency components of the image are extracted respectively, and then the high-frequency image composed of high-frequency components and the low-frequency image composed of low-frequency components are processed. However, due to the artifacts generated by the post-processing of the image collected by the terminal device (such as a smartphone) and the noise left by image compression, most of the noise is concentrated in the low-frequency area of the image, and the details that really need to be enhanced are concentrated in the image. The high frequency region, therefore, can suppress noise and artifacts while detail is enhanced by processing the high frequency and low frequency components of the image separately. That is, the high-frequency image generated by the trained high-frequency image generation network and the low-frequency image generated by the trained low-frequency image generation network can reconstruct a super-segmented image with less noise and artifacts and clearer details.

FIG. 5 shows a structure diagram of a trained high-frequency image generation network provided by an embodiment of the present application. In FIG. 5 , the trained high-frequency image generation network includes: a first feature extraction layer, a first feature conversion layer layer, magnification layer, and first fusion layer, where M indicates that the first feature conversion layer has M layers. After the second high-frequency image passes through the first feature extraction layer, the first feature conversion layer, the amplification layer, and the first fusion layer, the third high-frequency image will be obtained.

Specifically, the first feature extraction layer is used to extract features; the first feature conversion layer is used to combine the extracted features; the amplification layer is used to amplify the combined features; the first fusion layer is used to The features are fused into one image output.

Wherein, the amplification layer realizes the amplification function through an interpolation algorithm, and the first feature extraction layer, the first feature conversion layer and the first fusion layer all realize the corresponding functions through a convolution algorithm. Since the interpolation algorithm and the convolution algorithm are combined, the third high-frequency image obtained can reduce the phenomenon of checkerboard noise.

In some embodiments, the first feature extraction layer includes at least two layers, and the number of channels in the latter layer is greater than the number of channels in the previous layer; the first feature conversion layer includes at least two layers, and the number of channels in each layer is greater than that in the previous layer. The number of channels remains unchanged; the convolution kernel of the first feature extraction layer and the convolution kernel of the first fusion layer are both larger than the convolution kernel of the first feature conversion layer, and the first feature extraction layer , the image scaling ratios corresponding to the first fusion layer and the first feature conversion layer are all greater than or equal to 1.

Since both the convolution kernel of the first feature extraction layer and the convolution kernel of the first fusion layer are larger than the convolution kernel of the first feature conversion layer, the obtained third high-frequency image has a wider field of view. Larger, more realistic details. In addition, since the application needs to perform super-score processing on the image, it is necessary to ensure that the image scaling ratio cannot be less than 1.

Among them, the first feature extraction layer is a process of extracting low-level features of an image by using a convolution kernel with a larger receptive field, and these low-level features include gradients, brightness, and size relationships. In order to extract more details, the number of layers of the first feature extraction layer is limited to include at least 2 layers. Assuming that the first feature extraction layer has 2 layers, the number of channels in the second layer is more than the number of channels in the first layer. Table 1 below illustrates the specific parameters of the first feature extraction layer with an example.

Table 1:

The first feature conversion layer is a process of nonlinearly combining the low-level features extracted by the first feature extraction layer to obtain high-level features, and these high-level features include structure, shape, and the like. The more layers, the higher the degree of nonlinearity of the feature, the more complex the image structure can be expressed, and the more beneficial to improve the authenticity of the reconstructed image. Since the smaller the convolution kernel is, the faster its processing speed is, and the main purpose of the first feature conversion layer is to combine the obtained low-level features, so there is no need to obtain more information through more channels. Therefore, set the first A feature conversion layer selects a convolution with a small convolution kernel, the size is unchanged (x1), the number of channels is also unchanged, and it is repeated M times. Here we choose M=16, and the specific parameters can refer to Table 2.

Table 2:

Among them, the enlargement layer is not a convolution layer, but an interpolation enlargement layer, which enlarges the width and height of the feature obtained by the first feature conversion layer to the required ratio, for example, enlarges 2 times (x2), the first fusion layer Immediately after the amplification layer, the purpose is to fuse the features of multiple channels into one image output of one channel. The parameters of the two layers can be referred to in Table 3.

table 3:

It should be pointed out that the structure of the low-frequency image generation network to be trained is similar to the structure of the high-frequency image generation network to be trained. For details, please refer to FIG. 5 .

In some embodiments, the discriminative network to be trained comprises: a second feature extraction layer for extracting features, a second feature transformation layer for combining the extracted features, and a feature fusion for at least 2 channels is the second fusion layer output from an image, and the image scaling ratios corresponding to the second feature extraction layer, the second feature conversion layer, and the second fusion layer are all less than 1.

Specifically, Fig. 6 shows a schematic diagram of the structure of the discriminant network to be trained. In Fig. 6, the first high-frequency image and the third high-frequency image are respectively input into the discriminant network to be trained. After the second feature conversion layer and the second fusion layer, input the loss function of the discriminant network to be trained for comparison to determine whether to continue training the discriminant network to be trained.

It should be pointed out that the functions of the second feature extraction layer, the second feature conversion layer and the second fusion layer are respectively similar to the functions of the first feature extraction layer, the first feature conversion layer and the first fusion layer, and will not be repeated here. .

In this embodiment, since the discriminant network to be trained mainly distinguishes the fifth high-frequency image and the third high-frequency image generated by the high-frequency image generation network to be trained, it does not need to obtain more information on the authenticity of the constructed image. The favorable information is that it is not necessary to set more layers for the second feature extraction layer and the second feature conversion layer, which is also beneficial to improve the operation speed. In addition, setting the image scaling ratios corresponding to the second feature extraction layer, the second feature conversion layer, and the second fusion layer to be less than 1 is also beneficial to improve the operation speed.

Specifically, the number of layers of the second feature extraction layer can be set to 1 layer, and Table 4 can be referred to for parameters of the second feature extraction layer.

Table 4:

The number of layers of the second feature conversion layer can be set to 2, and the parameters of the second feature conversion layer can refer to Table 5.

table 5:

The second fusion layer follows the second feature conversion layer, and its specific parameters can refer to Table 6:

Table 6:

FIG. 7 shows a flowchart of another method for training a generative network provided by an embodiment of the present application.

In FIG. 7, the resolution of the first picture is higher than the resolution of the second picture. The first picture will be converted into YUV color space through the color space conversion operation, and the data of the Y channel will be extracted, and then the high-frequency components and low-frequency components in the Y channel will be extracted through a low-pass filter to obtain the high-frequency components. A third high frequency picture and a third low frequency picture composed of the low frequency component.

The second picture is also converted into YUV color space through the color space conversion operation, and the data of the Y channel is extracted, and then the high-frequency component and the low-frequency component in the Y channel are extracted through a low-pass filter to obtain the high-frequency component. A fourth high frequency picture and a fourth low frequency picture composed of the low frequency component. The fourth high-frequency image is passed through the high-frequency image generation network to be trained to obtain a fifth high-frequency image output by the high-frequency image generation network to be trained. The fourth low-frequency image is passed through the low-frequency image generation network to be trained to obtain a fifth low-frequency image output by the low-frequency image generation network to be trained.

Taking the third low-frequency picture and the fifth low-frequency picture as the input of the above formula (5) and formula (6), calculate the content loss and visual difference of the low-frequency image generation network, and judge whether to continue the low-frequency image generation network according to the calculation results. train.

Taking the third and fifth high-frequency pictures as the input of the above formula (5), formula (6), formula (7) and formula (8), calculate the content loss, visual difference, semantics of high-frequency image generation network Loss, and calculate the output value of the loss function of the discriminant network, and determine whether to continue training the high-frequency image generation network according to the calculated content loss, visual difference, semantic loss, and the output value of the loss function of the discriminant network.

Embodiment 2:

Corresponding to the image processing method described in Embodiment 1 above, FIG. 8 shows a structural block diagram of an image processing apparatus provided by an embodiment of the present application. For convenience of description, only parts related to the embodiment of the present application are shown.

8 , the image processing device 8 includes: an image acquisition unit 81 to be reconstructed, a high and low frequency image extraction unit 82, a second high frequency image generation unit 83, a second low frequency image generation unit 84, and an image reconstruction unit 85, wherein:

an image acquisition unit 81 to be reconstructed, configured to acquire an image to be reconstructed;

High and low frequency image extraction unit 82, configured to extract high frequency components and low frequency components of the image to be reconstructed, and obtain a first high frequency image composed of the high frequency components and a first low frequency image composed of the low frequency components , wherein the high-frequency component refers to a frequency component greater than or equal to a preset frequency threshold, and the low-frequency component refers to a frequency component less than the preset frequency threshold;

The second high-frequency image generation unit 83 is used to input the first high-frequency image into a trained high-frequency image generation network to obtain a second high-frequency image output by the high-frequency image generation network;

The second low-frequency image generation unit 84 is configured to input the first low-frequency image into a trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;

The image reconstruction unit 85 is configured to generate a reconstructed image according to the second high-frequency image and the second low-frequency image, where the resolution of the reconstructed image is higher than the resolution of the image to be reconstructed.

In some embodiments, in order to reduce the running consumption of the network and improve the processing speed, only the data of the Y channel of the image is processed. At this time, the high and low frequency image extraction unit 82 is specifically used for:

Converting the color space of the image to be reconstructed into the YUV color space, extracting the high-frequency component and the low-frequency component of the Y channel of the image to be reconstructed, to obtain the first high-frequency image composed of the high-frequency component of the Y channel, and a first low frequency image consisting of the low frequency components of the Y channel.

In the embodiment of the present application, since the high-frequency components and low-frequency components of the image are processed separately, noise and artifacts can be suppressed while details are enhanced, so that an ultra-high-frequency image with less noise and artifacts and clearer details can be reconstructed. split image.

It should be noted that the information exchange, execution process and other contents between the above-mentioned devices/units are based on the same concept as the method embodiments of the present application. For specific functions and technical effects, please refer to the method embodiments section. It is not repeated here.

Embodiment three:

FIG. 9 is a schematic structural diagram of a terminal device according to Embodiment 3 of the present application. As shown in FIG. 9 , the terminal device 9 in this embodiment includes: at least one processor 90 (only one processor is shown in FIG. 9 ), a memory 91 , and a memory 91 that is stored in the memory 91 and can be processed in the at least one processor. The computer program 92 running on the processor 90, when the processor 90 executes the computer program 92, implements the steps in any of the foregoing method embodiments:

Get the image to be reconstructed;

Extracting high-frequency components and low-frequency components of the image to be reconstructed, to obtain a first high-frequency image composed of the high-frequency components and a first low-frequency image composed of the low-frequency components, wherein the high-frequency components are Refers to a frequency component greater than or equal to a preset frequency threshold, and the low-frequency component refers to a frequency component less than the preset frequency threshold;

Inputting the first high-frequency image into the trained high-frequency image generation network to obtain a second high-frequency image output by the high-frequency image generation network;

Inputting the first low-frequency image into the trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;

A reconstructed image is generated according to the second high frequency image and the second low frequency image, and the resolution of the reconstructed image is higher than the resolution of the image to be reconstructed.

The terminal device 9 may be a computing device such as a desktop computer, a notebook, a palmtop computer, and a cloud server. The terminal device may include, but is not limited to, a processor 90 and a memory 91 . Those skilled in the art can understand that FIG. 9 is only an example of the terminal device 9, and does not constitute a limitation on the terminal device 9. It may include more or less components than the one shown, or combine some components, or different components , for example, may also include input and output devices, network access devices, and the like.

The so-called processor 90 may be a central processing unit (Central Processing Unit, CPU), and the processor 90 may also be other general-purpose processors, digital signal processors (Digital Signal Processors, DSP), application specific integrated circuits (Application Specific Integrated Circuits) , ASIC), field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 91 may be an internal storage unit of the terminal device 9 in some embodiments, such as a hard disk or a memory of the terminal device 9 . The memory 91 may also be an external storage device of the terminal device 9 in other embodiments, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, flash memory card (Flash Card), etc. Further, the memory 91 may also include both an internal storage unit of the terminal device 9 and an external storage device. The memory 91 is used to store an operating system, an application program, a boot loader (Boot Loader), data, and other programs, such as program codes of the computer program. The memory 91 can also be used to temporarily store data that has been output or will be output.

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. Module completion, that is, dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated in one processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit, and the above-mentioned integrated units may adopt hardware. It can also be realized in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present application. For the specific working processes of the units and modules in the above-mentioned system, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

An embodiment of the present application also provides a network device, the network device includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, the processor executing The computer program implements the steps in any of the foregoing method embodiments.

Embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented.

The embodiments of the present application provide a computer program product, when the computer program product runs on a mobile terminal, the steps in the foregoing method embodiments can be implemented when the mobile terminal executes the computer program product.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the present application realizes all or part of the processes in the methods of the above embodiments, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When executed by a processor, the steps of each of the above method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form, and the like. The computer-readable medium may include at least: any entity or device capable of carrying the computer program code to the photographing device/terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, U disk, mobile hard disk, disk or CD, etc. In some jurisdictions, under legislation and patent practice, computer readable media may not be electrical carrier signals and telecommunications signals.

In the foregoing embodiments, the description of each embodiment has its own emphasis. For parts that are not described or described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

In the embodiments provided in this application, it should be understood that the disclosed apparatus/network device and method may be implemented in other manners. For example, the apparatus/network device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units. Or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the above-mentioned embodiments, those of ordinary skill in the art should understand that: it can still be used for the above-mentioned implementations. The technical solutions described in the examples are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the application, and should be included in the within the scope of protection of this application.

Claims (10)

1. An image processing method, comprising:

acquiring an image to be reconstructed;

extracting a high-frequency component and a low-frequency component of the image to be reconstructed to obtain a first high-frequency image composed of the high-frequency component and a first low-frequency image composed of the low-frequency component, wherein the high-frequency component is a frequency component larger than or equal to a preset frequency threshold, and the low-frequency component is a frequency component smaller than the preset frequency threshold;

inputting the first high-frequency image into a trained high-frequency image generation network to obtain a second high-frequency image output by the high-frequency image generation network;

inputting the first low-frequency image into a trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;

and generating a reconstructed image according to the second high-frequency image and the second low-frequency image, wherein the resolution of the reconstructed image is higher than that of the image to be reconstructed.

2. The image processing method according to claim 1, wherein said extracting high frequency components and low frequency components of the image to be reconstructed to obtain a first high frequency image composed of the high frequency components and a first low frequency image composed of the low frequency components comprises:

and converting the color space of the image to be reconstructed into a YUV color space, extracting the high-frequency component and the low-frequency component of the Y channel of the image to be reconstructed, and obtaining a first high-frequency image consisting of the high-frequency component of the Y channel and a first low-frequency image consisting of the low-frequency component of the Y channel.

3. The image processing method of claim 1, wherein the high frequency image generation network is trained by:

acquiring a training sample set, wherein the training sample set stores: the method comprises the steps that a first image and a second image are acquired by terminal equipment, wherein the first image corresponds to the first image one by one, and the resolution of the first image is higher than that of the second image;

acquiring a first image from the training sample set, and extracting a third high-frequency image formed by high-frequency components of the first image;

acquiring a second image corresponding to the first image from the training sample set, and extracting a fourth high-frequency image formed by high-frequency components of the second image;

inputting the fourth high-frequency image into a high-frequency image generation network to be trained to obtain a fifth high-frequency image output by the high-frequency image generation network to be trained;

if the variation of the output value of the loss function of the high-frequency image generation network to be trained is within a preset range, stopping training the high-frequency image generation network to be trained to obtain the trained high-frequency image generation network, otherwise, adjusting the parameters of the high-frequency image generation network to be trained, and returning to execute the step of inputting the fourth high-frequency image into the high-frequency image generation network to be trained and the subsequent steps, wherein the loss function of the high-frequency image generation network to be trained comprises a discriminant network to be trained, the discriminant network to be trained is trained through the loss function of the discriminant network to be trained, and the input of the loss function of the discriminant network to be trained comprises the third high-frequency image and the fifth high-frequency image.

4. The image processing method of any of claims 1 to 3, wherein the trained high frequency image generation network comprises: the system comprises a first feature extraction layer for extracting features, a first feature conversion layer for combining the extracted features, an amplification layer for amplifying the combined features, and a first fusion layer for fusing the amplified features of at least 2 channels into an image and outputting the image; the amplification layer realizes an amplification function through an interpolation algorithm, and the first feature extraction layer, the first feature conversion layer and the first fusion layer realize corresponding functions through a convolution algorithm.

5. The image processing method of claim 4, wherein the first feature extraction layer comprises at least 2 layers, and the number of channels of a subsequent layer is greater than the number of channels of a previous layer;

the first characteristic conversion layer at least comprises 2 layers, and the number of channels of each layer is kept unchanged;

the convolution kernel of the first feature extraction layer and the convolution kernel of the first fusion layer are both larger than the convolution kernel of the first feature conversion layer, and the image scaling ratios corresponding to the first feature extraction layer, the first fusion layer and the first feature conversion layer are all larger than or equal to 1.

6. The image processing method of claim 3, wherein the loss function of the high frequency image generation network to be trained further comprises L representing a loss of content_content：

Wherein G is_HGenerating a network, x, for the high-frequency image to be trained_H ⁽ⁱ⁾Denotes the ith fourth high frequency image, m denotes the number of the fourth high frequency images in a batch, GT_H ⁽ⁱ⁾Representing the ith third high-frequency image, "| | | |" is a norm of L1.

7. The image processing method of claim 3, wherein the discriminant network to be trained comprises: the image fusion system comprises a second feature extraction layer used for extracting features, a second feature conversion layer used for combining the extracted features, and a second fusion layer used for fusing the features of at least 2 channels into one image to be output, wherein image scaling ratios corresponding to the second feature extraction layer, the second feature conversion layer and the second fusion layer are all smaller than 1.

8. An image processing apparatus characterized by comprising:

the image acquisition unit to be reconstructed is used for acquiring an image to be reconstructed;

the high-low frequency image extraction unit is used for extracting a high-frequency component and a low-frequency component of the image to be reconstructed to obtain a first high-frequency image consisting of the high-frequency component and a first low-frequency image consisting of the low-frequency component, wherein the high-frequency component is a frequency component which is greater than or equal to a preset frequency threshold, and the low-frequency component is a frequency component which is less than the preset frequency threshold;

a second high-frequency image generating unit, which is used for inputting the first high-frequency image into the trained high-frequency image generating network to obtain a second high-frequency image output by the high-frequency image generating network;

the second low-frequency image generation unit is used for inputting the first low-frequency image into the trained low-frequency image generation network to obtain a second low-frequency image output by the low-frequency image generation network;

and the image reconstruction unit is used for generating a reconstructed image according to the second high-frequency image and the second low-frequency image, and the resolution of the reconstructed image is higher than that of the image to be reconstructed.

9. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 7.

Images