WO2020062191A1

WO2020062191A1 - Image processing method, apparatus and device

Info

Publication number: WO2020062191A1
Application number: PCT/CN2018/108891
Authority: WO
Inventors: 谭文伟
Original assignee: 华为技术有限公司
Priority date: 2018-09-29
Filing date: 2018-09-29
Publication date: 2020-04-02
Also published as: CN112088393B; CN112088393A

Abstract

Disclosed are an image processing method, apparatus and device. The method comprises: acquiring a target image that requires super-resolution processing; and inputting the target image into a super-resolution network model for processing so as to obtain a high-resolution image, wherein network parameters of the super-resolution network model are obtained by carrying out adjustment according to multiple frames of sample image and a semantic feature map corresponding to each frame of sample image, and the semantic feature map is obtained by means of an image semantic network model carrying out semantic recognition, thereby improving the quality of the obtained high-resolution image.

Description

Image processing method, device and equipment

Technical field

The present invention relates to image processing technology, and in particular, to an image processing method, device, and device.

Background technique

With the development of multimedia technology, users have higher and higher requirements for multimedia information. For example, high-resolution multimedia information (picture information or video information) has become a mainstream multimedia file.

When the terminal needs high-resolution multimedia information interaction, the terminal needs high-speed broadband to transmit high-resolution multimedia information, which will greatly increase the cost of information exchange between the two sides of the interactive terminal. Therefore, users usually convert high-resolution multimedia information to low-resolution multimedia information, and then send the low-resolution multimedia information to other terminals, which reduces the interaction cost. After receiving the low-resolution multimedia information, the receiving terminal needs to restore the low-resolution multimedia information to high-resolution multimedia information in order to obtain more detailed information. In practice, it has been found that the quality of the high-resolution multimedia information restored is poor.

Summary of the Invention

The invention provides an image processing method, device and equipment, which improve the accuracy of converting a low-resolution image into a high-resolution image, so as to improve the quality of the high-resolution image.

In a first aspect, an embodiment of the present invention provides an image packet processing method. The method includes: acquiring a target image requiring super-resolution processing; and inputting the target image to a super-scoring network model for processing to obtain a high-resolution image. Image; wherein the network parameters of the super-segmented network model are obtained by adjusting multi-frame sample images and the semantic feature maps corresponding to the sample images in each frame, and the semantic feature maps are semantically identified through the image semantic network model owned.

In this technical solution, since the network parameters of the super-segmented network model are adjusted based on a large number of sample images and the semantic feature images of each frame of sample images, the semantic feature images include detailed feature information and edge structure information of the sample images. Therefore, the super-segment network model is a semantic-enhanced network model, that is, a semantic-enhanced super-segment network model can convert a low-resolution image into a semantic-enhanced high-resolution image, and a semantic-enhanced high-resolution image can be provided. More detailed feature information can provide high-resolution edge structure information, which improves the quality of high-resolution images.

Optionally, an error of the super-scoring network model is determined according to the multi-frame sample image and a semantic feature map corresponding to each of the sample images; when the error is greater than a preset error value, the super-scoring network is determined. The network parameters of the model are adjusted.

In the embodiment of the present invention, in order to improve the accuracy of the image processed by the ultra-scoring network, the network parameters of the ultra-scoring network model may be adjusted according to the multi-frame sample image and the semantic feature map corresponding to each sample image.

Optionally, obtain a high-resolution sub-image and a low-resolution sub-image corresponding to each frame of the multi-frame sample images, input each frame of the target sub-image into the image semantic network model, and perform semantic recognition to obtain each frame. The semantic feature image corresponding to the sample image, and the target sub-image is a high-resolution sub-image or a low-resolution sub-image corresponding to any of the sample images in the multi-frame sample image; Input to the super-scoring network model for processing to obtain a high-resolution feature image of the sample image of each frame; superimpose the high-resolution sub-image of the sample image of each frame with the semantic feature image of the corresponding sample image to obtain an overlay Determine the difference between the high-resolution feature image of each sample image and the superimposed image of the corresponding sample image; calculate the sum of the differences, and use the sum of the differences as the error of the super-scoring network model.

In the technical solution, a high-resolution sub-image of the sample image and a superimposed image of the semantic feature image of the sample image are used as reference images, and a low-resolution sub-image of the sample image is used as a training sample. The super image is calculated based on the reference image and the training sample image. The error of the sub-network model is in order to obtain the super-sub-network model with lower error.

Optionally, setting a weight for the image output by the image semantic network model; processing the semantic feature image of the sample image of each frame according to the weight to obtain a processed semantic feature map; and processing the sample image of each frame The superimposed high-resolution sub-image and the processed semantic feature image corresponding to the sample image are superimposed to obtain a superimposed image.

In this technical solution, the image processing device may set weights for the images output by the image semantic network model, in order to obtain an incapable super-segment network model to meet the different image needs of the user, that is, the larger the weight value, it indicates the semantics in the superimposed image The more information provided by the feature image, the higher the clarity of the superimposed image, and further, the high-resolution image output by the super-scoring network model is closer to the semantic feature image; otherwise, the smaller the weight value, it indicates that the semantic feature image in the superimposed image provides The less information there is, the lower the clarity of the superimposed image, and the closer the high-resolution image output by the super-network model is to the target sub-image.

Optionally, the image semantic network model includes a multi-layer neural network, and the target sub-image is input into the image semantic network model. The multi-layer neural network included in the image semantic network model performs semantic recognition and outputs multiple information. Frame candidate feature images, each layer of the neural network outputs a frame of candidate feature images; performing grayscale processing on the candidate feature images of each frame to obtain a grayscale image; determining parameter values of the grayscale images of each frame, and The gray image with the highest value is used as the semantic feature image of the sample image corresponding to the target sub-image, and the parameter value is determined according to the sharpness of the gray image and / or the amount of information provided by the gray image .

In this technical solution, a candidate image with a higher definition and / or a larger amount of information is selected as a semantic feature image from the multi-frame candidate feature images to improve the quality of the semantic feature images, and further, to improve the super-scoring network model Performance for processing high-resolution images.

Optionally, acquiring the type of the target image; determining a super-scoring network model matching the type of the target image; inputting the target image into a super-scoring network model matching the type of the target image for processing, Get high-resolution images.

In this technical solution, in order to improve the efficiency of processing images and the accuracy of processing images, a super-scoring network model that matches the type of the target image can be selected to process the target image.

In a second aspect, an embodiment of the present invention provides an image processing apparatus having a function of realizing the behavior in the implementation manner of the first aspect. This function can be realized by hardware, and can also be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules may be software and / or hardware. Based on the same inventive concept, since the principle and beneficial effects of the image processing apparatus for solving problems can be referred to the method implementation of the first aspect and the beneficial effects brought about by it, the implementation of the image processing apparatus can be referred to the method for the first aspect. Implementation manners, duplicates are not repeated.

According to a third aspect, an embodiment of the present invention provides an electronic device. The electronic device includes: a memory configured to store one or more programs; and a processor configured to call a program stored in the memory to implement the first aspect described above. For the solution in the method design of the method, the implementation manner and beneficial effects of the forwarding plane device for solving the problem, refer to the implementation manner and beneficial effects of the method of the first aspect described above, and repeated descriptions will not be repeated.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to explain the technical solutions in the embodiments of the present invention more clearly, the accompanying drawings used in the embodiments of the present invention will be described below.

FIG. 1 is a schematic flowchart of an image processing method according to an embodiment of the present invention;

2 is a schematic flowchart of an image processing method according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of another super-segment network model and an image semantic network model according to an embodiment of the present invention.

4 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

detailed description

The following describes the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention.

The picture processing device according to the embodiment of the present invention may be set in any electronic device and used for performing a high-resolution picture conversion operation on a picture. The electronic device includes, but is not limited to, smart mobile devices (such as mobile phones, PDAs, media players, etc.), wearable devices, headsets, personal computers, server computers, handheld or laptop devices, and so on.

The image processing method and related equipment provided in this application are further described below.

Please refer to FIG. 1. FIG. 1 is a schematic flowchart of an image processing method according to an embodiment of the present invention. The method may be executed by an image processing apparatus. The specific explanation of the image processing apparatus is as described above. As shown in FIG. 1, the image processing method may include the following steps.

S101. Obtain a target image that needs to be subjected to super-resolution processing.

In the embodiment of the present invention, the image processing apparatus may obtain a target image requiring super-resolution processing from a local database, or download a target image requiring super-resolution processing from a network. The target image refers to an image with a resolution lower than a preset resolution value, and the target image may refer to a captured image or any frame image in a captured video.

S102. Input the target image into a super-scoring network model for processing to obtain a high-resolution image.

The network parameters of the super-segmented network model are adjusted according to the multi-frame sample images and the semantic feature maps corresponding to the sample images in each frame, and the semantic feature maps are obtained through the semantic recognition of the image semantic network model.

In the embodiment of the present invention, since the network parameters of the super-segmented network model are adjusted according to the multi-frame sample images and the semantic feature images of each frame sample images, the semantic feature images include detailed feature information and edge structure information of the sample images. Therefore, the image processing device can input the target image into a super-scoring network model for processing to obtain a high-resolution image to improve the quality of the high-resolution image. The high-resolution image may refer to an image with a resolution greater than a preset resolution value. The high-resolution image may provide users with more detailed feature information and edge structure information.

Among them, the super-segmentation network model and the image semantic network model can be constituted by a convolutional neural network. In a convolutional neural network, there are usually multiple convolutional layers, and each convolutional layer includes multiple convolutional kernels. It is three-dimensional and contains data in three dimensions of C, H, and W. C, H, and W represent the depth, height, and width of the data, respectively. A convolution kernel is essentially a combination of a series of weights. By adjusting the weight of the convolution kernel in the super-segmented network model, the image conversion error of the super-segmented network model can be reduced. error.

Among them, the network parameter of the super-segment network model refers to the weight of the convolution kernel in the super-segment network model.

In one embodiment, in order to improve the efficiency of obtaining a high-resolution image, the image processing device may preprocess the target image, and input the pre-processed target image to a super-scoring network model for processing to obtain a high-resolution image. For example, preprocessing includes cropping the target image to extract areas that are of interest to the target image, such as cropping out the face of a person; or preprocessing includes scaling the target image to obtain a suitable image. The size of the super network model.

In one embodiment, the image processing apparatus may obtain the type of the target image, determine a super-scoring network model that matches the type of the target image, and input the target image to a super-scoring network model that matches the type of the target image for processing To get high-resolution images.

In order to improve the efficiency and accuracy of obtaining high-resolution images, the image processing device can obtain the type of the target image and classify it according to the content included in the target image. The type of the target image includes the person image type and the scene image type. Or animal image types, which are classified according to the state of the target image, and the type of the target image includes a static image type or a dynamic image type. According to the relationship between the image type and the super-scoring network model, a super-scoring network model matching the type of the target image is determined, and the target image is input to the super-scoring network model matching the type of the target image for processing to obtain a high-resolution image. For example, the target image is a person image type, and a super network model matching the person image type is obtained, and the target image is input into the matched super network model for processing to obtain a high-resolution image. The network parameters of the matched super-segmentation network model are adjusted through multiple frames including a person sample image and a semantic feature image corresponding to each frame of the person sample image.

In one embodiment, the image processing device may train different types of super-scoring network models according to different types of sample images and semantic feature images corresponding to the sample images, for example, using multiple frames of sample images including animals and each frame of sample images Corresponding semantic feature images are super-network models suitable for processing images including animals.

It can be seen that, by implementing the method described in FIG. 1, since the network parameters of the super-segmented network model are adjusted based on a large number of sample images and the semantic feature images of each frame of sample images, the semantic feature images contain detailed feature information of the sample images. And edge structure information. Therefore, the high-resolution image obtained through the super-scoring network model can provide more detailed feature information and provide high-resolution edge structure information, which improves the quality of high-resolution images.

Please refer to FIG. 2. FIG. 2 is a schematic flowchart of an image processing method according to an embodiment of the present invention. The method may be executed by an image processing apparatus. The specific explanation of the image processing apparatus is as described above. The difference between the embodiment of the present invention and the embodiment described in FIG. 1 is that the embodiment of the present invention calculates the error of the super-scoring network model by using multiple frame sample images and the semantic feature images of each frame sample image. When the error is greater than a preset error value , Adjusting the network parameters of the super-scoring network model to obtain a super-scoring network model with an error less than or equal to a preset error value. An embodiment of the present invention is shown in FIG. 2. The image processing method may include the following steps.

S201. Determine an error of the super-scoring network model according to the multi-frame sample image and the semantic feature map corresponding to each sample image.

In the embodiment of the present invention, the image processing apparatus may determine the error of the super-scoring network model according to the multi-frame sample image and the semantic feature map corresponding to each sample image. In one embodiment, step S201 includes steps S11 to S15. .

S11. Acquire a high-resolution sub-image and a low-resolution sub-image corresponding to each frame of the multi-frame sample images.

S12. Input the target sub-image of each frame into the image semantic network model and perform semantic recognition to obtain the semantic feature image corresponding to the sample image of each frame. The target sub-image is a high-resolution corresponding to any sample image in the multi-frame sample image. Rate sub-image or low-resolution sub-image.

S13. The low-resolution sub-images of each frame are input into the super-scoring network model and processed to obtain high-resolution feature images of the sample images of each frame.

S14. Superimpose the high-resolution sub-image of the sample image and the semantic feature image of the corresponding sample image to obtain a superimposed image.

S15. Determine the degree of difference between the high-resolution feature image of the sample image and the superimposed image of the corresponding sample image for each frame; calculate the sum of the degree of difference, and use the sum of the degree of difference as the error of the super-scoring network model.

In steps S11 to S15, the image processing device may perform sampling processing on each frame of the multi-frame sample image to obtain a low-resolution sub-image corresponding to each frame of the sample image, and perform enhancement processing on each frame of the sample image to obtain each frame. The high-resolution sub-image corresponding to the sample image. The low-resolution sub-images of each frame are input to the super-scoring network model for processing to obtain high-resolution feature images of the sample images of each frame, and the target sub-images of each frame are input to the image semantic network model for semantic recognition to obtain each frame. The semantic feature image corresponding to the sample image. The semantic feature image includes detailed feature information and edge structure information of the sample image.

Further, the superimposed high-resolution sub-image of the sample image and the semantic feature image of the corresponding sample image are superimposed to obtain a superimposed image. The superimposed image is a semantic-enhanced high-resolution image. The superimposed image and the corresponding sample image of each frame are superimposed. The high-resolution feature images of the two images are compared to obtain the degree of difference between the high-resolution feature image of the sample image and the superimposed image of the corresponding sample image. The greater the degree of difference, the smaller the similarity between the high-resolution feature image obtained by the super-segment network model and the superimposed image (that is, the semantic-enhanced high-resolution image), that is, the high-resolution feature is obtained by the super-segment network model. The quality of the image is poor; on the contrary, the smaller the difference, it indicates that the similarity between the high-resolution feature image obtained by the super-segmented network model and the superimposed image (that is, the high-resolution image with enhanced semantics) is greater, that is, The quality of the high-resolution feature images obtained by the sub-network model is better. Therefore, the sum of differences can be calculated, and the sum of differences can be used as the error of the super network model. The error of the super network model refers to the error that the super network model converts the image into a high-resolution image. The quality of the high-resolution images processed by the sub-network model is poor; on the contrary, the smaller the error, it indicates that the quality of the high-resolution images processed by the super-network model is better.

For example, as shown in Fig. 2, suppose that the super-segmentation network model is composed of two consecutive convolutional layers, and each convolutional layer includes N k * k convolution kernels, where N can be any of [20 100] Integer, k can be 3 or 5. The image processing device can obtain high-resolution sub-images and low-resolution sub-images of each frame of sample images in the N-frame sample images, and input the low-resolution sub-images of each frame of sample images into the super-scoring network model for processing. The high-resolution feature image corresponding to the frame sample image is extracted with feature information of each frame of the high-resolution feature image, and is identified as f _W (x _j ), where x _j represents the j-th sample image. The target sub-image is input into the super-scoring network and S operation is performed to obtain a semantic feature image. The semantic feature image is superimposed with a high-resolution sub-image to obtain a superimposed image. The feature information of the superimposed image is extracted and identified as f _s (y _j ) + z _j , y _{j is} the target image corresponding to the j-th sample image, f _s (y _j ) represents the feature information of the semantic feature image of the target image corresponding to the j-th sample image, and z _j is the height of the j-frame sample image. Feature information of the resolution sub-image. The feature information of the high-resolution feature image of each frame is compared with the feature information of the corresponding superimposed image to determine the degree of difference between the high-resolution feature image of the sample image and the superimposed image of the corresponding sample image; The difference sum is described, and the difference sum is used as an error of the super-scoring network model, and identified as W. The error of the super-segmentation network can be expressed by equation (1).

Among them, the formula (1) MSE (f _W (x _j ), fs (y _j ) + z _j ) represents the feature information of the superimposed image of the j-th sample image and the feature of the high-resolution feature image of the j-th sample image. The degree of difference in information.

In one embodiment, the image processing apparatus may set weights for the images output by the image semantic network model, and process the semantic feature images of the sample images in each frame according to the weights to obtain processed semantic feature maps. The high-resolution sub-image of the sample image and the processed semantic feature image corresponding to the sample image are superimposed to obtain a superimposed image.

The image processing device can set weights for the image output by the image semantic network model according to the scene or according to the needs of the user, and process the semantic feature images corresponding to each frame sample image according to the weights to obtain the processed semantic feature images. The super-resolution sub-image of the image is superimposed with the processed semantic feature image of the corresponding sample image to obtain a superimposed image. Among them, the larger the weight value, the more information the semantic feature image provides in the superimposed image, the higher the definition of the superimposed image, and further, the high-resolution image output by the super-network model is closer to the semantic feature image; otherwise, the weight The smaller the value, the less the information provided by the semantic feature image in the superimposed image, the lower the clarity of the superimposed image, and the closer the high-resolution image output by the super-scoring network model is to the target sub-image.

For example, suppose that the weight set for the image output by the semantic network model is λ. According to the weight λ, the semantic feature image corresponding to each frame of the sample image is processed to obtain the processed semantic feature image, and the height of each frame of the sample image is high. The resolution sub-image is superimposed with the processed semantic feature image of the corresponding sample image to obtain a superimposed image, and the feature information of the superimposed image is extracted. It can be identified as λf _s (y _j ) + z _j , and λf _s (y _j ) is the processed image. The feature information of the semantic feature image, and further, the error of the super-scoring network can be expressed by equation (2).

In one embodiment, step S12 includes: inputting a target sub-image into the image semantic network model, and performing multi-layer neural network recognition through a multi-layer neural network included in the image semantic network model to output multiple frames of candidate feature images, and each layer of the neural network output One frame of candidate feature image, performing grayscale processing on each frame of the candidate feature image to obtain a grayscale image, determining a parameter value of each grayscale image, and using the grayscale image with the largest parameter value as the corresponding target sub-image For the semantic feature image of the sample image, the parameter value is determined according to the sharpness of the grayscale image and / or the amount of information provided by the grayscale image.

In order to output a high-quality semantic feature image, the image processing apparatus may input a target sub-image into the image semantic network model, and perform multi-layer neural network included in the image semantic network model to perform semantic recognition and output multiple frames of candidate feature images. Perform grayscale processing on the candidate feature image of each frame to obtain a grayscale image, determine the parameter values of the grayscale image of each frame, and use the grayscale image with the largest parameter value as the semantic feature image of the sample image corresponding to the target sub-image That is, a grayscale image with clear edge structure and capable of providing rich detailed feature information is used as a semantic feature image, so that the network parameters of the super-separated network model can be trained through higher-quality semantic feature images to obtain high-quality high-quality Super-resolution network model for high-resolution images.

S202. Determine whether the error is less than or equal to a preset error value. The image processing device can determine whether the error is less than or equal to the preset error value. When the error is less than or equal to the preset error value, it indicates that the super-scoring network model can output a high-quality high-resolution image, and step S204 can be performed; otherwise, when the error If it is greater than the preset error value, it indicates that the super-scoring network model cannot output a high-quality high-resolution image, and step S205 may be performed.

S203. Adjust network parameters of the super-segment network model.

When the error is greater than the preset error value, adjust the network parameters of the super network model and repeat S201. The error of the execution of the super network model is less than or equal to the preset error value, so that the super network model can output high-quality High resolution image.

S204. Acquire a target image that needs to be subjected to super-resolution processing.

When the error of the super-segmentation network model is smaller than the preset error, it indicates that the super-segmentation network model can output a high-quality high-resolution image, and the image processing device can obtain a target image that needs to be super-resolution-processed.

S205. The target image is input to a super-scoring network model for processing to obtain a high-resolution image.

The target image is input to the super-scoring network model for processing to obtain a high-resolution image, so that more detailed feature information and higher-definition edge feature information can be obtained from the high-resolution image.

It can be seen that, by implementing the method described in FIG. 2, since the network parameters of the super-segmented network model are adjusted according to a large number of sample images and the semantic feature images of each frame of the sample images, the semantic feature images contain detailed feature information of the sample images And edge structure information. Therefore, the high-resolution image obtained through the super-scoring network model can provide more detailed feature information and provide high-resolution edge structure information, which improves the quality of high-resolution images.

Refer to FIG. 4, which is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus described in this embodiment includes:

The obtaining module 401 is configured to obtain a target image that needs to be subjected to super-resolution processing.

A processing module 402, configured to input the target image into a super-scoring network model for processing to obtain a high-resolution image;

Wherein, the network parameters of the super-segmented network model are adjusted according to the multi-frame sample images and the semantic feature maps corresponding to the sample images in each frame, and the semantic feature maps are obtained through semantic recognition of the image semantic network model. .

A determining module 403 is configured to determine an error of the super-scoring network model according to the multi-frame sample images and a semantic feature map corresponding to each of the sample images.

An adjustment module 404 is configured to adjust network parameters of the ultra-scoring network model when the error is greater than a preset error value.

The determining module 403 is specifically configured to obtain a high-resolution sub-image and a low-resolution sub-image corresponding to each frame of the multi-frame sample images; input each frame of the target sub-image into the image semantic network model for semantics The semantic feature image corresponding to each of the sample images is identified, and the target sub-image is a high-resolution sub-image or a low-resolution sub-image corresponding to any one of the multi-frame sample images; The low-resolution sub-images are input into the super-scoring network model and processed to obtain high-resolution feature images of the sample image in each frame; the high-resolution sub-images of the sample image in each frame and the semantic features of the corresponding sample image Superimpose the images to obtain a superimposed image; determine the difference between the high-resolution feature image of each sample image and the superimposed image of the corresponding sample image; calculate the sum of the differences, and use the sum of the differences as the error.

A setting module 405 is configured to set a weight for an image output by the image semantic network model.

The processing module 402 is further configured to process the semantic feature image of the sample image of each frame according to the weights to obtain a processed semantic feature map.

The determining module 403 is specifically configured to superimpose the high-resolution sub-image of the sample image of each frame and the processed semantic feature image of the corresponding sample image to obtain a superimposed image.

The determining module 403 is specifically configured to input the target sub-image into the image semantic network model, and perform multi-layer neural network included in the image semantic network model to perform semantic recognition and output multiple frame candidate feature images, each layer The neural network outputs a frame of candidate feature images; performing grayscale processing on each frame of the candidate feature images to obtain a grayscale image; determining a parameter value of the grayscale image of each frame, and using the grayscale image with the largest parameter value as the grayscale image The parameter value of the semantic feature of the sample image corresponding to the target sub-image is determined according to the sharpness of the gray image and / or the amount of information provided by the gray image.

The obtaining module 401 is further configured to obtain a type of the target image; and determine a super-scoring network model that matches the type of the target image.

The processing module 402 is configured to input the target image into a super-scoring network model matching the type of the target image for processing to obtain a high-resolution image.

It can be seen that, by implementing the device described in FIG. 4, since the network parameters of the super-segmented network model are adjusted based on a large number of sample images and the semantic feature images of each frame of the sample images, the semantic feature images contain detailed feature information of the sample images. And edge structure information. Therefore, the high-resolution image obtained through the super-scoring network model can provide more detailed feature information and provide high-resolution edge structure information, which improves the quality of high-resolution images.

Refer to FIG. 5, which is a schematic structural diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a processor 501, a memory 502, a communication interface 503, and a power source 504. The processor 501, the memory 502, the communication interface 503, and the power source 504 are connected to each other through a bus.

The processor 501 may be one or more CPUs. In the case where the processor 501 is a CPU, the CPU may be a single-core CPU or a multi-core CPU. The processor 501 and the processor 501 may include a modem for The signal received by the transceiver 805 is subjected to modulation or demodulation processing.

The memory 502 includes, but is not limited to, RAM, ROM), EPROM, and CD-ROM. The memory 502 is used to store instructions, an operating system, various applications, and data.

The communication interface 503 is connected to a forwarding plane device or other control plane devices. For example, the communication interface 503 includes multiple interfaces, which are respectively connected to multiple terminals or connected to a forwarding plane device. The communication interface 503 may be a wired interface, a wireless interface, or a combination thereof. The wired interface may be, for example, an Ethernet interface. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. The wireless interface may be, for example, a wireless local area network (English: wireless local area network, abbreviation: WLAN) interface, a cellular network interface, or a combination thereof.

The power supply 504 is configured to supply power to a control plane device.

The memory 502 is also used to store program instructions. The processor 501 may call the program instructions stored in the memory 502 to implement the image processing method as shown in the foregoing embodiments of the present application.

Based on the same inventive concept, the principle of solving the problem of the control plane device provided in the embodiment of the present invention is similar to that of the method embodiment of the present invention. Therefore, the implementation and beneficial effects of the control plane device can be referred to as well as the beneficial effects. More details.

The present invention also provides a computer-readable storage medium on which a computer program is stored. For implementation manners and beneficial effects of the program, refer to the foregoing implementation manners and beneficial effects of the image processing method in FIG. 1 and FIG. The details are not repeated here.

The implementation of the present invention also provides a computer program product. The computer program product includes a non-volatile computer-readable storage medium storing a computer program. When the computer program is executed, the computer executes the corresponding embodiments of FIG. 1 and FIG. 2 described above. For the steps of the image processing method in FIG. 1, and the implementation and beneficial effects of the computer program product for solving the problem, refer to the implementation and beneficial effects of the image processing method in FIG. 1 and FIG. 2 described above.

A person of ordinary skill in the art can understand that all or part of the processes in the method of the foregoing embodiment can be implemented by using a computer program to instruct related hardware. The above program can be stored in a computer-readable storage medium. When executed, the processes of the embodiments of the methods described above may be included.

Claims

An image processing method, wherein the method includes:

Obtain a target image that needs to be super-resolution processed;

Inputting the target image into a super-scoring network model for processing to obtain a high-resolution image;

Wherein, the network parameters of the super-segmented network model are adjusted according to the multi-frame sample images and the semantic feature maps corresponding to the sample images in each frame, and the semantic feature maps are obtained through semantic recognition of the image semantic network model. .
The method according to claim 1, further comprising:

Determining an error of the super-scoring network model according to the multi-frame sample image and the semantic feature map corresponding to each sample image;

When the error is greater than a preset error value, network parameters of the ultra-scoring network model are adjusted.
The method according to claim 2, wherein determining the error of the super-scoring network model according to the multi-frame sample image and the semantic feature map corresponding to each frame of the sample image comprises:

Obtaining a high-resolution sub-image and a low-resolution sub-image corresponding to each frame of the multi-frame sample images;

Input the target sub-image of each frame into the image semantic network model and perform semantic recognition to obtain the semantic feature image corresponding to the sample image of each frame, and the target sub-image is corresponding to any sample image in the multi-frame sample image High-resolution sub-images or low-resolution sub-images;

Inputting the low-resolution sub-image of each frame into the super-scoring network model for processing to obtain a high-resolution feature image of the sample image of each frame;

Superimposing the high-resolution sub-image of the sample image of each frame with the semantic feature image of the corresponding sample image to obtain a superimposed image;

Determining the degree of difference between the high-resolution feature image of the sample image and the superimposed image of the corresponding sample image in each frame;

Calculate the sum of the differences and use the sum of the differences as the error of the super-score network model.
The method according to claim 3, further comprising:

Setting weights for images output by the image semantic network model;

Processing the semantic feature image of the sample image of each frame according to the weights to obtain a processed semantic feature map;

The superimposing the high-resolution sub-image of each sample image and the semantic feature image of the corresponding sample image to obtain a superimposed image includes:

A superimposed high-resolution sub-image of the sample image and the processed semantic feature image of the corresponding sample image are superimposed to obtain a superimposed image.
The method according to claim 3, wherein the image semantic network model comprises a multilayer neural network, and the target sub-image of each frame is input into the image semantic network model for semantic recognition to obtain a sample image of each frame Corresponding semantic feature images, including:

Inputting the target sub-image into the image semantic network model, and performing multi-layer neural network recognition through the image semantic network model to output multiple frames of candidate feature images, and each layer of the neural network outputs one frame of candidate features image;

Performing grayscale processing on the candidate feature image of each frame to obtain a grayscale image;

Determine the parameter value of the grayscale image for each frame, and use the grayscale image with the largest parameter value as the semantic feature image of the sample image corresponding to the target sub-image, and the parameter value is based on the sharpness and And / or determined by the amount of information provided by the grayscale image.
The method according to any one of claims 1-5, further comprising:

Acquiring the type of the target image;

Determining a super-scoring network model that matches the type of the target image;

The inputting the target image into a super-scoring network model for processing to obtain a high-resolution image includes:

The target image is input to a super-scoring network model that matches the type of the target image for processing to obtain a high-resolution image.
An image processing device, wherein the device includes:

An acquisition module for acquiring a target image that needs to be subjected to super-resolution processing;

A processing module, configured to input the target image into a super-scoring network model for processing to obtain a high-resolution image;

Wherein, the network parameters of the super-segmented network model are adjusted according to the multi-frame sample images and the semantic feature maps corresponding to the sample images in each frame, and the semantic feature maps are obtained through semantic recognition of the image semantic network model. .
The apparatus according to claim 7, further comprising:

The device further includes:

A determining module, configured to determine an error of the ultra-scoring network model according to the multi-frame sample image and the semantic feature map corresponding to each sample image;

An adjustment module is configured to adjust network parameters of the ultra-scoring network model when the error is greater than a preset error value.
The device according to claim 8, characterized in that:

The determining module is specifically configured to obtain a high-resolution sub-image and a low-resolution sub-image corresponding to each frame of the multi-frame sample images; input each frame of the target sub-image into the image semantic network model for semantic recognition Obtain the semantic feature image corresponding to each sample image, and the target sub-image is a high-resolution sub-image or a low-resolution sub-image corresponding to any of the sample images in the multi-frame sample image; The resolution sub-image is input to the super-scoring network model and processed to obtain a high-resolution feature image of the sample image of each frame; the high-resolution sub-image of the sample image of each frame and the semantic feature image of the corresponding sample image are processed Superimpose to obtain a superimposed image; determine the degree of difference between the high-resolution feature image of the sample image and the superimposed image of the corresponding sample image for each frame; calculate the sum of the degree of difference, and use the sum of the degree of difference as the error of the super score network model .
The apparatus according to claim 9, further comprising:

A setting module, configured to set a weight for an image output by the image semantic network model;

The processing module is further configured to process the semantic feature image of the sample image of each frame according to the weights to obtain a processed semantic feature map;

The determining module is specifically configured to superimpose the high-resolution sub-image of the sample image of each frame and the processed semantic feature image of the corresponding sample image to obtain a superimposed image.
The device according to claim 9, wherein:

The determining module is specifically configured to input the target sub-image into the image semantic network model, and perform multi-layer neural network included in the image semantic network model to perform semantic recognition and output multiple frames of candidate feature images. The neural network outputs a frame of candidate feature images; performs grayscale processing on each of the candidate feature images to obtain a grayscale image; determines the parameter value of the grayscale image of each frame, and uses the grayscale image with the largest parameter value as all The semantic feature image of the sample image corresponding to the target sub-image, and the parameter value is determined according to the sharpness of the gray image and / or the amount of information provided by the gray image.
The device according to any one of claims 7-11, wherein the device further comprises:

The acquisition module is further configured to acquire the type of the target image; determine a super-scoring network model that matches the type of the target image;

The processing module is configured to input the target image into a super-scoring network model matching the type of the target image for processing to obtain a high-resolution image.
An electronic device includes at least one processor, a memory, and instructions stored on the memory and executed by the at least one processor, wherein the at least one processor executes the instruction, To implement the steps of the image processing method according to any one of claims 1 to 6.
A computer-readable storage medium, characterized in that the computer storage medium stores a computer program, wherein the computer program includes program instructions, and the program instructions, when executed by a processor, cause the processor to execute the program according to claim 1 The steps of the image processing method according to any one of 6 to 6.
A computer program product, characterized in that the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program executes operations to cause a computer to implement The steps of the image processing method described above.