CN112088393A - Image processing method, device and equipment - Google Patents

Image processing method, device and equipment Download PDF

Info

Publication number
CN112088393A
CN112088393A CN201880093293.9A CN201880093293A CN112088393A CN 112088393 A CN112088393 A CN 112088393A CN 201880093293 A CN201880093293 A CN 201880093293A CN 112088393 A CN112088393 A CN 112088393A
Authority
CN
China
Prior art keywords
image
resolution
frame
network model
sample
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201880093293.9A
Other languages
Chinese (zh)
Other versions
CN112088393B (en
Inventor
谭文伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Publication of CN112088393A publication Critical patent/CN112088393A/en
Application granted granted Critical
Publication of CN112088393B publication Critical patent/CN112088393B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

The embodiment of the invention discloses an image processing method, device and equipment, wherein the method comprises the following steps: acquiring a target image needing super-resolution processing; inputting the target image into a hyper-resolution network model for processing to obtain a high-resolution image; the network parameters of the hyper-division network model are obtained by adjusting according to a plurality of frames of sample images and semantic feature maps corresponding to the sample images of each frame, and the semantic feature maps are obtained by performing semantic recognition through an image semantic network model, so that the quality of the obtained high-resolution images is improved.

Description

Image processing method, device and equipment Technical Field
The present invention relates to image processing technologies, and in particular, to an image processing method, an image processing apparatus, and an image processing device.
Background
With the development of multimedia technology, users have higher and higher requirements for multimedia information, for example, high-resolution multimedia information (picture information, video information, etc.) becomes a mainstream multimedia file.
When the terminal needs to perform high-resolution multimedia information interaction, the terminal needs a high-speed broadband to transmit high-resolution multimedia information, so that the information interaction cost of both sides of the interactive terminal can be greatly improved. Therefore, the user usually converts the high-resolution multimedia information into the low-resolution multimedia information, and then sends the low-resolution multimedia information to other terminals, so that the interaction cost is reduced. After receiving the low-resolution multimedia information, the receiving terminal needs to restore the low-resolution multimedia information to the high-resolution multimedia information to acquire more detailed information.
Disclosure of Invention
The invention provides an image processing method, device and equipment, which can improve the precision of converting a low-resolution image into a high-resolution image so as to improve the quality of the high-resolution image.
In a first aspect, an embodiment of the present invention provides an image packet processing method, where the method includes: acquiring a target image needing super-resolution processing; inputting the target image into a hyper-resolution network model for processing to obtain a high-resolution image; the network parameters of the hyper-division network model are obtained by adjusting according to a plurality of frames of sample images and semantic feature maps corresponding to the sample images of each frame, and the semantic feature maps are obtained by performing semantic recognition through an image semantic network model.
In the technical scheme, because the network parameters of the super-resolution network model are obtained by adjusting according to a large number of sample images and semantic feature images of each frame of sample image, and the semantic feature images contain detail feature information and edge structure information of the sample images, the super-resolution network model is a semantically enhanced network model, namely, a low-resolution image can be converted into a semantically enhanced high-resolution image through the semantically enhanced super-resolution network model, the obtained semantically enhanced high-resolution image can provide more detail feature information and high-definition edge structure information, and the quality of the high-resolution image is improved.
Optionally, determining an error of the hyper-division network model according to the multiple frames of sample images and the semantic feature map corresponding to each frame of sample image; and when the error is larger than a preset error value, adjusting the network parameters of the hyper-division network model.
In the embodiment of the invention, in order to improve the precision of processing images by the hyper-division network, network parameters of the hyper-division network model can be adjusted according to the multiple frames of sample images and the semantic feature maps corresponding to the sample images.
Optionally, a high-resolution sub-image and a low-resolution sub-image corresponding to each frame of sample image in a plurality of frames of sample images are obtained, each frame of target sub-image is input into the image semantic network model for semantic recognition to obtain a semantic feature image corresponding to each frame of sample image, and the target sub-image is the high-resolution sub-image or the low-resolution sub-image corresponding to any one sample image in the plurality of frames of sample images; inputting each frame of low-resolution sub-image into the hyper-division network model for processing to obtain a high-resolution characteristic image of each frame of sample image; superposing the high-resolution sub-image of each frame of sample image and the semantic feature image of the corresponding sample image to obtain a superposed image; determining the difference degree between the high-resolution characteristic image of each frame of sample image and the superposed image of the corresponding sample image; and calculating the sum of the difference degrees, and taking the sum of the difference degrees as the error of the hyper-division network model.
In the technical scheme, a high-resolution sub-image of a sample image and a superposed image of semantic feature images of the sample image are used as reference images, a low-resolution sub-image of the sample image is used as a training sample, and the error of a hyper-division network model is calculated according to the reference images and the training sample image, so that the hyper-division network model with low error can be obtained conveniently.
Optionally, setting a weight for an image output by the image semantic network model; processing the semantic feature image of each frame of sample image according to the weight to obtain a processed semantic feature image; and superposing the high-resolution sub-image of each frame of sample image and the processed semantic feature image of the corresponding sample image to obtain a superposed image.
In the technical scheme, the image processing device can set weight for the image output by the image semantic network model so as to obtain the beyond-divided network model incapable of performance, so as to meet different image requirements of users, namely, the larger the weight value is, the more information provided by the semantic feature image in the superposed image is, the higher the definition of the superposed image is, and further, the high-resolution image output by the beyond-divided network model is closer to the semantic feature image; on the contrary, the smaller the weight value is, the less the information provided by the semantic feature image in the superimposed image is, and the definition of the superimposed image is lower, so that the high-resolution image output by the hyper-division network model is closer to the target sub-image.
Optionally, the image semantic network model includes a multilayer neural network, the target sub-images are input into the image semantic network model, a plurality of candidate feature images are output through semantic recognition performed by the multilayer neural network included in the image semantic network model, and one candidate feature image is output by each layer of the neural network; carrying out gray level processing on each frame of candidate characteristic image to obtain a gray level image; and determining the parameter value of each frame of the gray scale image, and taking the gray scale image with the maximum parameter value as the semantic feature image of the sample image corresponding to the target sub-image, wherein the parameter value is determined according to the definition of the gray scale image and/or the information quantity provided by the gray scale image.
In the technical scheme, the candidate images with higher definition and/or more provided information amount are selected from the plurality of candidate feature images as the semantic feature images, so that the quality of the semantic feature images is improved, and further, the performance of the super-resolution network model for processing the high-resolution images is improved.
Optionally, the type of the target image is obtained; determining a hyper-divided network model matched with the type of the target image; and inputting the target image into a hyper-resolution network model matched with the type of the target image for processing to obtain a high-resolution image.
In the technical scheme, in order to improve the efficiency of processing the image and the accuracy of processing the image, a hyper-resolution network model matched with the type of the target image can be selected to process the target image.
In a second aspect, an embodiment of the present invention provides an image processing apparatus having a function of implementing the behaviors in the implementation manner of the first aspect. The function can be realized by hardware, and can also be realized by executing corresponding software by hardware. The hardware or software includes one or more modules corresponding to the above functions, and the modules may be software and/or hardware. Based on the same inventive concept, as the principle and the advantageous effects of the image processing apparatus for solving the problems can be referred to the method implementation manner of the first aspect and the advantageous effects brought thereby, the implementation of the image processing apparatus can be referred to the method implementation manner of the first aspect, and repeated details are not repeated.
In a third aspect, an embodiment of the present invention provides an electronic device, including: a memory for storing one or more programs; the processor is configured to call a program stored in the memory to implement the scheme in the method design of the first aspect, and for the implementation and the beneficial effects of the forwarding plane device for solving the problem, reference may be made to the implementation and the beneficial effects of the method of the first aspect, and repeated details are not described here.
Drawings
In order to more clearly illustrate the technical solution in the embodiment of the present invention, the drawings required to be used in the embodiment of the present invention will be described below.
Fig. 1 is a schematic flowchart of an image processing method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating an image processing method according to an embodiment of the present invention;
fig. 3 is a schematic diagram of another hyper-separation network model and an image semantic network model according to an embodiment of the present invention.
FIG. 4 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The embodiments of the present invention will be described below with reference to the drawings.
The image processing device of the embodiment of the invention can be arranged in any electronic equipment and is used for carrying out high-resolution image conversion operation on the image. The electronic devices include, but are not limited to, smart mobile devices (e.g., mobile phones, palmtop computers, media players, etc.), wearable devices, head-mounted devices, personal computers, server computers, hand-held or laptop devices, and the like.
The following further describes the image processing method and related apparatus provided in the present application.
Referring to fig. 1, fig. 1 is a flowchart illustrating an image processing method according to an embodiment of the present invention, where the method can be executed by an image processing apparatus, and a detailed explanation of the image processing apparatus is as described above. As shown in fig. 1, the image processing method may include the following steps.
S101, acquiring a target image needing super-resolution processing.
In the embodiment of the present invention, the image processing apparatus may acquire the target image that needs to be super-resolution processed from a local database, or download the target image that needs to be super-resolution processed from a network. The target image refers to an image with a resolution lower than a preset resolution value, and the target image may refer to a shot image or any frame image in a shot video.
And S102, inputting the target image into a hyper-resolution network model for processing to obtain a high-resolution image.
The network parameters of the hyper-division network model are obtained by adjusting according to a plurality of frames of sample images and semantic feature maps corresponding to the sample images of each frame, and the semantic feature maps are obtained by performing semantic recognition through an image semantic network model.
In the embodiment of the invention, because the network parameters of the hyper-resolution network model are obtained by adjusting according to the multiple frames of sample images and the semantic feature images of the multiple frames of sample images, and the semantic feature images comprise the detail feature information and the edge structure information of the sample images, the image processing device can input the target image into the hyper-resolution network model for processing to obtain the high-resolution image so as to improve the quality of the high-resolution image. The high-resolution image may be an image with a resolution greater than a preset resolution value, and the high-resolution image may provide more detail feature information and edge structure information for a user.
The hyper-division network model and the image semantic network model can be formed by a convolutional neural network, in the convolutional neural network, a plurality of convolutional layers are usually arranged, each convolutional layer comprises a plurality of convolutional kernels, the convolutional kernels are three-dimensional and comprise C, H and W three-dimensional data, and C, H and W respectively represent the depth, height and width of the data. The convolution kernel is essentially a combination of a series of weights. The image conversion error of the hyper-division network model can be reduced by adjusting the weight of the convolution kernel in the hyper-division network model, and the semantic recognition error of the image semantic network model can be reduced by adjusting the weight of the convolution kernel in the image semantic network model.
The network parameters of the hyper-division network model refer to the weights of convolution kernels in the hyper-division network model.
In one embodiment, to improve the efficiency of acquiring a high-resolution image, the image processing apparatus may perform preprocessing on the target image, and input the preprocessed target image into the hyper-resolution network model for processing, so as to obtain the high-resolution image. For example, the preprocessing includes cropping the target image to extract a region in which the user is interested in the target image, e.g., a region in which a human face is cropped; alternatively, the pre-processing includes scaling the target image to obtain a size suitable for the hyper-resolution network model processing.
In one embodiment, the image processing device may acquire a type of a target image, determine a hyper-segmentation network model matching the type of the target image, and input the target image into the hyper-segmentation network model matching the type of the target image for processing, so as to obtain a high-resolution image.
In order to improve the efficiency of acquiring a high-resolution image and the accuracy of acquiring a high-resolution image, the image processing apparatus may acquire a type of a target image, classify the target image according to contents included in the target image, the type of the target image including a person image type, a scene image type, or an animal image type, classify the target image according to a state of the target image, and the type of the target image including a still image type or a moving image type. And determining a hyper-resolution network model matched with the type of the target image according to the relationship between the image type and the hyper-resolution network model, and inputting the target image into the hyper-resolution network model matched with the type of the target image for processing to obtain a high-resolution image. For example, the target image is of a character image type, a hyper-division network model matched with the character image type is obtained, and the target image is input into the matched hyper-division network model for processing to obtain a high-resolution image. The network parameters of the matched hyper-division network model are adjusted through a plurality of frames of semantic feature images comprising character sample images and corresponding to each frame of character sample images.
In one embodiment, the image processing apparatus may train different types of hyper-segmentation network models according to different types of sample images and semantic feature images corresponding to the sample images, for example, a plurality of frames of sample images including animals and semantic feature images corresponding to each frame of sample images are used to process the hyper-segmentation network models including the images of the animals.
It can be seen that, by implementing the method described in fig. 1, since the network parameters of the hyper-resolution network model are obtained by adjusting according to a large number of sample images and the semantic feature images of each frame of sample image, and the semantic feature images include the detail feature information and the edge structure information of the sample images, the high-resolution images obtained by the hyper-resolution network model can provide more detail feature information and provide high-definition edge structure information, thereby improving the quality of the high-resolution images.
Referring to fig. 2, fig. 2 is a flowchart illustrating an image processing method according to an embodiment of the present invention, where the method can be executed by an image processing apparatus, and a detailed explanation of the image processing apparatus is as described above. The embodiment of the present invention is different from the embodiment described in fig. 1 in that the embodiment of the present invention calculates an error of the hyper-division network model through the multiple frames of sample images and the semantic feature image of each frame of sample image, and adjusts the network parameter of the hyper-division network model when the error is greater than a preset error value, so as to obtain the hyper-division network model with the error less than or equal to the preset error value. As shown in fig. 2, the image processing method may include the following steps.
S201, determining the error of the hyper-division network model according to the multiple frames of sample images and the semantic feature map corresponding to each frame of sample image.
In an embodiment of the present invention, the image processing apparatus may determine the error of the hyper-resolution network model according to the plurality of frames of sample images and the semantic feature map corresponding to each frame of the sample images, and in an embodiment, the step S201 includes steps S11 to S15.
And S11, acquiring a high-resolution sub-image and a low-resolution sub-image corresponding to each frame of sample image in the multi-frame sample images.
S12, inputting each frame of target sub-image into the image semantic network model for semantic recognition to obtain a semantic feature image corresponding to each frame of sample image, wherein the target sub-image is a high-resolution sub-image or a low-resolution sub-image corresponding to any sample image in the multi-frame sample images.
And S13, inputting each frame of low-resolution sub-image into the hyper-division network model for processing to obtain a high-resolution characteristic image of each frame of sample image.
And S14, superposing the high-resolution sub-image of each frame of sample image and the semantic feature image of the corresponding sample image to obtain a superposed image.
S15, determining the difference between the high-resolution characteristic image of each frame of sample image and the superposed image of the corresponding sample image; and calculating the sum of the difference degrees, and taking the sum of the difference degrees as the error of the hyper-division network model.
In steps S11 to S15, the image processing apparatus may perform sampling processing on each frame of sample image in the multiple frames of sample images to obtain a low-resolution sub-image corresponding to each frame of sample image, and perform enhancement processing on each frame of sample image to obtain a high-resolution sub-image corresponding to each frame of sample image. Inputting each frame of low-resolution sub-image into a hyper-division network model for processing to obtain a high-resolution feature image of each frame of sample image, inputting each frame of target sub-image into an image semantic network model for semantic recognition to obtain a semantic feature image corresponding to each frame of sample image, wherein the semantic feature image comprises detail feature information and edge structure information of the sample image.
Further, superposing the high-resolution sub-image of each frame of sample image and the semantic feature image of the corresponding sample image to obtain a superposed image, wherein the superposed image is a high-resolution image with enhanced semantic meaning, and comparing the superposed image with the high-resolution feature image of each frame of sample image to obtain the difference degree between the high-resolution feature image of each frame of sample image and the superposed image of the corresponding sample image. The larger the difference is, the smaller the similarity between the high-resolution characteristic image obtained by the hyper-resolution network model and the superimposed image (i.e. the high-resolution image with enhanced semantics) is, i.e. the quality of the high-resolution characteristic image obtained by the hyper-resolution network model is poorer; on the contrary, the smaller the difference is, the greater the similarity between the high-resolution feature image obtained by the hyper-resolution network model and the superimposed image (i.e. the semantically enhanced high-resolution image), i.e. the better the quality of the high-resolution feature image obtained by the hyper-resolution network model. Therefore, the sum of the difference degrees can be calculated, the sum of the difference degrees is used as the error of the hyper-division network model, the error of the hyper-division network model refers to the error of the hyper-division network model for converting the image into the high-resolution image, and the larger the error is, the poorer the quality of the high-resolution image obtained by processing the hyper-division network model is; on the contrary, the smaller the error is, the better the quality of the high-resolution image obtained by the super-resolution network model processing is.
For example, as shown in FIG. 2, assume that the hyper-divided network model consists of two successive convolutional layers, each convolutional layerThe layer includes N convolution kernels of k x k, where N may be [ 20100 ]]K may be 3 or 5. The image processing device can acquire a high-resolution sub-image and a low-resolution sub-image of each frame of sample image in N frames of sample images, input the low-resolution sub-image of each frame of sample image into the hyper-division network model for processing to obtain a high-resolution feature image corresponding to each frame of sample image, extract feature information of each frame of high-resolution feature image, and the identifier is fW(x j)，x jRepresenting the sample image of the j-th frame. Inputting the target subimage into a hyper-division network to perform S operation to obtain a semantic feature image, superposing the semantic feature image and the high-resolution subimage to obtain a superposed image, extracting feature information of the superposed image, and identifying the characteristic information as fs(y j)+z j，y jTarget image corresponding to sample image of frame j, fs(y j) Feature information indicating semantic feature image of target image corresponding to sample image of j-th frame, zjFeature information representing a high resolution sub-image of the sample image of the j-th frame. Comparing the characteristic information of each frame of high-resolution characteristic image with the characteristic information of the corresponding superposed image to obtain the difference between the high-resolution characteristic image of each frame of sample image and the superposed image of the corresponding sample image; and calculating the difference sum, and taking the difference sum as the error of the hyper-division network model, wherein the difference sum is marked as W. The error of the hyper-divided network can be represented by equation (1).
Figure PCTCN2018108891-APPB-000001
Wherein, formula (1) MSE (f)W(x j),fs(y j)+z j) And representing the difference degree of the characteristic information of the superposed image of the j frame sample image and the characteristic information of the high-resolution characteristic image of the j frame sample image.
In an embodiment, the image processing apparatus may set a weight for an image output by the image semantic network model, process the semantic feature image of each frame of the sample image according to the weight to obtain a processed semantic feature map, and superimpose the high-resolution sub-image of each frame of the sample image with the processed semantic feature image of the corresponding sample image to obtain a superimposed image.
The image processing device can set a weight for an image output by the image semantic network model according to a scene or according to the requirements of a user, process the semantic feature image corresponding to each frame of sample image according to the weight to obtain a processed semantic feature image, and superpose the high-resolution sub-image of each frame of sample image and the processed semantic feature image of the corresponding sample image to obtain a superposed image. The higher the weight value is, the more information provided by the semantic feature image in the superimposed image is, the higher the definition of the superimposed image is, and further, the high-resolution image output by the hyper-division network model is closer to the semantic feature image; on the contrary, the smaller the weight value is, the less the information provided by the semantic feature image in the superimposed image is, and the definition of the superimposed image is lower, so that the high-resolution image output by the hyper-division network model is closer to the target sub-image.
For example, assuming that the weight set for the image output by the semantic network model is λ, the semantic feature image corresponding to each frame of sample image is processed according to the weight λ to obtain a processed semantic feature image, the high-resolution sub-image of each frame of sample image is superimposed with the processed semantic feature image corresponding to the sample image to obtain a superimposed image, and the feature information of the superimposed image is extracted and can be identified as λ fs(y j)+z j，λf s(y j) For the feature information of the processed semantic feature image, further, the error of the hyper-division network can be represented by equation (2).
Figure PCTCN2018108891-APPB-000002
In one embodiment, step S12 includes: inputting a target sub-image into the image semantic network model, performing semantic recognition through a multilayer neural network included in the image semantic network model to output a plurality of frames of candidate characteristic images, outputting one frame of candidate characteristic image by each layer of neural network, performing gray level processing on each frame of candidate characteristic image to obtain a gray level image, determining a parameter value of each frame of gray level image, and taking the gray level image with the maximum parameter value as the semantic characteristic image of a sample image corresponding to the target sub-image, wherein the parameter value is determined according to the definition of the gray level image and/or the information content provided by the gray level image.
In order to output a semantic feature image with higher quality, the image processing device can input the target sub-image into the image semantic network model, and perform semantic recognition through a multilayer neural network included in the image semantic network model to output a plurality of frames of candidate feature images. And carrying out gray level processing on each frame of candidate characteristic image to obtain a gray level image, determining a parameter value of each frame of gray level image, and taking the gray level image with the maximum parameter value as a semantic characteristic image of a sample image corresponding to the target sub-image, namely taking the gray level image with clear edge structure and capable of providing rich detail characteristic information as the semantic characteristic image, so that network parameters of the hyper-distributed network model can be trained through the semantic characteristic image with higher quality to obtain the hyper-distributed network model capable of outputting high-quality high-resolution images.
S202, judging whether the error is smaller than or equal to a preset error value. The image processing device may determine whether the error is less than or equal to a preset error value, and when the error is less than or equal to the preset error value, it indicates that the hyper-division network model can output a high-resolution image with higher quality, and may perform step S204; otherwise, when the error is greater than the preset error value, which indicates that the hyper-segmentation network model cannot output a high-resolution image with higher quality, step S205 may be performed.
S203, adjusting the network parameters of the hyper-division network model.
When the error is greater than the preset error value, the network parameters of the hyper-division network model are adjusted, and S201 is repeatedly executed, and the error of the hyper-division network model is smaller than or equal to the preset error value, so that the hyper-division network model can output a high-quality high-resolution image.
And S204, acquiring a target image needing super-resolution processing.
When the error of the hyper-resolution network model is smaller than the preset error, the hyper-resolution network model can output a high-resolution image with higher quality, and the image processing device can acquire a target image needing super-resolution processing.
And S205, inputting the target image into the hyper-resolution network model for processing to obtain a high-resolution image.
And inputting the target image into a hyper-resolution network model for processing to obtain a high-resolution image, so that more detail characteristic information and edge characteristic information with higher definition can be obtained from the high-resolution image.
It can be seen that, by implementing the method described in fig. 2, since the network parameters of the hyper-resolution network model are obtained by adjusting according to a large number of sample images and the semantic feature images of each frame of sample image, and the semantic feature images include the detail feature information and the edge structure information of the sample images, the high-resolution images obtained by the hyper-resolution network model can provide more detail feature information and provide high-definition edge structure information, thereby improving the quality of the high-resolution images.
Please refer to fig. 4, which is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus described in this embodiment, the apparatus comprising:
an acquiring module 401, configured to acquire a target image that needs to be super-resolution processed.
A processing module 402, configured to input the target image into a hyper-resolution network model for processing, so as to obtain a high-resolution image;
the network parameters of the hyper-division network model are obtained by adjusting according to a plurality of frames of sample images and semantic feature maps corresponding to the sample images of each frame, and the semantic feature maps are obtained by performing semantic recognition through an image semantic network model.
A determining module 403, configured to determine an error of the hyper-division network model according to the multiple frames of sample images and the semantic feature map corresponding to each frame of the sample images.
An adjusting module 404, configured to adjust a network parameter of the hyper-division network model when the error is greater than a preset error value.
The determining module 403 is specifically configured to obtain a high-resolution sub-image and a low-resolution sub-image corresponding to each frame of sample image in multiple frames of sample images; inputting each frame of target sub-image into the image semantic network model for semantic recognition to obtain a semantic feature image corresponding to each frame of sample image, wherein the target sub-image is a high-resolution sub-image or a low-resolution sub-image corresponding to any sample image in the multi-frame sample images; inputting each frame of low-resolution sub-image into the hyper-division network model for processing to obtain a high-resolution characteristic image of each frame of sample image; superposing the high-resolution sub-image of each frame of sample image and the semantic feature image of the corresponding sample image to obtain a superposed image; determining the difference degree between the high-resolution characteristic image of each frame of sample image and the superposed image of the corresponding sample image; and calculating the sum of the difference degrees, and taking the sum of the difference degrees as the error of the hyper-division network model.
And a setting module 405, configured to set a weight for the image output by the image semantic network model.
The processing module 402 is further configured to process the semantic feature image of each frame of the sample image according to the weight to obtain a processed semantic feature map.
The determining module 403 is specifically configured to superimpose the high-resolution sub-image of each frame of the sample image and the processed semantic feature image of the corresponding sample image to obtain a superimposed image.
The determining module 403 is specifically configured to input the target sub-image into the image semantic network model, perform semantic recognition through a multilayer neural network included in the image semantic network model, and output multiple candidate feature images, where each layer of the neural network outputs one candidate feature image; carrying out gray level processing on each frame of candidate characteristic image to obtain a gray level image; and determining the parameter value of each frame of the gray scale image, and taking the gray scale image with the maximum parameter value as the semantic feature image of the sample image corresponding to the target sub-image, wherein the parameter value is determined according to the definition of the gray scale image and/or the information quantity provided by the gray scale image.
The obtaining module 401 is further configured to obtain a type of the target image; and determining a hyper-segmentation network model matched with the type of the target image.
The processing module 402 is configured to input the target image into a hyper-resolution network model matched with the type of the target image, and process the target image to obtain a high-resolution image.
It can be seen that, by implementing the apparatus described in fig. 4, since the network parameters of the hyper-resolution network model are obtained by adjusting according to a large number of sample images and the semantic feature images of each frame of sample image, and the semantic feature images include the detail feature information and the edge structure information of the sample images, the high-resolution image obtained by the hyper-resolution network model can provide more detail feature information and provide high-definition edge structure information, thereby improving the quality of the high-resolution image.
Please refer to fig. 5, which is a schematic structural diagram of an electronic device according to an embodiment of the present invention. The electronic device includes: the system comprises a processor 501, a memory 502, a communication interface 503 and a power supply 504, wherein the processor 501, the memory 502, the communication interface 503 and the power supply 504 are connected with each other through a bus.
The processor 501 may be one or more CPUs, and in the case that the processor 501 is one CPU, the CPU may be a single-core CPU or a multi-core CPU, and the processor 501 may include a modem for performing modulation or demodulation processing on a signal received by the transceiver 805.
Memory 502 includes, but is not limited to, RAM, ROM), EPROM, CD-ROM, and the memory 502 is used to store instructions, an operating system, various applications, and data.
The communication interface 503 is connected to a forwarding plane device or other control plane device. For example, the communication interface 503 includes a plurality of interfaces respectively connected to a plurality of terminals or connected to a forwarding plane device. The communication interface 503 may be a wired interface, a wireless interface, or a combination thereof. The wired interface may be, for example, an ethernet interface. The ethernet interface may be an optical interface, an electrical interface, or a combination thereof. The wireless interface may be, for example, a Wireless Local Area Network (WLAN) interface, a cellular network interface, or a combination thereof.
And a power supply 504 for supplying power to the control plane device.
The memory 502 is also used to store program instructions. The processor 501 may call the program instructions stored in the memory 502 to implement the image processing method according to the embodiments of the present application.
Based on the same inventive concept, the principle of the control plane device provided in the embodiment of the present invention to solve the problem is similar to that of the embodiment of the method of the present invention, so the implementation and beneficial effects of the control plane device may be referred to and beneficial effects, and for brevity, detailed descriptions are omitted here.
The present invention also provides a computer-readable storage medium, on which a computer program is stored, and the implementation and advantageous effects of the program for solving the problems can refer to the implementation and advantageous effects of the image processing method shown in fig. 1 and fig. 2, and repeated details are omitted.
The embodiment of the present invention further provides a computer program product, where the computer program product includes a non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed, the computer program causes the computer to execute the steps of the image processing method in the embodiment corresponding to fig. 1 and fig. 2, and the implementation and beneficial effects of the computer program product for solving the problem may refer to the implementation and beneficial effects of the image processing method in fig. 1 and fig. 2, and repeated details are not repeated.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above.

Claims (15)

  1. An image processing method, characterized in that the method comprises:
    acquiring a target image needing super-resolution processing;
    inputting the target image into a hyper-resolution network model for processing to obtain a high-resolution image;
    the network parameters of the hyper-division network model are obtained by adjusting according to a plurality of frames of sample images and semantic feature maps corresponding to the sample images of each frame, and the semantic feature maps are obtained by performing semantic recognition through an image semantic network model.
  2. The method of claim 1, further comprising:
    determining the error of the hyper-division network model according to the multiple frames of sample images and the semantic feature map corresponding to each frame of sample image;
    and when the error is larger than a preset error value, adjusting the network parameters of the hyper-division network model.
  3. The method according to claim 2, wherein the determining the error of the hyper-division network model according to the plurality of frames of sample images and the semantic feature map corresponding to each frame of the sample images comprises:
    acquiring a high-resolution sub-image and a low-resolution sub-image corresponding to each frame of sample image in a plurality of frames of sample images;
    inputting each frame of target sub-image into the image semantic network model for semantic recognition to obtain a semantic feature image corresponding to each frame of sample image, wherein the target sub-image is a high-resolution sub-image or a low-resolution sub-image corresponding to any sample image in the multi-frame sample images;
    inputting each frame of low-resolution sub-image into the hyper-division network model for processing to obtain a high-resolution characteristic image of each frame of sample image;
    superposing the high-resolution sub-image of each frame of sample image and the semantic feature image of the corresponding sample image to obtain a superposed image;
    determining the difference degree between the high-resolution characteristic image of each frame of sample image and the superposed image of the corresponding sample image;
    and calculating the sum of the difference degrees, and taking the sum of the difference degrees as the error of the hyper-division network model.
  4. The method of claim 3, further comprising:
    setting weights for the images output by the image semantic network model;
    processing the semantic feature image of each frame of sample image according to the weight to obtain a processed semantic feature image;
    the superimposing the high-resolution sub-image of each frame of the sample image and the semantic feature image of the corresponding sample image to obtain a superimposed image includes:
    and superposing the high-resolution sub-image of each frame of sample image and the processed semantic feature image of the corresponding sample image to obtain a superposed image.
  5. The method according to claim 3, wherein the image semantic network model comprises a multilayer neural network, and the step of inputting each frame of target sub-image into the image semantic network model for semantic recognition to obtain a semantic feature image corresponding to each frame of sample image comprises:
    inputting the target sub-images into the image semantic network model, performing semantic recognition through a multilayer neural network included in the image semantic network model to output a plurality of frames of candidate characteristic images, and outputting one frame of candidate characteristic image through each layer of the neural network;
    carrying out gray level processing on each frame of candidate characteristic image to obtain a gray level image;
    and determining the parameter value of each frame of the gray scale image, and taking the gray scale image with the maximum parameter value as the semantic feature image of the sample image corresponding to the target sub-image, wherein the parameter value is determined according to the definition of the gray scale image and/or the information quantity provided by the gray scale image.
  6. The method according to any one of claims 1-5, further comprising:
    acquiring the type of the target image;
    determining a hyper-divided network model matched with the type of the target image;
    the step of inputting the target image into a hyper-resolution network model for processing to obtain a high-resolution image comprises the following steps:
    and inputting the target image into a hyper-resolution network model matched with the type of the target image for processing to obtain a high-resolution image.
  7. An image processing apparatus, characterized in that the apparatus comprises:
    the acquisition module is used for acquiring a target image needing super-resolution processing;
    the processing module is used for inputting the target image into a hyper-resolution network model for processing to obtain a high-resolution image;
    the network parameters of the hyper-division network model are obtained by adjusting according to a plurality of frames of sample images and semantic feature maps corresponding to the sample images of each frame, and the semantic feature maps are obtained by performing semantic recognition through an image semantic network model.
  8. The apparatus of claim 7, further comprising:
    the device further comprises:
    the determining module is used for determining the error of the hyper-division network model according to the multi-frame sample images and the semantic feature map corresponding to each frame of sample image;
    and the adjusting module is used for adjusting the network parameters of the hyper-division network model when the error is larger than a preset error value.
  9. The apparatus of claim 8,
    the determining module is specifically used for acquiring a high-resolution sub-image and a low-resolution sub-image corresponding to each frame of sample image in a plurality of frames of sample images; inputting each frame of target sub-image into the image semantic network model for semantic recognition to obtain a semantic feature image corresponding to each frame of sample image, wherein the target sub-image is a high-resolution sub-image or a low-resolution sub-image corresponding to any sample image in the multi-frame sample images; inputting each frame of low-resolution sub-image into the hyper-division network model for processing to obtain a high-resolution characteristic image of each frame of sample image; superposing the high-resolution sub-image of each frame of sample image and the semantic feature image of the corresponding sample image to obtain a superposed image; determining the difference degree between the high-resolution characteristic image of each frame of sample image and the superposed image of the corresponding sample image; and calculating the sum of the difference degrees, and taking the sum of the difference degrees as the error of the hyper-division network model.
  10. The apparatus of claim 9, further comprising:
    the setting module is used for setting weight for the image output by the image semantic network model;
    the processing module is further used for processing the semantic feature image of each frame of the sample image according to the weight to obtain a processed semantic feature image;
    the determining module is specifically configured to superimpose the high-resolution sub-image of each frame of the sample image with the processed semantic feature image of the corresponding sample image to obtain a superimposed image.
  11. The apparatus of claim 9,
    the determining module is specifically configured to input the target sub-image into the image semantic network model, perform semantic recognition through a multilayer neural network included in the image semantic network model, and output a plurality of candidate feature images, where each layer of the neural network outputs one candidate feature image; carrying out gray level processing on each frame of candidate characteristic image to obtain a gray level image; and determining the parameter value of each frame of the gray scale image, and taking the gray scale image with the maximum parameter value as the semantic feature image of the sample image corresponding to the target sub-image, wherein the parameter value is determined according to the definition of the gray scale image and/or the information quantity provided by the gray scale image.
  12. The apparatus according to any one of claims 7-11, further comprising:
    the acquisition module is further used for acquiring the type of the target image; determining a hyper-divided network model matched with the type of the target image;
    and the processing module is used for inputting the target image into a hyper-resolution network model matched with the type of the target image for processing to obtain a high-resolution image.
  13. An electronic device comprising at least one processor, memory, and instructions stored on the memory and executed by the at least one processor, wherein the at least one processor executes the instructions to implement the steps of the image processing method of any one of claims 1 to 6.
  14. A computer-readable storage medium, characterized in that the computer storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the steps of the image processing method according to any one of claims 1 to 6.
  15. A computer program product, characterized in that it comprises a non-transitory computer-readable storage medium storing a computer program, the computer program performing operations to cause a computer to implement the steps of the image processing method of any one of claims 1 to 6.
CN201880093293.9A 2018-09-29 2018-09-29 Image processing method, device and equipment Active CN112088393B (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/108891 WO2020062191A1 (en) 2018-09-29 2018-09-29 Image processing method, apparatus and device

Publications (2)

Publication Number Publication Date
CN112088393A true CN112088393A (en) 2020-12-15
CN112088393B CN112088393B (en) 2022-09-23

Family

ID=69952653

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201880093293.9A Active CN112088393B (en) 2018-09-29 2018-09-29 Image processing method, device and equipment

Country Status (2)

Country Link
CN (1) CN112088393B (en)
WO (1) WO2020062191A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116883236A (en) * 2023-05-22 2023-10-13 阿里巴巴（中国）有限公司 Image superdivision method and image data processing method

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112016542A (en) * 2020-05-08 2020-12-01 珠海欧比特宇航科技股份有限公司 Urban waterlogging intelligent detection method and system
CN111709878B (en) * 2020-06-17 2023-06-23 北京百度网讯科技有限公司 Face super-resolution implementation method and device, electronic equipment and storage medium
CN111932463B (en) * 2020-08-26 2023-05-30 腾讯科技（深圳）有限公司 Image processing method, device, equipment and storage medium
CN113971763A (en) * 2020-12-21 2022-01-25 河南铮睿科达信息技术有限公司 Small target segmentation method and device based on target detection and super-resolution reconstruction
CN113592709B (en) * 2021-02-19 2023-07-25 腾讯科技（深圳）有限公司 Image super processing method, device, equipment and storage medium
CN116612466B (en) * 2023-07-20 2023-09-29 腾讯科技（深圳）有限公司 Content identification method, device, equipment and medium based on artificial intelligence

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105793891A (en) * 2013-11-30 2016-07-20 夏普株式会社 Method and device for determining a high resolution output image
CN106780363A (en) * 2016-11-21 2017-05-31 北京金山安全软件有限公司 Picture processing method and device and electronic equipment
WO2018035805A1 (en) * 2016-08-25 2018-03-01 Intel Corporation Coupled multi-task fully convolutional networks using multi-scale contextual information and hierarchical hyper-features for semantic image segmentation
CN108229455A (en) * 2017-02-23 2018-06-29 北京市商汤科技开发有限公司 Object detecting method, the training method of neural network, device and electronic equipment
CN108537731A (en) * 2017-12-29 2018-09-14 西安电子科技大学 Image super-resolution rebuilding method based on compression multi-scale feature fusion network

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080162561A1 (en) * 2007-01-03 2008-07-03 International Business Machines Corporation Method and apparatus for semantic super-resolution of audio-visual data
CN107169450A (en) * 2017-05-15 2017-09-15 中国科学院遥感与数字地球研究所 The scene classification method and system of a kind of high-resolution remote sensing image
CN108428212A (en) * 2018-01-30 2018-08-21 中山大学 A kind of image magnification method based on double laplacian pyramid convolutional neural networks

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105793891A (en) * 2013-11-30 2016-07-20 夏普株式会社 Method and device for determining a high resolution output image
WO2018035805A1 (en) * 2016-08-25 2018-03-01 Intel Corporation Coupled multi-task fully convolutional networks using multi-scale contextual information and hierarchical hyper-features for semantic image segmentation
US20190164290A1 (en) * 2016-08-25 2019-05-30 Intel Corporation Coupled multi-task fully convolutional networks using multi-scale contextual information and hierarchical hyper-features for semantic image segmentation
CN106780363A (en) * 2016-11-21 2017-05-31 北京金山安全软件有限公司 Picture processing method and device and electronic equipment
CN108229455A (en) * 2017-02-23 2018-06-29 北京市商汤科技开发有限公司 Object detecting method, the training method of neural network, device and electronic equipment
CN108537731A (en) * 2017-12-29 2018-09-14 西安电子科技大学 Image super-resolution rebuilding method based on compression multi-scale feature fusion network

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116883236A (en) * 2023-05-22 2023-10-13 阿里巴巴（中国）有限公司 Image superdivision method and image data processing method
CN116883236B (en) * 2023-05-22 2024-04-02 阿里巴巴（中国）有限公司 Image superdivision method and image data processing method

Also Published As

Publication number Publication date
WO2020062191A1 (en) 2020-04-02
CN112088393B (en) 2022-09-23

Similar Documents

Publication Publication Date Title
CN112088393B (en) Image processing method, device and equipment
CN110555795A (en) High resolution style migration
CN111507333B (en) Image correction method and device, electronic equipment and storage medium
CN112990219B (en) Method and device for image semantic segmentation
CN103824269B (en) Face effect processing method and system
CN110569844B (en) Ship recognition method and system based on deep learning
CN114445633B (en) Image processing method, apparatus and computer readable storage medium
CN114863539B (en) Portrait key point detection method and system based on feature fusion
CN110619334B (en) Portrait segmentation method based on deep learning, architecture and related device
CN112308866A (en) Image processing method, image processing device, electronic equipment and storage medium
CN110866938B (en) Full-automatic video moving object segmentation method
US20230153965A1 (en) Image processing method and related device
CN112037239B (en) Text guidance image segmentation method based on multi-level explicit relation selection
WO2023036157A1 (en) Self-supervised spatiotemporal representation learning by exploring video continuity
CN118015644B (en) Social media keyword data analysis method and device based on pictures and characters
CN115482529A (en) Method, equipment, storage medium and device for recognizing fruit image in near scene
CN115661829A (en) Image-text recognition method and data processing method of image-text recognition model
CN115187456A (en) Text recognition method, device, equipment and medium based on image enhancement processing
CN116912268A (en) Skin lesion image segmentation method, device, equipment and storage medium
CN116245157A (en) Facial expression representation model training method, facial expression recognition method and facial expression recognition device
CN113505247B (en) Content-based high-duration video pornography content detection method
CN115294557A (en) Image processing method, image processing apparatus, electronic device, and storage medium
CN116310615A (en) Image processing method, device, equipment and medium
CN114219977A (en) Age estimation method, age estimation system, electronic equipment and storage medium
CN110163031A (en) The device and method and storage medium at the position for test object

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant