WO2021179826A1 - 图像处理方法及相关产品 - Google Patents

图像处理方法及相关产品 Download PDF

Info

Publication number
WO2021179826A1
WO2021179826A1 PCT/CN2021/073971 CN2021073971W WO2021179826A1 WO 2021179826 A1 WO2021179826 A1 WO 2021179826A1 CN 2021073971 W CN2021073971 W CN 2021073971W WO 2021179826 A1 WO2021179826 A1 WO 2021179826A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
processed
super
training
images
Prior art date
Application number
PCT/CN2021/073971
Other languages
English (en)
French (fr)
Inventor
孙哲
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Publication of WO2021179826A1 publication Critical patent/WO2021179826A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Definitions

  • the super-resolution (SR) algorithm is an image processing task that is used to map low-resolution images to high-resolution images in order to enhance image details.
  • the current image SR algorithm is generally implemented by a generative adversarial network (GAN).
  • GAN generative adversarial network
  • the image generated by GAN is prone to blurring in some details, resulting in poor image processing effect.
  • the embodiments of the present application provide an image processing method and related products, which can improve the super-resolution processing effect of an image.
  • the first aspect of the embodiments of the present application provides an image processing method, including:
  • N is a positive integer greater than or equal to 2;
  • the de-occlusion processed image containing the ROI is input into the trained super-resolution model to obtain a result image.
  • a second aspect of the embodiments of the present application provides an image processing device, including:
  • the acquiring unit is used to acquire the image to be processed
  • a segmentation unit configured to segment the image to be processed into N segmented images, where N is a positive integer greater than or equal to 2;
  • the occlusion removal unit is configured to input the N block images into the trained occlusion removal model to obtain N block images after the occlusion removal;
  • a splicing unit configured to splice the N pieces of unobstructed block images according to the segmentation order of the image to be processed to obtain an unobstructed processed image
  • a determining unit configured to determine a region of interest ROI in the unobstructed processed image according to the difference between the unobstructed processed image and the to-be-processed image, and obtain an unobstructed processed image containing the ROI;
  • the super-resolution unit is used to input the ROI-containing unobstructed processed image into the trained super-resolution model to obtain a result image.
  • a third aspect of the embodiments of the present application provides a terminal device, including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program The instruction executes the step instruction in the first aspect of the embodiment of the present application.
  • a fourth aspect of the embodiments of the present application provides a computer-readable storage medium, wherein the above-mentioned computer-readable storage medium stores a computer program for electronic data exchange, wherein the above-mentioned computer program enables a computer to execute Some or all of the steps described in one aspect.
  • the fifth aspect of the embodiments of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to make a computer execute Example part or all of the steps described in the first aspect.
  • the computer program product may be a software installation package.
  • the terminal device obtains the image to be processed, and divides the image to be processed into N block images, where N is a positive integer greater than or equal to 2; Obstruct the object model to obtain N block images after removing the obstruction; stitch the N block images after removing the obstruction according to the segmentation order of the image to be processed to obtain the processed image for removing the obstruction; The difference between the de-occlusion processed image and the image to be processed determines the region of interest ROI in the de-occluded processed image to obtain a de-occluded processed image containing the ROI; and the de-occluded processed image containing the ROI Input the trained super-resolution model to get the result image.
  • the image to be processed is divided into N block images and input to the trained de-occlusion model for de-occlusion processing, which can be calculated in parallel to improve the speed of the de-occlusion processing.
  • the occlusion removal process determine the ROI in the occlusion removal process image. You can only perform super-resolution processing on the ROI in the occlusion removal process image, which can reduce the calculation amount of super-resolution processing, and do super resolution for details. The resolution can make the image after removing the obstruction clearer, and can improve the super-resolution processing effect of the image.
  • FIG. 1 is a schematic structural diagram of a system architecture provided by an embodiment of the present application.
  • FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • Fig. 3a is a schematic diagram of segmentation of a to-be-processed image provided by an embodiment of the present application
  • FIG. 3b is a schematic diagram of rain removal processing of a block image provided by an embodiment of the present application.
  • FIG. 3c is a schematic diagram of stitching block images after rain has been removed according to an embodiment of the present application.
  • Fig. 4a is a schematic diagram of a training process of a de-occlusion model provided by an embodiment of the present application
  • Fig. 4b is a schematic diagram of a training process of a super-resolution model provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of an image processing flow provided by an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of an image processing device provided by an embodiment of the application.
  • Fig. 7 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.
  • the terminal devices involved in the embodiments of the present application may include various handheld devices with wireless communication functions, vehicle-mounted devices, wearable devices, computing devices or other processing devices connected to wireless modems, as well as various forms of user equipment (user equipment).
  • equipment UE
  • mobile station mobile station
  • terminal device terminal device
  • FIG. 1 is a schematic structural diagram of a system architecture provided by an embodiment of the present application.
  • the system architecture includes a server 100 and at least one terminal device 101 communicatively connected with the server 100.
  • a client may be installed on the terminal device 101, and a server may be installed on the server 100.
  • the client refers to a program that corresponds to the server and provides local services to customers, such as image processing services.
  • the server is also a program installed on the server.
  • the server serves the client.
  • the content of the service is to provide computing or application services to the client, provide resources to the client, save client data, etc., for example, the server can Provide the client with a computing model for image processing.
  • the server 100 can directly establish a communication connection with the terminal device 101 via the Internet, and the server 100 can also establish a communication connection with the terminal device 101 via other servers via the Internet.
  • the embodiments of this application do not make limitations.
  • FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application. As shown in FIG. 2, the image processing method may include the following steps.
  • the terminal device obtains an image to be processed, and divides the image to be processed into N block images, where N is a positive integer greater than or equal to 2.
  • the image to be processed may be an image containing fine obstructions such as raindrops, snowflakes, sand and dust. For example, an image taken under a rainy scene.
  • the terminal device can use an image segmentation algorithm to segment the image to be processed into N block images.
  • any two block images have the same size. That is, in the N block images, any two block images contain the same number of pixels under the same display ratio.
  • the image to be processed has 1000*2000 pixels, which can be divided into 4 500*1000 block images.
  • the image segmentation algorithm is an algorithm for segmentation according to the image size and the number of image pixels.
  • the divided image 1, the divided image 2, the divided image 3, and the divided image 4 can be divided. According to the order from top to bottom and from left to right, they are divided image 1, divided image 2, divided image 3, and divided image 4, respectively.
  • Fig. 3a takes raindrops as the shelter as an example for illustration. It can be seen from Figure 3a that there are raindrops in the block image 1, the block image 3, and the block image 4. These raindrops will occlude the background object and affect the display effect of the background object. There are no raindrops in the tiled image 2.
  • the terminal device inputs the N block images into the trained de-occlusion model to obtain N block images after the de-occlusion.
  • the de-obstructing object model may include a de-raining model, a snow-removing model, a dust-removing model, and other fine obstruction models.
  • the fine obstruction may be a tiny, dense obstruction, such as raindrops, snowflakes, dust, sand and so on.
  • the trained de-occlusion model may include a trained de-occlusion generation module.
  • the de-occlusion generating module may be a generative adversarial network (GAN) model.
  • GAN generative adversarial network
  • Fig. 3b takes the shelter as raindrops and the removal of the shelter model is the rain removal model as an example for illustration.
  • Block image (block image after rain removal 1, block image after rain removal 2, block image after rain removal 3, and block image after rain removal 4).
  • input block image 1 into the trained rain removal model to get the block image 1 after rain removal
  • input block image 2 into the trained rain removal model to get the block after rain removal Image 2
  • Input block image 3 into the trained rain removal model to get the block image 3 after rain removal
  • input block image 4 into the trained rain removal model to get the block after rain removal Image 4.
  • the occlusion object model may include an occlusion object generation module and an occlusion object identification module.
  • the original clear image used for occlusion removal training and the occluded image corresponding to the original clear image are obtained, and the occluded image is input to the occluder generation module, and the unoccluded image is output.
  • the unobstructed object image and the original clear image are input to the obstructing object discrimination module, and the obstructing object discrimination module judges whether the unobstructed object image is a real image.
  • the goal of the de-occlusion generation module is to generate as real images as possible to deceive the de-occlusion discrimination module.
  • the goal of the de-occlusion discrimination module is to separate the image generated by the de-occlusion generation module from the real image as much as possible.
  • the unobstructed object generation module and the unobstructed object discrimination module constitute a dynamic "game process.”
  • the de-occlusion model is considered to be a trained de-occlusion model.
  • it can generate an image that is "real and fake" enough for the de-occlusion generation module. It is difficult for the de-occlusion identification module to determine whether the image generated by the de-occlusion generation module is real or not.
  • the terminal device stitches the N pieces of unobstructed block images according to the segmentation order of the image to be processed, to obtain the unobstructed processed image.
  • the terminal device stitches the N blocks of unobstructed images according to the segmentation order of the image to be processed, so that the difference between the obtained unobstructed processed image and the to-be-processed image is only in the unobstructed processing , The background content of the image has not changed.
  • Fig. 3c takes raindrops as an example for illustration.
  • the block image 4) is spliced according to the segmentation order of the image to be processed, and the rain-removing processed image can be obtained.
  • the terminal device determines the region of interest ROI in the de-occlusion processed image according to the difference between the de-occlusion processed image and the image to be processed, and obtains the de-occluded processed image including the ROI.
  • the terminal device can determine the occluder area in the occluder processed image according to the difference between the occluded object processed image and the image to be processed, and the occluder area and stitching gap in the processed image of the occluder removed , Local corners and other locations are regarded as regions of interest (ROI).
  • ROI regions of interest
  • the difference between the to-be-processed image and the de-obstructed processed image is mainly the obstructed area, stitching gaps, and local corners in the de-obstructed processed image.
  • the occlusion area, stitching gaps, and local corners in the image to be processed may appear unnatural or blurred. Based on this, further super-resolution is needed. Processing to eliminate the blur or unnatural phenomenon of the unobstructed processed image after the unobstructed processing and the splicing processing of the to-be-processed image.
  • step 204 may specifically include the following steps:
  • the terminal device performs pixel subtraction processing on the processed image for removing the obstruction and the image to be processed to obtain an attention map
  • the terminal device performs pixel multiplication processing on the attention map and the unobstructed processed image to obtain an unobstructed processed image containing an ROI.
  • the terminal device performs pixel value subtraction processing on the processed image of the de-occlusion object and the image to be processed, and the resulting attention map has a pixel with a smaller pixel value (for example, a pixel with a pixel value of 0).
  • Pure black pixels are background images (that is, areas that have not been de-occluded), and points with larger pixel values in the attention map (for example, pure white pixels with a pixel value of 255) are occluded Area (that is, the area where the occluder processing in the past, the splicing gap, the local corner, etc.).
  • the attention map can be visualized as a pile of white dots on a black background.
  • step (12) the terminal device performs pixel value multiplication processing on the attention map and the de-occluded processed image, and the pixel with a smaller pixel value in the attention map is the same as the pixel corresponding to the de-occluded processed image. After multiplying, what you get is still the pixel point (background area, the unobstructed object that has not been processed by the de-occlusion object) corresponding to the processed image of the de-occlusion object.
  • the result is the ROI in the processed image of the de-occluder (that is, the area where the past occluder processing was performed, the stitching gap, the local corner, etc.), and the ROI in the processed image of the de-occluded object can be determined ROI, get the occluded processed image containing the ROI.
  • the embodiment of the present application provides a method for quickly determining the ROI in the unobstructed processed image. Subsequent only needs to perform super-resolution processing on the ROI in the unobstructed processed image, thereby reducing the calculation amount of the super-resolution processing.
  • the terminal device inputs the processed image of the unobstructed object containing the ROI into the trained super-resolution model to obtain a result image.
  • the trained super-resolution model may include a trained super-resolution generation module.
  • the super-resolution generation module may be a generative adversarial network (GAN) model.
  • the super-resolution model may include a super-resolution generation module and a super-resolution discrimination module.
  • the original clear image used for super-resolution training and the blurred image corresponding to the original clear image are obtained, the blurred image corresponding to the original clear image is input to the super-resolution generation module, and the super-resolution is output
  • the processed image, the super-resolution processed image and the original clear image are input to the super-resolution determination module, and the super-resolution determination module determines whether the super-resolution processed image is a real image.
  • the trained super-resolution model can perform super-resolution processing on the ROI in the de-occluded processed image to obtain a result image.
  • step 205 may include the following steps:
  • the terminal device divides the unobstructed processed image containing the ROI into N block processed images
  • the terminal device determines the M block-processed images that contain the ROI and the P block-processed images that do not contain the ROI in the N block-processed images;
  • the terminal device inputs the M segmented processed images into the trained super-resolution model to obtain M segmented result images;
  • the terminal device splices the M block result images and the P block processed images according to the segmentation order of the ROI-containing de-occlusion processed image to obtain a result image.
  • the ROI in the de-occlusion processed image containing ROI, the ROI may be relatively concentrated. At this time, it is only necessary to divide the de-occlusion processed image containing the ROI into N block processed images, and determine the N blocks
  • the segmentation method of the image processed by removing the obstruction of the ROI is similar to the segmentation method of the image to be processed shown in FIG.
  • the splicing method of the result image (including the M segmented result images and P segmented processed images) is similar to the splicing method of the unobstructed processed image shown in FIG.
  • the terminal device can input M block processed images into the trained super-resolution model to obtain M block result images, which can be calculated in parallel to improve the speed of super-resolution processing.
  • the image to be processed is divided into N block images and input to the trained de-occlusion model for de-occlusion processing, which can be calculated in parallel to improve the speed of the de-occlusion processing.
  • the occlusion removal process determine the ROI in the occlusion removal process image. You can only perform super-resolution processing on the ROI in the occlusion removal process image, which can reduce the calculation amount of super-resolution processing, and do super resolution for details. The resolution can make the image after removing the obstruction clearer, and can improve the super-resolution processing effect of the image.
  • step 202 the following steps may also be performed:
  • the terminal device acquires image training samples used for occlusion removal training, where the image training samples used for occlusion removal training include an original clear image and an occluded image corresponding to the original clear image;
  • the terminal device inputs the original clear image and the occluded image corresponding to the original clear image into the de-occluded object model to obtain a de-occluded training result;
  • the terminal device optimizes the model parameters of the unobstructed object model according to the unobstructed object training result.
  • the training samples may be multiple image training samples used for occlusion removal training to form a set of image training samples used for occlusion removal training.
  • the training samples are also different.
  • the training samples include the original clear image and the raindrop image corresponding to the original clear image;
  • the training sample includes the original clear image and the original clear image.
  • the dust-removing model the training sample includes the original clear image and the dust-added image corresponding to the original clear image;
  • the dust-removing model the training sample includes the original clear image and the original clear image. The image corresponds to the dust image.
  • the terminal may select image training samples from a set of image training samples used for de-occlusion training, and each image training sample includes an original clear image and an occluded image corresponding to the original clear image.
  • one original clear image can correspond to multiple occluded images.
  • the original clear image refers to a clear image without obstructions, for example, a clear image taken on a sunny day.
  • a clear image refers to an image with no jitter and no blur that the resolution of the image meets certain requirements.
  • the occluded image corresponding to the original clear image refers to an image with occluder effect added to the original clear image, and the background of the occluded image is the same as the background of the corresponding original clear image.
  • the terminal device can input the original clear image and the occluded image corresponding to the original clear image into the de-occlusion model to obtain the training loss, optimize the model parameters of the occlusion model based on the training loss, and then input another original image
  • the clear image and the occluded image corresponding to the other original clear image are input to the de-occluded model, and another training loss is obtained.
  • the de-occluded model is determined to be a trained de-occluded object Model.
  • the original clear image can be used as the label map for this training, and the occluded image corresponding to the original clear image is input to the de-occluded object model to generate an unoccluded processed image, which will be de-occluded
  • the object processed image is compared with the original clear image, the error is calculated, and the model parameters of the unobstructed object model are optimized according to the error.
  • the training loss can be characterized by a loss function or by an error function.
  • the loss function can be optimized using an adaptive gradient descent method (for example, the Adam optimization method).
  • step (31) the terminal device acquiring image training samples used for occlusion removal training may include the following steps:
  • the terminal device obtains the original clear image, and performs blocking processing on the original clear image to obtain an initial image with blocking object corresponding to the original clear image;
  • the terminal device performs the occlusion effect processing on the occluder-added initial image to obtain an occluder-added effect image corresponding to the original clear image; the occluder-added initial image and the occluder-added effect image are An image with an obstructed object corresponding to the original clear image.
  • the terminal device may construct an image training sample set used for training of removing obstructions.
  • a terminal device can obtain many original clear images, and the acquisition channels include the original clear image taken by the terminal device, the original clear image obtained from the network, and the original clear image transmitted from other devices.
  • the obstructed image corresponding to the original clear image is not easy to obtain .
  • the terminal device After acquiring the original clear image, the terminal device performs the shielding processing on the original clear image to obtain the shielded initial image corresponding to the original clear image.
  • the terminal device may use image processing software to perform the masking process on the original clear image to obtain the masked initial image.
  • image processing software to perform the masking process on the original clear image to obtain the masked initial image.
  • the size and effect of the occluder of the initial image with the occluder are fixed.
  • the occluder effect processing is further performed on the initial image with the occluder to obtain an image with the occluder effect corresponding to the original clear image.
  • the raindrop effect can include: light rain effect (under strong wind or light wind), moderate rain effect (under strong wind or light wind), heavy rain effect (under strong wind or light wind), heavy rain (under strong wind or light wind) The effect, the drizzle effect (under strong or small wind), the effect of increasing raindrops and the effect of decreasing raindrops, etc.
  • the raindrop effect processing may specifically include: adding noise in various directions to the initial image of adding raindrops, and generating a raindrop effect image through operations such as expansion.
  • data enhancement can be performed to enrich the image training sample set used for occlusion removal training.
  • the training effect of subsequent training to remove obstructions can be improved.
  • step (31) to step (33) can also be executed by the server. After the server has trained the trained model, it can be deployed on the terminal device.
  • step 205 Before performing step 205, the following steps may be performed:
  • the terminal device acquires image training samples used for super-resolution training, where the image training samples used for super-resolution training include an original clear segmented image and a blur processed image corresponding to the original clear segmented image;
  • the terminal device inputs the original clear segmented image and the blur processed image corresponding to the original clear segmented image into the super-resolution model to obtain a super-resolution training result;
  • the terminal device optimizes the model parameters of the super-resolution model according to the super-resolution training result.
  • each image training sample includes an original clear segmented image and a blurred image corresponding to the original clear segmented image.
  • one original clear segmented image can correspond to at least one blurry processed image.
  • the original clear segmented image can be segmented from an original clear image, and each original clear segmented image has the same size. For example, you can use the same size (for example, the image resolution is 300*500) to randomly crop at different positions on an original clear image, you can get multiple original clear segmented images, each of which is originally clear segmented
  • the size of the image is the same, but the content is not exactly the same. Randomly crop the same original clear image to obtain multiple original clear segmented images of the same size, which can expand the image training sample set for super-resolution training.
  • the original clear segmented image contains fewer pixels than the original clear image, it occupies a relatively small storage space, which can increase the speed of super-resolution training.
  • the original clear segmented image refers to an image with no jitter and no blur that the resolution of the image meets certain requirements (for example, the image resolution is greater than 100*100 and less than 1000*1000).
  • the fuzzy processed image corresponding to the original sharply segmented image refers to an image that is fuzzy processed on the basis of the original sharply segmented image, and the background of the blurred processed image is the same as the background of the corresponding original sharply segmented image.
  • the terminal device can input the original clear segmented image and the fuzzy processed image corresponding to the original clear segmented image into the super-resolution model to obtain the training loss, optimize the model parameters of the super-resolution model according to the training loss, and then input another image
  • the original clear segmented image and the blurred image corresponding to the other original clear segmented image are input to the super-resolution model, and another training loss is obtained. Until the training loss meets expectations and converges, it is determined that the super-resolution model is a trained super-resolution model. Resolution model.
  • the original clear segmented image can be used as the label map for this training, and the blur processed image corresponding to the original clear segmented image is input into the super-resolution model to generate the super-resolution processed image, and the super-resolution processed image is generated.
  • the resolution processed image is compared with the original clear segmented image, the error is calculated, and the model parameters of the super-resolution model are optimized according to the error.
  • the training loss can be characterized by a loss function or by an error function.
  • the loss function can be optimized using an adaptive gradient descent method (for example, the Adam optimization method).
  • step (41) the terminal device acquiring image training samples for super-resolution training may include the following steps:
  • the terminal device obtains the original clear segmented image, performs blur processing on the original clear segmented image, and obtains a blurred processed image corresponding to the original clear segmented image.
  • the terminal device may use image processing software to perform blur processing on the original clear segmented image to obtain a blur processed image corresponding to the original clear segmented image.
  • step (41) the terminal device acquiring image training samples for super-resolution training may include the following steps:
  • the terminal device obtains an original clear image, performs blur processing on the original clear image to obtain an original blur processed image corresponding to the original clear image, and intercepts multiple blur processed images of the same size from the original blurred image.
  • multiple blurred images of the same size are intercepted from the original blurred image, data enhancement can be performed, and the image training sample set used for super-resolution training can be enriched. In turn, the training effect of subsequent super-resolution training can be improved.
  • step (41) to step (43) above can also be executed by the server. After the super-resolution model is trained by the server, it can be deployed on the terminal device.
  • FIG. 4a is a schematic diagram of a training process of a de-occlusion model provided by an embodiment of the present application.
  • the original image is input to the occlusion model (for example, rain GAN, snow GAN, dust GAN, dust GAN, etc.), and the occlusion model outputs a generated image. Compare the generated image with the original When the error of the label map corresponding to the image is the smallest, it can be determined that the occluder model is a trained occluder model.
  • the occlusion model for example, rain GAN, snow GAN, dust GAN, dust GAN, etc.
  • FIG. 4b is a schematic diagram of a training process of a super-resolution model provided by an embodiment of the present application.
  • the original image is input into a super-resolution model (for example, super-resolution GAN), the super-resolution model outputs a generated image, and the generated image is compared with the error of the label image corresponding to the original image.
  • a super-resolution model for example, super-resolution GAN
  • the super-resolution model outputs a generated image
  • the generated image is compared with the error of the label image corresponding to the original image.
  • the super-resolution model is a trained super-resolution model.
  • FIG. 5 is a schematic diagram of an image processing flow provided by an embodiment of the present application.
  • the original image is divided into multiple sub-images, and the multiple sub-images are input into the de-occlusion model (for example, the de-occlusion GAN), and the block image after the de-occlusion processing is obtained.
  • the de-occlusion model for example, the de-occlusion GAN
  • the block image after the de-occlusion processing is obtained.
  • a stitched image is obtained.
  • the stitched image is subtracted from the original image to obtain the inconsistencies.
  • the attention map the attention map is multiplied by the stitched image to obtain the attention map.
  • the attention map is input to a super-resolution model (for example, a super-resolution GAN), and a clear result map is obtained.
  • a super-resolution model for example, a super-resolution GAN
  • the terminal device includes a hardware structure and/or software module corresponding to each function.
  • this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software-driven hardware depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.
  • the embodiment of the present application may divide the terminal device into functional units according to the foregoing method examples.
  • each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit.
  • the above-mentioned integrated unit can be implemented in the form of hardware or software functional unit. It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical function division, and there may be other division methods in actual implementation.
  • FIG. 6 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the application.
  • the image processing apparatus 600 is applied to terminal equipment.
  • the image processing apparatus 600 may include an acquiring unit 601, The unit 602, the obstruction removing unit 603, the splicing unit 604, the determining unit 605 and the super-resolution unit 606, wherein:
  • the segmentation unit 602 is configured to segment the image to be processed into N segmented images, where N is a positive integer greater than or equal to 2;
  • the occlusion removal unit 603 is configured to input the N block images into the trained occlusion removal model to obtain N block images after the occlusion removal;
  • the splicing unit 604 is configured to splice the N pieces of unobstructed block images according to the segmentation order of the image to be processed to obtain the unobstructed processed image;
  • the determining unit 605 is configured to determine the region of interest ROI in the unobstructed processed image according to the difference between the unobstructed processed image and the image to be processed, and obtain the unobstructed processed image including the ROI;
  • the super-resolution unit 606 is configured to input the ROI-containing unobstructed processed image into the trained super-resolution model to obtain a result image.
  • the determining unit 605 determines the region of interest ROI in the unobstructed processed image according to the difference between the unobstructed processed image and the image to be processed, and obtains the unobstructed processed image including the ROI, Specifically: performing pixel subtraction processing on the unobstructed processed image and the to-be-processed image to obtain an attention map; performing pixel multiplication processing on the attention map and the unobstructed processed image to obtain an ROI Remove the occluder and process the image.
  • the super-resolution unit 606 inputs the de-occluded processed image containing the ROI into the trained super-resolution model to obtain a result image, specifically: segmenting the de-occluded processed image containing the ROI Into N block-processed images; determine M block-processed images containing ROI and P block-processed images that do not contain ROI in the N block-processed images; input the M block-processed images into training A good super-resolution model to obtain M segmented result images; the M segmented processed images and the P segmented processed images are spliced according to the segmentation order of the de-obstructed processed image containing the ROI, Get the result image.
  • the image processing apparatus 600 may further include a training unit 607 and an optimization unit 608;
  • the acquiring unit 601 is further configured to acquire image training samples for de-occlusion training before the de-occlusion unit 603 inputs the N block images into the trained de-occlusion model ,
  • the image training sample used for the training of removing the occluder includes an original clear image and an occluded image corresponding to the original clear image;
  • the training unit 607 is configured to input the original clear image and the occluded image corresponding to the original clear image into the de-occluder model to obtain a de-occluded training result;
  • the optimization unit 608 is configured to optimize the model parameters of the de-occluder model according to the result of the de-occluder training.
  • the acquiring unit 601 acquires image training samples used for de-occlusion training, specifically: acquiring the original clear image, and performing an occlusion process on the original clear image to obtain a comparison with the original clear image The corresponding initial image with occluder; occluder effect processing is performed on the initial image of occluder to obtain an occluder effect image corresponding to the original clear image; the occluder initial image and the occluder The effect image is an obstructed image corresponding to the original clear image.
  • the acquiring unit 601 is further configured to acquire the data used for super-resolution training before the super-resolution unit 606 inputs the de-occlusion processed image containing the ROI into the trained super-resolution model.
  • Image training samples where the image training samples used for super-resolution training include an original clear segmented image and a blur processed image corresponding to the original clear segmented image;
  • the training unit 607 is configured to input the original sharply segmented image and the blurred processed image corresponding to the original sharply segmented image into the super-resolution model to obtain a super-resolution training result;
  • the optimization unit 608 is configured to optimize the model parameters of the super-resolution model according to the super-resolution training result.
  • the acquiring unit 601 acquires image training samples used for super-resolution training, specifically: acquiring the original clear segmented image, performing blur processing on the original clear segmented image, and obtaining the difference between the original clear segmented image and the original clear segmented image The image corresponding to the blurred processed image.
  • the acquisition unit 601, the segmentation unit 602, the occlusion removal unit 603, the splicing unit 604, the determination unit 605, the super-resolution unit 606, the training unit 607, and the optimization unit 608 in the embodiment of the present application may be processing in the terminal device.
  • the acquisition unit 601, the segmentation unit 602, the occlusion removal unit 603, the splicing unit 604, the determination unit 605, the super-resolution unit 606, the training unit 607, and the optimization unit 608 in the embodiment of the present application may be processing in the terminal device. Device.
  • the image to be processed is divided into N block images and input to the trained de-occlusion model for de-occlusion processing, which can be calculated in parallel to improve the speed of the de-occlusion processing.
  • the occlusion removal process determine the ROI in the occlusion removal process image. You can only perform super-resolution processing on the ROI in the occlusion removal process image, which can reduce the calculation amount of super-resolution processing, and do super resolution for details. The resolution can make the image after removing the obstruction clearer, and can improve the super-resolution processing effect of the image.
  • FIG. 7 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.
  • the terminal device 700 includes a processor 701 and a memory 702. 703 are connected to each other.
  • the communication bus 703 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc.
  • the communication bus 703 can be divided into an address bus, a data bus, a control bus, and so on. For ease of representation, only one thick line is used in FIG. 7, but it does not mean that there is only one bus or one type of bus.
  • the memory 702 is used to store a computer program.
  • the computer program includes program instructions.
  • the processor 701 is configured to call the program instructions.
  • the above program includes a method for executing the method shown in FIG. 2.
  • the processor 701 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the programs in the above scheme.
  • CPU central processing unit
  • ASIC application-specific integrated circuit
  • the memory 702 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, random access memory (RAM), or other types that can store information and instructions
  • the dynamic storage device can also be electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), CD-ROM (Compact Disc Read-Only Memory, CD-ROM) or other optical disc storage, optical disc storage (Including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store desired program codes in the form of instructions or data structures and can be used by a computer Any other media accessed, but not limited to this.
  • the memory can exist independently and is connected to the processor through a bus.
  • the memory can also be integrated with the processor.
  • the terminal device 700 may also include general components such as a communication interface and an antenna, which will not be described in detail here.
  • the image to be processed is divided into N block images and input to the trained de-occlusion model for de-occlusion processing, which can be calculated in parallel to improve the speed of the de-occlusion processing .
  • the occlusion removal process determine the ROI in the occlusion removal process image. You can only perform super-resolution processing on the ROI in the occlusion removal process image, which can reduce the calculation amount of super-resolution processing, and do super resolution for details. The resolution can make the image after removing the obstruction clearer, and can improve the super-resolution processing effect of the image.
  • the embodiments of the present application also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, and the computer program causes a computer to execute any image recorded in the above method embodiments. Part or all of the steps of the treatment method.
  • the disclosed device may be implemented in other ways.
  • the device embodiments described above are merely illustrative.
  • the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or may be Integrate into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • each functional unit in each embodiment may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit can be implemented in the form of hardware or in the form of software program modules.
  • the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer readable memory.
  • the technical solution of the present application essentially or the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory.
  • a number of instructions are included to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application.
  • the foregoing memory includes: U disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.
  • the program can be stored in a computer-readable memory, and the memory can include: a flash disk , Read-only memory, random access device, magnetic or optical disk, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

本申请实施例提供一种图像处理方法及相关产品，该图像处理方法包括：终端设备获取待处理图像，将所述待处理图像分割成N个分块图像，N为大于或等于2的正整数；将所述N个分块图像输入训练好的去遮挡物模型，得到N个去遮挡物后的分块图像；将所述N个去遮挡物后的分块图像按照所述待处理图像的分割顺序进行拼接，得到去遮挡物处理图像；根据所述去遮挡物处理图像与所述待处理图像的差异确定所述去遮挡物处理图像中的感兴趣区域ROI，得到包含ROI的去遮挡物处理图像；将所述包含ROI的去遮挡物处理图像输入训练好的超分辨率模型，得到结果图像。本申请实施例可以提高图像的超分辨率处理效果。

Description

图像处理方法及相关产品 技术领域
本申请涉及图像处理技术领域，具体涉及一种图像处理方法及相关产品。
背景技术
在图像处理领域中，超分辨率(super resolution，SR)算法是一项图像处理任务，用于将低分辨率的图像映射至高分辨率，以期达到增强图像细节的作用。
目前的图像SR算法一般采用生成式对抗网络(generative adversarial network，GAN)来实现。采用GAN生成的图像，在一些细节部分容易产生模糊现象，导致图像处理效果较差。
发明内容
本申请实施例提供一种图像处理方法及相关产品，可以提高图像的超分辨率处理效果。
本申请实施例的第一方面提供了一种图像处理方法，包括：
获取待处理图像，将所述待处理图像分割成N个分块图像，N为大于或等于2的正整数；
将所述N个分块图像输入训练好的去遮挡物模型，得到N个去遮挡物后的分块图像；
将所述N个去遮挡物后的分块图像按照所述待处理图像的分割顺序进行拼接，得到去遮挡物处理图像；
根据所述去遮挡物处理图像与所述待处理图像的差异确定所述去遮挡物处理图像中的感兴趣区域ROI，得到包含ROI的去遮挡物处理图像；
将所述包含ROI的去遮挡物处理图像输入训练好的超分辨率模型，得到结果图像。
本申请实施例的第二方面提供了一种图像处理装置，包括：
获取单元，用于获取待处理图像；
分割单元，用于将所述待处理图像分割成N个分块图像，N为大于或等于2的正整数；
去遮挡物单元，用于将所述N个分块图像输入训练好的去遮挡物模型，得到N个去遮挡物后的分块图像；
拼接单元，用于将所述N个去遮挡物后的分块图像按照所述待处理图像的分割顺序进行拼接，得到去遮挡物处理图像；
确定单元，用于根据所述去遮挡物处理图像与所述待处理图像的差异确定所述去遮挡物处理图像中的感兴趣区域ROI，得到包含ROI的去遮挡物处理图像；
超分辨率单元，用于将所述包含ROI的去遮挡物处理图像输入训练好的超分辨率模型，得到结果图像。
本申请实施例的第三方面提供了一种终端设备，包括处理器和存储器，所述存储器用于存储计算机程序，所述计算机程序包括程序指令，所述处理器被配置用于调用所述程序指令，执行如本申请实施例第一方面中的步骤指令。
本申请实施例的第四方面提供了一种计算机可读存储介质，其中，上述计算机可读存储介质存储用于电子数据交换的计算机程序，其中，上述计算机程序使得计算机执行如本申请实施例第一方面中所描述的部分或全部步骤。
本申请实施例的第五方面提供了一种计算机程序产品，其中，上述计算机程序产品包括存储了计算机程序的非瞬时性计算机可读存储介质，上述计算机程序可操作来使计算机执行如本申请实施例第一方面中所描述的部分或全部步骤。该计算机程序产品可以为一个软件安装包。
本申请实施例中，终端设备获取待处理图像，将所述待处理图像分割成N个分块图像，N为大于或等于2的正整数；将所述N个分块图像输入训练好的去遮挡物模型，得到N个去遮挡物后的分块图像；将所述N个去遮挡物后的分块图像按照所述待处理图像的分割顺序进行拼接，得到去遮挡物处理图像；根据所述去遮挡物处理图像与所述待处理图像的差异确定所述去遮挡物处理图像中的感兴趣区域ROI，得到包含ROI的去遮挡物处理图像；将所述包含ROI的去遮挡物处理图像输入训练好的超分辨率模型，得到结果图像。
本申请实施例中，将待处理图像分割成N个分块图像分别输入到训练好的去遮挡物模型进行去遮挡物处理，可以并行计算，提高去遮挡物处理的速度。在去遮挡物处理之后，确定去遮挡物处理图像中的ROI，可以仅对去遮挡物处理图像中的ROI进行超分辨率处理，可以减少超分辨率处理的计算量，并且针对细节部分做超分辨率可以使得去遮挡物后的图像更加清晰，可以提高图像的超分辨率处理效果。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请实施例提供的一种系统架构的结构示意图；
图2是本申请实施例提供的一种图像处理方法的流程示意图；
图3a是本申请实施例提供的一种待处理图像的分割示意图；
图3b是本申请实施例提供的一种分块图像的去雨处理示意图；
图3c是本申请实施例提供的一种去雨后的分块图像的拼接示意图；
图4a是本申请实施例提供的一种去遮挡物模型的训练流程示意图；
图4b是本申请实施例提供的一种超分辨率模型的训练流程示意图；
图5是本申请实施例提供的一种图像处理的流程示意图；
图6为本申请实施例提供的一种图像处理装置的结构示意图；
图7是本申请实施例提供的一种终端设备的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别不同对象，而不是用于描述特定顺序。此外，术语“包括”和“具有”以及它们任何变形，意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、系统、产品或设备没有限定于已列出的步骤或单元，而是可选地还包括没有列出的步骤或单元，或可选地还包括对于这些过程、方法、产品或设备固有的其他步骤或单元。
在本申请中提及“实施例”意味着，结合实施例描述的特定特征、结构或特性可以包含在本申请的至少一个实施例中。在说明书中的各个位置出现该短语并不一定均是指相同的实施例，也不是与其它实施例互斥的独立的或备选的实施例。本领域技术人员显式地和隐式地理解的是，本申请所描述的实施例可以与其它实施例相结合。
本申请实施例所涉及到的终端设备可以包括各种具有无线通信功能的手持设备、车载设备、可穿戴设备、计算设备或连接到无线调制解调器的其他处理设备，以及各种形式的 用户设备(user equipment，UE)，移动台(mobile station，MS)，终端设备(terminal device)等等。为方便描述，上面提到的设备统称为终端设备。
请参阅图1，图1是本申请实施例提供的一种系统架构的结构示意图，如图1所示，该系统架构包括服务器100和与服务器100通信连接的至少一个终端设备101。终端设备101上可以安装有客户端，服务器100上可以安装有服务端。客户端是指与服务器相对应，为客户提供本地服务的程序，比如提供图像处理的服务。服务端也是在服务器上安装的一段程序，服务端是为客户端服务的，服务的内容诸如向客户端提供计算或者应用服务，向客户端提供资源，保存客户端数据等，比如，服务端可以向客户端提供图像处理的计算模型。服务器100可以直接与终端设备101通过互联网建立通信连接，服务端100也可以通过其他服务器与终端设备101通过互联网建立通信连接。本申请实施例不做限定。
请参阅图2，图2是本申请实施例提供的一种图像处理方法的流程示意图。如图2所示，该图像处理方法可以包括如下步骤。
201，终端设备获取待处理图像，将待处理图像分割成N个分块图像，N为大于或等于2的正整数。
本申请实施例中，在雨天、雪天、沙尘暴天等微细遮挡物的情况下，由于场景的能见度低和背景场景被遮挡，拍摄出来的图像中目标的对比度和颜色等特征都会出现不同程度的衰减，导致背景信息(即目标图像)表达不明确，这使得一些视频或图像系统不能正常工作，因此需要消除遮挡物对图像的影响。
其中，待处理图像可以是包含雨滴、雪花、沙尘等微细遮挡物的图像。比如，在下雨场景下拍摄的图像。终端设备可以采用图像分割算法将待处理图像分割成N个分块图像。其中，N个分块图像中，任意两个分块图像的尺寸相同。也即，N个分块图像中，任意两个分块图像在同一显示比例下所包含的像素点的个数相同。比如，待处理图像为1000*2000个像素点，可以分成4个500*1000的分块图像。图像分割算法是按照图像尺寸和图像像素点个数进行分割的算法。
在图像分割过程中，可以按照一定的分割顺序。比如，如图3a所示，以N＝4为例，对于待处理图像，可以分割成分块图像1、分块图像2、分块图像3和分块图像4。按照从上到下，从左到右的顺序，则分别为分块图像1、分块图像2、分块图像3和分块图像4。图3a以遮挡物为雨滴为例进行说明。从图3a可以看出，分块图像1、分块图3、分块图4中存在雨滴，这些雨滴会对背景物体造成遮挡，影响背景物体的显示效果。分块图像2中没有雨滴存在。
202，终端设备将N个分块图像输入训练好的去遮挡物模型，得到N个去遮挡物后的分块图像。
本申请实施例中，去遮挡物模型可以包括去雨模型、去雪模型、去沙尘模型等去微细遮挡物模型。微细遮挡物可以是微小的、密密麻麻的遮挡物，比如，雨滴、雪花、灰尘、沙尘等。
训练好的去遮挡物模型可以包括训练好的去遮挡物生成模块。去遮挡物生成模块可以是生成式对抗网络(generative adversarial network，GAN)模型。其中，训练好的去遮挡物模型可以有多个，将N个分块图像输入训练好的去遮挡物模型，得到N个去遮挡物后的分块图像。比如，如图3b所示，图3b以遮挡物是雨滴，去遮挡物模型是去雨模型作为示例进行说明。以N＝4为例，对于4个分块图像(分块图像1、分块图像2、分块图像3和分块图像4)，输入训练好的去雨模型，则可以得到去雨后的分块图像(去雨后的分块图像1、去雨后的分块图像2、去雨后的分块图像3和去雨后的分块图像4)。具体的，将分块图像1输入训练好的去雨模型，则可以得到去雨后的分块图像1；将分块图像2输入训练好的去雨模型，则可以得到去雨后的分块图像2；将分块图像3输入训练好的去雨模型，则可以 得到去雨后的分块图像3；将分块图像4输入训练好的去雨模型，则可以得到去雨后的分块图像4。
其中，去遮挡物模型可以包括去遮挡物生成模块和去遮挡物判别模块。去遮挡物模型在训练过程中，获取用于去遮挡物训练的原始清晰图像和该原始清晰图像对应的加遮挡物图像，将加遮挡物图像输入去遮挡物生成模块，输出无遮挡物图像，将该无遮挡物图像和该原始清晰图像输入去遮挡物判别模块，去遮挡物判别模块判别该无遮挡物图像是否为真实的图像。
在训练过程中，去遮挡物生成模块的目标就是尽量生成真实的图像去欺骗去遮挡物判别模块。而去遮挡物判别模块的目标就是尽量把去遮挡物生成模块生成的图像和真实的图像分别开来。这样，去遮挡物生成模块和去遮挡物判别模块构成了一个动态的“博弈过程”。当模型收敛时，则认为去遮挡物模型为训练好的去遮挡物模型。当模型收敛时，对于去遮挡物生成模块而言，它可以生成足以“以假乱真”的图像。对于去遮挡物判别模块来说，它难以判定去遮挡物生成模块生成的图像究竟是不是真实的。
203，终端设备将N个去遮挡物后的分块图像按照待处理图像的分割顺序进行拼接，得到去遮挡物处理图像。
本申请实施例中，终端设备将N个去遮挡物后的分块图像按照待处理图像的分割顺序进行拼接，以使得到的去遮挡物处理图像与待处理图像的区别仅在于去遮挡物处理，图像的背景内容部分并没有变化。
比如，如图3c所示，图3c以遮挡物是雨滴作为示例进行说明。以N＝4为例，对于4个去雨后的分块图像(去雨后的分块图像1、去雨后的分块图像2、去雨后的分块图像3和去雨后的分块图像4)，按照待处理图像的分割顺序进行拼接，即可得到去雨处理图像。
204，终端设备根据去遮挡物处理图像与待处理图像的差异确定去遮挡物处理图像中的感兴趣区域ROI，得到包含ROI的去遮挡物处理图像。
本申请实施例中，终端设备可以根据该去遮挡物处理图像与该待处理图像的差异确定该去遮挡物处理图像中的遮挡物区域，将去遮挡物处理图像中的遮挡物区域、拼接缝隙、局部角落等位置作为感兴趣区域(region of interest，ROI)。
该待处理图像与该去遮挡物处理图像的差异主要为该去遮挡物处理图像中的遮挡物区域、拼接缝隙、局部角落等。该待处理图像经过训练好的去遮挡物模型处理后，该待处理图像中的遮挡物区域、拼接缝隙、局部角落等可能会出现不自然或者模糊的现象，基于此，需要进一步做超分辨率处理，消除该待处理图像在去遮挡物处理和拼接处理后的该去遮挡物处理图像的模糊或者不自然的现象。
为了降低超分辨率处理的计算量，可以找出该去遮挡物处理图像中的感兴趣区域ROI，得到包含ROI的去遮挡物处理图像，后续只需要针对去遮挡物处理图像中的ROI进行超分辨率处理，从而降低超分辨率处理的计算量。
举例来说，参见图3a至图3c可知，该去雨处理图像中的原始分块图像2中不存在雨滴，该分块图像2中不包含ROI，因此，在做超分辨率处理时，无需对分块图像2进行超分辨率处理，从而降低超分辨率处理的计算量。
可选的，步骤204具体可以包括如下步骤：
(11)终端设备将所述去遮挡物处理图像与所述待处理图像进行像素相减处理，得到注意力图；
(12)终端设备将所述注意力图与所述去遮挡物处理图像进行像素相乘处理，得到包含ROI的去遮挡物处理图像。
本申请实施例中，终端设备将所述去遮挡物处理图像与所述待处理图像进行像素值相减处理，得到的注意力图中像素值为较小的像素点(比如，像素值为0的纯黑的像素点) 为背景图(即，未进行去遮挡物处理的区域)，注意力图中像素值较大的点(比如，像素值为255的纯白的像素点)为有遮挡物的区域(即，进行过去遮挡物处理的区域、拼接缝隙、局部角落等)。注意力图可以形象化的理解为黑色背景上一堆白点点的图。
步骤(12)中，终端设备将该注意力图与该去遮挡物处理图像进行像素值相乘处理，该注意力图中像素值为较小的像素点与该去遮挡物处理图像对应的像素点相乘后，得到的仍然是该去遮挡物处理图像对应的像素点(背景区域，未进行去遮挡物处理的去遮挡物)，该注意力图中像素值为较大的像素点与该去遮挡物处理图像对应的像素点相乘后，得到的是该去遮挡物处理图像中的ROI(即，进行过去遮挡物处理的区域、拼接缝隙、局部角落等)，可以确定去遮挡物处理图像中的ROI，得到包含ROI的去遮挡物处理图像。
本申请实施例提供一种快速确定去遮挡物处理图像中的ROI的方法，后续只需要针对去遮挡物处理图像中的ROI进行超分辨率处理，从而降低超分辨率处理的计算量。
205，终端设备将包含ROI的去遮挡物处理图像输入训练好的超分辨率模型，得到结果图像。
本申请实施例中，训练好的超分辨率模型可以包括训练好的超分辨率生成模块。超分辨率生成模块可以是生成式对抗网络(generative adversarial network，GAN)模型。
其中，超分辨率模型可以包括超分辨率生成模块和超分辨率判别模块。超分辨率模型在训练过程中，获取用于超分辨率训练的原始清晰图像和该原始清晰图像对应的模糊图像，将该原始清晰图像对应的模糊图像输入超分辨率生成模块，输出超分辨率处理图像，将该超分辨率处理图像和该原始清晰图像输入超分辨率判别模块，超分辨率判别模块判别该超分辨率处理图像是否为真实的图像。
终端设备将包含ROI的去遮挡物处理图像输入训练好的超分辨率模型后，训练好的超分辨率模型可以对去遮挡物处理图像中的ROI进行超分辨率处理，得到结果图像。
可选的，步骤205可以包括如下步骤：
(21)终端设备将所述包含ROI的去遮挡物处理图像分割成N个分块处理图像；
(22)终端设备确定所述N个分块处理图像中包含ROI的M个分块处理图像和未包含ROI的P个分块处理图像；
(23)终端设备将所述M个分块处理图像输入训练好的超分辨率模型，得到M个分块结果图像；
(24)终端设备将所述M个分块结果图像与所述P个分块处理图像按照所述包含ROI的去遮挡物处理图像的分割顺序进行拼接，得到结果图像。
本申请实施例中，包含ROI的去遮挡物处理图像中，ROI有可能比较集中，此时，只需要将包含ROI的去遮挡物处理图像分割成N个分块处理图像，确定N个分块处理图像中包含ROI的M个分块处理图像和未包含ROI的P个分块处理图像。其中，P+M＝N。其中，ROI的去遮挡物处理图像的分割方式与图3a所示的待处理图像的分割方式类似，可以参见图3a，此处不再赘述。结果图像(包含M个分块结果图像与P个分块处理图像)的拼接方式与图3c所示的去遮挡物处理图像的拼接方式类似，可以参见图3c，此处不再赘述。
终端设备可以将M个分块处理图像输入训练好的超分辨率模型，得到M个分块结果图像，可以并行计算，提高超分辨率处理的速度。
本申请实施例中，将待处理图像分割成N个分块图像分别输入到训练好的去遮挡物模型进行去遮挡物处理，可以并行计算，提高去遮挡物处理的速度。在去遮挡物处理之后，确定去遮挡物处理图像中的ROI，可以仅对去遮挡物处理图像中的ROI进行超分辨率处理，可以减少超分辨率处理的计算量，并且针对细节部分做超分辨率可以使得去遮挡物后的图像更加清晰，可以提高图像的超分辨率处理效果。
可选的，在执行步骤202之前，还可以执行如下步骤：
(31)终端设备获取用于去遮挡物训练的图像训练样本，所述用于去遮挡物训练的图像训练样本包括原始清晰图像和与所述原始清晰图像对应的加遮挡物图像；
(32)终端设备将所述原始清晰图像和与所述原始清晰图像对应的加遮挡物图像输入所述去遮挡物模型，得到去遮挡物训练结果；
(33)终端设备根据所述去遮挡物训练结果对所述去遮挡物模型的模型参数进行优化。
本申请实施例中，用于去遮挡物训练的图像训练样本可以有多个，构成用于去遮挡物训练的图像训练样本集。对于不同的去遮挡物模型，训练样本也不同。比如，对于去雨模型而言，训练样本包括原始清晰图像和与所述原始清晰图像对应的加雨滴图像；对于去雪模型而言，训练样本包括原始清晰图像和与所述原始清晰图像对应的加雪花图像；对于去灰尘模型而言，训练样本包括原始清晰图像和与所述原始清晰图像对应的加灰尘图像；对于去沙尘模型而言，训练样本包括原始清晰图像和与所述原始清晰图像对应的加沙尘图像。
终端可以从用于去遮挡物训练的图像训练样本集中选择图像训练样本，每个图像训练样本都包括原始清晰图像和与该原始清晰图像对应的加遮挡物图像。其中，一张原始清晰图像可以对应多张加遮挡物图像。原始清晰图像是指没有遮挡物的清晰图像，比如，在晴天拍摄的清晰图像。清晰图像指的是没有抖动，没有模糊的图像分辨率达到一定要求的图像。与原始清晰图像对应的加遮挡物图像，指的是在原始清晰图像的基础上增加遮挡物效果的图像，加遮挡物图像的背景与对应的原始清晰图像的背景相同。
终端设备可以将原始清晰图像和与该原始清晰图像对应的加遮挡物图像输入去遮挡物模型，得到训练损失，根据训练损失对去遮挡物模型的模型参数进行优化，然后再输入另一张原始清晰图像和与该另一原始清晰图像对应的加遮挡物图像输入去遮挡物模型，得到另一个训练损失，直到训练损失符合预期并且收敛，则确定该去遮挡物模型为训练好的去遮挡物模型。
具体的，在一次训练过程中，原始清晰图像可以作为这次训练的标签图，将与该原始清晰图像对应的加遮挡物图像输入去遮挡物模型，生成去遮挡物处理图像，将该去遮挡物处理图像与该原始清晰图像进行对比，计算误差，根据误差对该去遮挡物模型的模型参数进行优化。
其中，训练损失可以用损失函数来表征，也可以通过误差函数来表征。损失函数可以采用自适应梯度下降方法(比如，Adam优化方法)进行优化。
可选的，步骤(31)中，终端设备获取用于去遮挡物训练的图像训练样本，可以包括如下步骤：
(311)终端设备获取所述原始清晰图像，对所述原始清晰图像进行加遮挡物处理，得到与所述原始清晰图像对应的加遮挡物初始图像；
(321)终端设备对所述加遮挡物初始图像进行遮挡物效果处理，得到与所述原始清晰图像对应的加遮挡物效果图像；所述加遮挡物初始图像和所述加遮挡物效果图像为所述原始清晰图像对应的加遮挡物图像。
本申请实施例中，终端设备可以构建用于去遮挡物训练的图像训练样本集。一般而言，终端设备可以获取很多原始清晰图像，获取的渠道包括终端设备拍摄的原始清晰图像，从网络上获取的原始清晰图像，从其它设备传输过来的原始清晰图像等。但是，由于有遮挡物存在(比如，下雨天)与无遮挡物存在(比如，晴天)的同一场景下拍摄的图像的背景容易发生变化，与原始清晰图像对应的加遮挡物图像则不容易获得。
终端设备在获取原始清晰图像之后，对该原始清晰图像进行加遮挡物处理，得到与该原始清晰图像对应的加遮挡物初始图像。其中，终端设备可以采用图像处理软件对该原始清晰图像进行加遮挡物处理，得到加遮挡物初始图像。其中，加遮挡物初始图像的遮挡物大小、遮挡物效果都是固定的。
在加遮挡物初始图像的基础上，进一步对加遮挡物初始图像进行遮挡物效果处理，得到与该原始清晰图像对应的加遮挡物效果图像。原始清晰图像对应的加遮挡物效果图像可以有多张。
下面以遮挡物效果为雨滴效果为例进行说明。
其中，雨滴效果可以包括：(大风或小风下的)小雨效果、(大风或小风下的)中雨效果、(大风或小风下的)大雨效果、(大风或小风下的)暴雨效果、(大风或小风下的)毛毛雨效果、雨滴变大效果和雨滴变小效果等。
雨滴效果处理具体可以包括：在该加雨滴初始图像上添加各个方向的噪声，经过膨胀等操作生成加雨滴效果图像。
本申请实施例，通过对原始清晰图像进行加遮挡物处理以及对加遮挡物初始图像进行遮挡物效果处理，可以进行数据增强，丰富用于去遮挡物训练的图像训练样本集。进而可以提高后续去遮挡物训练的训练效果。
需要说明的是，上述步骤(31)至步骤(33)的去遮挡物训练过程也可以由服务器进行执行，当服务器训练得到训练好的去遮挡物模型之后，可以部署在终端设备上。
可选的，在执行步骤205之前，还可以执行如下步骤：
(41)终端设备获取用于超分辨率训练的图像训练样本，所述用于超分辨率训练的图像训练样本包括原始清晰分割图像和与所述原始清晰分割图像对应的模糊处理图像；
(42)终端设备将所述原始清晰分割图像和与所述原始分割清晰图像对应的模糊处理图像输入所述超分辨率模型，得到超分辨率训练结果；
(43)终端设备根据所述超分辨率训练结果对所述超分辨率模型的模型参数进行优化。
本申请实施例中，用于超分辨率训练的图像训练样本可以有多个，构成用于超分辨率训练的图像训练样本集。终端可以从用于超分辨率训练的图像训练样本集中选择图像训练样本，每个图像训练样本都包括原始清晰分割图像和与该原始清晰分割图像对应的模糊处理图像。其中，一张原始清晰分割图像可以对应至少一张模糊处理图像。
原始清晰分割图像可以从一张原始清晰图像分割得到，每张原始清晰分割图像具有相同的尺寸。举例来说，可以在一张原始清晰图像上，采用相同的尺寸(比如，图像分辨率为300*500)在不同的位置随机裁剪，即可得到多张原始清晰分割图像，每张原始清晰分割图像的尺寸相同，但是内容不完全相同。在同一张原始清晰图像中随机裁剪，得到多个相同尺寸的原始清晰分割图像，可以扩充用于超分辨率训练的图像训练样本集。此外，由于原始清晰分割图像包含的像素相对原始清晰图像要少，其占用的存储空间相对较小，可以提高超分辨率训练的速度。
原始清晰分割图像是指没有抖动，没有模糊的图像分辨率达到一定要求(比如，图像分辨率大于100*100并且小于1000*1000)的图像。与原始清晰分割图像对应的模糊处理图像，指的是在原始清晰分割图像的基础上进行模糊处理的图像，模糊处理图像的背景与对应的原始清晰分割图像的背景相同。
终端设备可以将原始清晰分割图像和与该原始清晰分割图像对应的模糊处理图像输入超分辨率模型，得到训练损失，根据训练损失对超分辨率模型的模型参数进行优化，然后再输入另一张原始清晰分割图像和与该另一原始清晰分割图像对应的模糊处理图像输入超分辨率模型，得到另一个训练损失，直到训练损失符合预期并且收敛，则确定该超分辨率模型为训练好的超分辨率模型。
具体的，在一次训练过程中，原始清晰分割图像可以作为这次训练的标签图，将与该原始清晰分割图像对应的模糊处理图像输入超分辨率模型，生成超分辨率处理图像，将该超分辨率处理图像与该原始清晰分割图像进行对比，计算误差，根据误差对该超分辨率模型的模型参数进行优化。
其中，训练损失可以用损失函数来表征，也可以通过误差函数来表征。损失函数可以采用自适应梯度下降方法(比如，Adam优化方法)进行优化。
可选的，步骤(41)中，终端设备获取用于超分辨率训练的图像训练样本，可以包括如下步骤：
终端设备获取所述原始清晰分割图像，对所述原始清晰分割图像进行模糊处理，得到与所述原始清晰分割图像对应的模糊处理图像。
本申请实施例中，终端设备可以采用图像处理软件对该原始清晰分割图像进行模糊处理，得到与该原始清晰分割图像对应的模糊处理图像。
可选的，步骤(41)中，终端设备获取用于超分辨率训练的图像训练样本，可以包括如下步骤：
终端设备获取原始清晰图像，对所述原始清晰图像进行模糊处理，得到与所述原始清晰图像对应的原始模糊处理图像，从该原始模糊处理图像中截取尺寸相同的多个模糊处理图像。
本申请实施例中，从原始模糊处理图像中截取尺寸相同的多个模糊处理图像，可以进行数据增强，丰富用于超分辨率训练的图像训练样本集。进而可以提高后续超分辨率训练的训练效果。
需要说明的是，上述步骤(41)至步骤(43)的超分辨率训练过程也可以由服务器进行执行，当服务器训练得到训练好的超分辨率模型之后，可以部署在终端设备上。
请参阅图4a，图4a是本申请实施例提供的一种去遮挡物模型的训练流程示意图。如图4a所示，将原始图经输入去遮挡物模型(比如，去雨GAN、去雪GAN、去灰尘GAN、去沙尘GAN等)，去遮挡物模型输出生成图，比较生成图与原始图对应的标签图的误差，当误差最小，则可以确定该去遮挡物模型为训练好的去遮挡物模型。
请参阅图4b，图4b是本申请实施例提供的一种超分辨率模型的训练流程示意图。如图4b所示，将原始图经输入超分辨率模型(比如，超分辨率GAN)，超分辨率模型输出生成图，比较生成图与原始图对应的标签图的误差，当误差最小，则可以确定该超分辨率模型为训练好的超分辨率模型。
请参阅图5，图5是本申请实施例提供的一种图像处理的流程示意图。如图5所示，将原图分割成多个分块图，将多个分块图分别输入去遮挡物模型(比如，去遮挡物GAN)，得到去遮挡物处理后的分块图，将多个去遮挡物处理后的分块图进行拼接，得到拼接图，将拼接图与原图进行像素相减，得到不一致的地方，作为注意力图，注意力图乘以拼接图，得到注意力图，将注意力图输入超分辨率模型(比如，超分辨率GAN)，得到清晰的结果图。
上述主要从方法侧执行过程的角度对本申请实施例的方案进行了介绍。可以理解的是，终端设备为了实现上述功能，其包含了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，结合本文中所提供的实施例描述的各示例的单元及算法步骤，本申请能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
本申请实施例可以根据上述方法示例对终端设备进行功能单元的划分，例如，可以对应各个功能划分各个功能单元，也可以将两个或两个以上的功能集成在一个处理单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。需要说明的是，本申请实施例中对单元的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。
与上述一致的，请参阅图6，图6为本申请实施例提供的一种图像处理装置的结构示意图，该图像处理装置600应用于终端设备，该图像处理装置600可以包括获取单元601、分割单元602、去遮挡物单元603、拼接单元604、确定单元605和超分辨率单元606，其中：
所述获取单元601，用于获取待处理图像；
所述分割单元602，用于将所述待处理图像分割成N个分块图像，N为大于或等于2的正整数；
所述去遮挡物单元603，用于将所述N个分块图像输入训练好的去遮挡物模型，得到N个去遮挡物后的分块图像；
所述拼接单元604，用于将所述N个去遮挡物后的分块图像按照所述待处理图像的分割顺序进行拼接，得到去遮挡物处理图像；
所述确定单元605，用于根据所述去遮挡物处理图像与所述待处理图像的差异确定所述去遮挡物处理图像中的感兴趣区域ROI，得到包含ROI的去遮挡物处理图像；
所述超分辨率单元606，用于将所述包含ROI的去遮挡物处理图像输入训练好的超分辨率模型，得到结果图像。
可选的，所述确定单元605根据所述去遮挡物处理图像与所述待处理图像的差异确定所述去遮挡物处理图像中的感兴趣区域ROI，得到包含ROI的去遮挡物处理图像，具体为：将所述去遮挡物处理图像与所述待处理图像进行像素相减处理，得到注意力图；将所述注意力图与所述去遮挡物处理图像进行像素相乘处理，得到包含ROI的去遮挡物处理图像。
可选的，所述超分辨率单元606将所述包含ROI的去遮挡物处理图像输入训练好的超分辨率模型，得到结果图像，具体为：将所述包含ROI的去遮挡物处理图像分割成N个分块处理图像；确定所述N个分块处理图像中包含ROI的M个分块处理图像和未包含ROI的P个分块处理图像；将所述M个分块处理图像输入训练好的超分辨率模型，得到M个分块结果图像；将所述M个分块处理图像与所述P个分块处理图像按照所述包含ROI的去遮挡物处理图像的分割顺序进行拼接，得到结果图像。
可选的，该图像处理装置600还可以包括训练单元607和优化单元608；
可选的，所述获取单元601，还用于在所述去遮挡物单元603将所述N个分块图像输入训练好的去遮挡物模型之前，获取用于去遮挡物训练的图像训练样本，所述用于去遮挡物训练的图像训练样本包括原始清晰图像和与所述原始清晰图像对应的加遮挡物图像；
所述训练单元607，用于将所述原始清晰图像和与所述原始清晰图像对应的加遮挡物图像输入所述去遮挡物模型，得到去遮挡物训练结果；
所述优化单元608，用于根据所述去遮挡物训练结果对所述去遮挡物模型的模型参数进行优化。
可选的，所述获取单元601获取用于去遮挡物训练的图像训练样本，具体为：获取所述原始清晰图像，对所述原始清晰图像进行加遮挡物处理，得到与所述原始清晰图像对应的加遮挡物初始图像；对所述加遮挡物初始图像进行遮挡物效果处理，得到与所述原始清晰图像对应的加遮挡物效果图像；所述加遮挡物初始图像和所述加遮挡物效果图像为所述原始清晰图像对应的加遮挡物图像。
可选的，所述获取单元601，还用于在所述超分辨率单元606将所述包含ROI的去遮挡物处理图像输入训练好的超分辨率模型之前，获取用于超分辨率训练的图像训练样本，所述用于超分辨率训练的图像训练样本包括原始清晰分割图像和与所述原始清晰分割图像对应的模糊处理图像；
所述训练单元607，用于将所述原始清晰分割图像和与所述原始分割清晰图像对应的模糊处理图像输入所述超分辨率模型，得到超分辨率训练结果；
所述优化单元608，用于根据所述超分辨率训练结果对所述超分辨率模型的模型参数进行优化。
可选的，所述获取单元601获取用于超分辨率训练的图像训练样本，具体为：获取所述原始清晰分割图像，对所述原始清晰分割图像进行模糊处理，得到与所述原始清晰分割图像对应的模糊处理图像。
其中，本申请实施例中的获取单元601、分割单元602、去遮挡物单元603、拼接单元604、确定单元605、超分辨率单元606、训练单元607和优化单元608可以是终端设备中的处理器。
本申请实施例中，将待处理图像分割成N个分块图像分别输入到训练好的去遮挡物模型进行去遮挡物处理，可以并行计算，提高去遮挡物处理的速度。在去遮挡物处理之后，确定去遮挡物处理图像中的ROI，可以仅对去遮挡物处理图像中的ROI进行超分辨率处理，可以减少超分辨率处理的计算量，并且针对细节部分做超分辨率可以使得去遮挡物后的图像更加清晰，可以提高图像的超分辨率处理效果。
请参阅图7，图7是本申请实施例提供的一种终端设备的结构示意图，如图7所示，该终端设备700包括处理器701和存储器702，处理器701、存储器702可以通过通信总线703相互连接。通信总线703可以是外设部件互连标准(Peripheral Component Interconnect，简称PCI)总线或扩展工业标准结构(Extended Industry Standard Architecture，简称EISA)总线等。通信总线703可以分为地址总线、数据总线、控制总线等。为便于表示，图7中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。存储器702用于存储计算机程序，计算机程序包括程序指令，处理器701被配置用于调用程序指令，上述程序包括用于执行图2所示的方法。
处理器701可以是通用中央处理器(CPU)，微处理器，特定应用集成电路(application-specific integrated circuit，ASIC)，或一个或多个用于控制以上方案程序执行的集成电路。
存储器702可以是只读存储器(read-only memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(random access memory，RAM)或者可存储信息和指令的其他类型的动态存储设备，也可以是电可擦可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，EEPROM)、只读光盘(Compact Disc Read-Only Memory，CD-ROM)或其他光盘存储、光碟存储(包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质，但不限于此。存储器可以是独立存在，通过总线与处理器相连接。存储器也可以和处理器集成在一起。
此外，该终端设备700还可以包括通信接口、天线等通用部件，在此不再详述。
本申请实施例中，本申请实施例中，将待处理图像分割成N个分块图像分别输入到训练好的去遮挡物模型进行去遮挡物处理，可以并行计算，提高去遮挡物处理的速度。在去遮挡物处理之后，确定去遮挡物处理图像中的ROI，可以仅对去遮挡物处理图像中的ROI进行超分辨率处理，可以减少超分辨率处理的计算量，并且针对细节部分做超分辨率可以使得去遮挡物后的图像更加清晰，可以提高图像的超分辨率处理效果。
本申请实施例还提供一种计算机可读存储介质，其中，该计算机可读存储介质存储用于电子数据交换的计算机程序，该计算机程序使得计算机执行如上述方法实施例中记载的任何一种图像处理方法的部分或全部步骤。
需要说明的是，对于前述的各方法实施例，为了简单描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本申请并不受所描述的动作顺序的限制，因为依据本申请，某些步骤可以采用其他顺序或者同时进行。其次，本领域技术人员也应该知 悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一定是本申请所必须的。
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。
在本申请所提供的几个实施例中，应该理解到，所揭露的装置，可通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在申请明各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件程序模块的形式实现。
所述集成的单元如果以软件程序模块的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储器中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储器中，包括若干指令用以使得一台计算机设备(可为个人计算机、服务器或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储器包括：U盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、移动硬盘、磁碟或者光盘等各种可以存储程序代码的介质。
本领域普通技术人员可以理解上述实施例的各种方法中的全部或部分步骤是可以通过程序来指令相关的硬件来完成，该程序可以存储于一计算机可读存储器中，存储器可以包括：闪存盘、只读存储器、随机存取器、磁盘或光盘等。
以上对本申请实施例进行了详细介绍，本文中应用了具体个例对本申请的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本申请的方法及其核心思想；同时，对于本领域的一般技术人员，依据本申请的思想，在具体实施方式及应用范围上均会有改变之处，综上所述，本说明书内容不应理解为对本申请的限制。

Claims (20)

  1. 一种图像处理方法，其特征在于，包括：
    获取待处理图像，将所述待处理图像分割成N个分块图像，N为大于或等于2的正整数；
    将所述N个分块图像输入训练好的去遮挡物模型，得到N个去遮挡物后的分块图像；
    将所述N个去遮挡物后的分块图像按照所述待处理图像的分割顺序进行拼接，得到去遮挡物处理图像；
    根据所述去遮挡物处理图像与所述待处理图像的差异确定所述去遮挡物处理图像中的感兴趣区域ROI，得到包含ROI的去遮挡物处理图像；
    将所述包含ROI的去遮挡物处理图像输入训练好的超分辨率模型，得到结果图像。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述去遮挡物处理图像与所述待处理图像的差异确定所述去遮挡物处理图像中的感兴趣区域ROI，得到包含ROI的去遮挡物处理图像，包括：
    将所述去遮挡物处理图像与所述待处理图像进行像素相减处理，得到注意力图；
    将所述注意力图与所述去遮挡物处理图像进行像素相乘处理，得到包含ROI的去遮挡物处理图像。
  3. 根据权利要求1或2所述的方法，其特征在于，所述将所述包含ROI的去遮挡物处理图像输入训练好的超分辨率模型，得到结果图像，包括：
    将所述包含ROI的去遮挡物处理图像分割成N个分块处理图像；
    确定所述N个分块处理图像中包含ROI的M个分块处理图像和未包含ROI的P个分块处理图像；
    将所述M个分块处理图像输入训练好的超分辨率模型，得到M个分块结果图像；
    将所述M个分块处理图像与所述P个分块处理图像按照所述包含ROI的去遮挡物处理图像的分割顺序进行拼接，得到结果图像。
  4. 根据权利要求1～3任一项所述的方法，其特征在于，所述将所述N个分块图像输入训练好的去遮挡物模型之前，所述方法还包括：
    获取用于去遮挡物训练的图像训练样本，所述用于去遮挡物训练的图像训练样本包括原始清晰图像和与所述原始清晰图像对应的加遮挡物图像；
    将所述原始清晰图像和与所述原始清晰图像对应的加遮挡物图像输入所述去遮挡物模型，得到去遮挡物训练结果；
    根据所述去遮挡物训练结果对所述去遮挡物模型的模型参数进行优化。
  5. 根据权利要求4所述的方法，其特征在于，所述获取用于去遮挡物训练的图像训练样本，包括：
    获取所述原始清晰图像，对所述原始清晰图像进行加遮挡物处理，得到与所述原始清晰图像对应的加遮挡物初始图像；
    对所述加遮挡物初始图像进行遮挡物效果处理，得到与所述原始清晰图像对应的加遮挡物效果图像；所述加遮挡物初始图像和所述加遮挡物效果图像为所述原始清晰图像对应的加遮挡物图像。
  6. 根据权利要求1～3任一项所述的方法，其特征在于，所述将所述包含ROI的去遮挡物处理图像输入训练好的超分辨率模型之前，所述方法还包括：
    获取用于超分辨率训练的图像训练样本，所述用于超分辨率训练的图像训练样本包括原始清晰分割图像和与所述原始清晰分割图像对应的模糊处理图像；
    将所述原始清晰分割图像和与所述原始分割清晰图像对应的模糊处理图像输入所述超分辨率模型，得到超分辨率训练结果；
    根据所述超分辨率训练结果对所述超分辨率模型的模型参数进行优化。
  7. 根据权利要求6所述的方法，其特征在于，所述获取用于超分辨率训练的图像训练样本，包括：
    获取所述原始清晰分割图像，对所述原始清晰分割图像进行模糊处理，得到与所述原始清晰分割图像对应的模糊处理图像。
  8. 根据权利要求6所述的方法，其特征在于，所述获取用于超分辨率训练的图像训练样本，包括：
    获取所述原始清晰图像，对所述原始清晰图像进行模糊处理，得到与所述原始清晰图像对应的原始模糊处理图像，从该原始模糊处理图像中截取尺寸相同的多个模糊处理图像。
  9. 根据权利要求1所述的方法，其特征在于，所述N个分块图像中，任意两个分块图像的尺寸相同。
  10. 一种图像处理装置，其特征在于，包括：
    获取单元，用于获取待处理图像；
    分割单元，用于将所述待处理图像分割成N个分块图像，N为大于或等于2的正整数；
    去遮挡物单元，用于将所述N个分块图像输入训练好的去遮挡物模型，得到N个去遮挡物后的分块图像；
    拼接单元，用于将所述N个去遮挡物后的分块图像按照所述待处理图像的分割顺序进行拼接，得到去遮挡物处理图像；
    确定单元，用于根据所述去遮挡物处理图像与所述待处理图像的差异确定所述去遮挡物处理图像中的感兴趣区域ROI，得到包含ROI的去遮挡物处理图像；
    超分辨率单元，用于将所述包含ROI的去遮挡物处理图像输入训练好的超分辨率模型，得到结果图像。
  11. 根据权利要求1所述的图像处理装置，其特征在于，所述确定单元根据所述去遮挡物处理图像与所述待处理图像的差异确定所述去遮挡物处理图像中的感兴趣区域ROI，得到包含ROI的去遮挡物处理图像，具体为：
    将所述去遮挡物处理图像与所述待处理图像进行像素相减处理，得到注意力图；将所述注意力图与所述去遮挡物处理图像进行像素相乘处理，得到包含ROI的去遮挡物处理图像。
  12. 根据权利要求10或11所述的图像处理装置，其特征在于，所述超分辨率单元将所述包含ROI的去遮挡物处理图像输入训练好的超分辨率模型，得到结果图像，具体为：
    将所述包含ROI的去遮挡物处理图像分割成N个分块处理图像；确定所述N个分块处理图像中包含ROI的M个分块处理图像和未包含ROI的P个分块处理图像；将所述M个 分块处理图像输入训练好的超分辨率模型，得到M个分块结果图像；将所述M个分块处理图像与所述P个分块处理图像按照所述包含ROI的去遮挡物处理图像的分割顺序进行拼接，得到结果图像。
  13. 根据权利要求10～12任一项所述的图像处理装置，其特征在于，所述图像处理装置还包括训练单元和优化单元；
    所述获取单元，还用于在所述去遮挡物单元将所述N个分块图像输入训练好的去遮挡物模型之前，获取用于去遮挡物训练的图像训练样本，所述用于去遮挡物训练的图像训练样本包括原始清晰图像和与所述原始清晰图像对应的加遮挡物图像；
    所述训练单元，用于将所述原始清晰图像和与所述原始清晰图像对应的加遮挡物图像输入所述去遮挡物模型，得到去遮挡物训练结果；
    所述优化单元，用于根据所述去遮挡物训练结果对所述去遮挡物模型的模型参数进行优化。
  14. 根据权利要求13所述的图像处理装置，其特征在于，所述获取单元获取用于去遮挡物训练的图像训练样本，具体为：
    获取所述原始清晰图像，对所述原始清晰图像进行加遮挡物处理，得到与所述原始清晰图像对应的加遮挡物初始图像；对所述加遮挡物初始图像进行遮挡物效果处理，得到与所述原始清晰图像对应的加遮挡物效果图像；所述加遮挡物初始图像和所述加遮挡物效果图像为所述原始清晰图像对应的加遮挡物图像。
  15. 根据权利要求10～12任一项所述的方法，其特征在于，
    所述获取单元，还用于在所述超分辨率单元将所述包含ROI的去遮挡物处理图像输入训练好的超分辨率模型之前，获取用于超分辨率训练的图像训练样本，所述用于超分辨率训练的图像训练样本包括原始清晰分割图像和与所述原始清晰分割图像对应的模糊处理图像；
    所述训练单元，用于将所述原始清晰分割图像和与所述原始分割清晰图像对应的模糊处理图像输入所述超分辨率模型，得到超分辨率训练结果；
    所述优化单元，用于根据所述超分辨率训练结果对所述超分辨率模型的模型参数进行优化。
  16. 根据权利要求15所述的方法，其特征在于，所述获取单元获取用于超分辨率训练的图像训练样本，具体为：
    获取所述原始清晰分割图像，对所述原始清晰分割图像进行模糊处理，得到与所述原始清晰分割图像对应的模糊处理图像。
  17. 根据权利要求15所述的方法，其特征在于，所述获取单元获取用于超分辨率训练的图像训练样本，具体为：
    获取所述原始清晰图像，对所述原始清晰图像进行模糊处理，得到与所述原始清晰图像对应的原始模糊处理图像，从该原始模糊处理图像中截取尺寸相同的多个模糊处理图像。
  18. 根据权利要求10所述的方法，其特征在于，所述N个分块图像中，任意两个分块图像的尺寸相同。
  19. 一种终端设备，其特征在于，包括处理器和存储器，所述存储器用于存储计算机程序，所述计算机程序包括程序指令，所述处理器被配置用于调用所述程序指令，执行如权利要求1～9任一项所述的方法。
  20. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有计算机程序，所述计算机程序包括程序指令，所述程序指令当被处理器执行时使所述处理器执行如权利要求1～9任一项所述的方法。
PCT/CN2021/073971 2020-03-09 2021-01-27 图像处理方法及相关产品 WO2021179826A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010158548.4 2020-03-09
CN202010158548.4A CN111325667B (zh) 2020-03-09 2020-03-09 图像处理方法及相关产品

Publications (1)

Publication Number Publication Date
WO2021179826A1 true WO2021179826A1 (zh) 2021-09-16

Family

ID=71173179

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/073971 WO2021179826A1 (zh) 2020-03-09 2021-01-27 图像处理方法及相关产品

Country Status (2)

Country Link
CN (1) CN111325667B (zh)
WO (1) WO2021179826A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111275627A (zh) * 2019-02-27 2020-06-12 中国科学院沈阳自动化研究所 一种基于雪模型和深度学习融合的图像去雪算法
CN112288738A (zh) * 2020-11-18 2021-01-29 新疆大学 基于深度学习的单幅图像去雪花方法、装置及存储介质
CN114972093A (zh) * 2022-05-26 2022-08-30 平安科技（深圳）有限公司 图像增强方法、装置、设备及存储介质

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111325667B (zh) * 2020-03-09 2023-05-30 Oppo广东移动通信有限公司 图像处理方法及相关产品
CN112509025B (zh) * 2020-12-03 2022-11-11 山东省科学院海洋仪器仪表研究所 一种基于三维欧氏距离的岩石空间结构距离图的计算方法
CN113887737B (zh) * 2021-09-23 2024-05-17 北京工商大学 一种基于机器学习的样本集自动生成方法

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180165513A1 (en) * 2012-10-01 2018-06-14 The Regents Of The University Of California Unified face representation for individual recognition in surveillance videos and vehicle logo super-resolution system
CN108681994A (zh) * 2018-05-11 2018-10-19 京东方科技集团股份有限公司 一种图像处理方法、装置、电子设备及可读存储介质
CN109360156A (zh) * 2018-08-17 2019-02-19 上海交通大学 基于生成对抗网络的图像分块的单张图像去雨方法
CN109886900A (zh) * 2019-03-15 2019-06-14 西北大学 一种基于字典训练和稀疏表示的合成雨图除雨方法
CN109934193A (zh) * 2019-03-20 2019-06-25 福建师范大学 全局上下文先验约束的抗遮挡人脸超分辨率方法及其系统
CN110570376A (zh) * 2019-09-10 2019-12-13 广州工程技术职业学院 一种图像去雨方法、装置、设备和计算机可读存储介质
CN110580681A (zh) * 2019-09-12 2019-12-17 杭州海睿博研科技有限公司 高分辨率心机运动模式分析装置和方法
CN111325667A (zh) * 2020-03-09 2020-06-23 Oppo广东移动通信有限公司 图像处理方法及相关产品

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108805828B (zh) * 2018-05-22 2023-08-04 腾讯科技（深圳）有限公司 图像处理方法、装置、计算机设备和存储介质
CN109978805A (zh) * 2019-03-18 2019-07-05 Oppo广东移动通信有限公司 拍照处理方法、装置、移动终端以及存储介质
CN110458752B (zh) * 2019-07-18 2022-11-11 西北工业大学 一种基于局部遮挡条件下的图像换脸方法

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180165513A1 (en) * 2012-10-01 2018-06-14 The Regents Of The University Of California Unified face representation for individual recognition in surveillance videos and vehicle logo super-resolution system
CN108681994A (zh) * 2018-05-11 2018-10-19 京东方科技集团股份有限公司 一种图像处理方法、装置、电子设备及可读存储介质
CN109360156A (zh) * 2018-08-17 2019-02-19 上海交通大学 基于生成对抗网络的图像分块的单张图像去雨方法
CN109886900A (zh) * 2019-03-15 2019-06-14 西北大学 一种基于字典训练和稀疏表示的合成雨图除雨方法
CN109934193A (zh) * 2019-03-20 2019-06-25 福建师范大学 全局上下文先验约束的抗遮挡人脸超分辨率方法及其系统
CN110570376A (zh) * 2019-09-10 2019-12-13 广州工程技术职业学院 一种图像去雨方法、装置、设备和计算机可读存储介质
CN110580681A (zh) * 2019-09-12 2019-12-17 杭州海睿博研科技有限公司 高分辨率心机运动模式分析装置和方法
CN111325667A (zh) * 2020-03-09 2020-06-23 Oppo广东移动通信有限公司 图像处理方法及相关产品

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
DI ZHANG, JIAZHONG HE, YUN ZHAO: "Learning to perform joint image super-resolution and rain removal via a single-convolutional neural network", JOURNAL OF ELECTRONIC IMAGING, vol. 27, no. 6, 15 December 2018 (2018-12-15), pages 1 - 11, XP055846036, DOI: 10.1117/1.JEI.27.6.063024 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111275627A (zh) * 2019-02-27 2020-06-12 中国科学院沈阳自动化研究所 一种基于雪模型和深度学习融合的图像去雪算法
CN111275627B (zh) * 2019-02-27 2023-04-11 中国科学院沈阳自动化研究所 一种基于雪模型和深度学习融合的图像去雪算法
CN112288738A (zh) * 2020-11-18 2021-01-29 新疆大学 基于深度学习的单幅图像去雪花方法、装置及存储介质
CN112288738B (zh) * 2020-11-18 2022-07-29 新疆大学 基于深度学习的单幅图像去雪花方法、装置及存储介质
CN114972093A (zh) * 2022-05-26 2022-08-30 平安科技（深圳）有限公司 图像增强方法、装置、设备及存储介质

Also Published As

Publication number Publication date
CN111325667B (zh) 2023-05-30
CN111325667A (zh) 2020-06-23

Similar Documents

Publication Publication Date Title
WO2021179826A1 (zh) 图像处理方法及相关产品
US10692197B2 (en) Systems and techniques for automatic image haze removal across multiple video frames
US9311901B2 (en) Variable blend width compositing
CN111275626B (zh) 一种基于模糊度的视频去模糊方法、装置及设备
Tai et al. Correction of spatially varying image and video motion blur using a hybrid camera
WO2016206087A1 (zh) 一种低照度图像处理方法和装置
KR100951965B1 (ko) Ｌｔｉ／ｃｔｉ에 의한 비디오 선명도와 대비도의 강조
US9142009B2 (en) Patch-based, locally content-adaptive image and video sharpening
US10600158B2 (en) Method of video stabilization using background subtraction
US8411981B2 (en) Method of removing blur without ringing-artifact
WO2023273868A1 (zh) 图像降噪方法、装置、终端、存储介质
WO2021013049A1 (zh) 前景图像获取方法、前景图像获取装置和电子设备
CN111353955A (zh) 一种图像处理方法、装置、设备和存储介质
JP6857224B2 (ja) 画像内の乱流の影響を減少させるシステム及び方法
CN108234826B (zh) 图像处理方法及装置
WO2023019870A1 (zh) 视频处理方法及装置、电子设备、存储介质、计算机程序、计算机程序产品
US11727540B2 (en) Image sharpening
WO2023221636A1 (zh) 视频处理方法、装置、设备、存储介质和程序产品
JP3818044B2 (ja) ノイズ除去装置及びノイズ除去方法並びにコンピュータ読み取り可能な記録媒体
CN111161299A (zh) 影像分割方法、计算机程序、存储介质及电子装置
CN111986095B (zh) 基于边缘提取的图像处理方法及图像处理装置
CN116418932A (zh) 一种视频帧内插方法及装置
CN111986096A (zh) 基于边缘提取的漫画生成方法及漫画生成装置
CN114373153B (zh) 一种基于多尺度阵列相机的视频成像优化系统与方法
WO2021102702A1 (zh) 图像处理方法和装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21767071

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21767071

Country of ref document: EP

Kind code of ref document: A1