KR20240089729A - Image processing methods, devices, storage media and electronic devices - Google Patents

Abstract

This application discloses image processing methods, devices, storage media, and electronic devices. Here, the image processing method includes acquiring an image to be processed including a shaded area, and inputting the image to be processed into a trained neural network to obtain a deshading image, and the neural network is a two-stage cascade-connected first stage. It includes a network and a second-stage network, where the first-stage network receives the image to be processed and outputs a shaded area mask image, and the second-stage network simultaneously receives the image to be processed and the shaded area mask image and outputs a shadow-removed image. This application can solve the technical problem of removing shadow areas among the prior art, which tends to cause side effects on the background layer of the image, and requires a high hardware platform.

Description

Image processing methods, devices, storage media and electronic devices

This application is Chinese Patent Application No. 202111210502 filed on October 18, 2021. 3, priority is claimed, and the contents of the Chinese patent application are incorporated herein as part of this application.

(technology field)

This application relates to image processing technology, and specifically to image processing methods, devices, storage media, and electronic devices.

When people take pictures of documents with mobile phones, there are always shadows left on the documents due to the shielding of the hands or mobile phones against the light rays and other objects in the environment against the light rays, which affects the visual experience of the captured images. By processing the image with computer vision processing technology, the quality of the image can be effectively improved by removing the shading and restoring the content of the text or picture behind the shading. Removing the shading of the document is an important technology that improves the quality of the captured image. It can be greatly improved and the market outlook is bright.

Effectively removing the shadow layer while not causing significant side effects on the background layer, as well as having fast execution speed and acceptable hardware configuration requirements are the basic requirements and main challenges in applying the shadow removal method to mobile phones, and currently The shading removal method cannot completely remove shading, loses background layer information, or has slow execution speed, all of which are disadvantageous for general users.

Conventional shading removal methods have used a neural network including a global localization module, an appearance modeling module, and a semantic modeling module. The global positioning module detects the shaded area and acquires the location characteristics of the shaded area, and the appearance modeling module learns the features of the non-shaded area so that the output of the network and the labeling data (Ground Truth, GT) match in the non-shaded area. , The semantic modeling module restores the original content behind the shading. However, in this method, rather than directly outputting the background image with the shading removed, the ratio between the shading image and the background image must be output, and further, the background image must be obtained by dividing the shading image and the output of the network for each pixel. In addition to the introduction of a large output quantity, there is a possibility that output stability will be affected due to the problem of division by 0 in division.

Therefore, there is a need to propose an image processing technique that can effectively remove shadows while not causing significant side effects on the background layer, while also having fast execution speed and acceptable hardware configuration requirements.

Embodiments of the present application are, at least among the prior art, image processing methods, devices, storage media, and electronics to solve the technical problem of removing shaded areas while being prone to side effects on the background layer of the image and placing high demands on the hardware platform. Provides equipment.

According to one aspect of the embodiment of the present application, the method includes obtaining an image to be processed including a shaded area, and inputting the image to be processed into a trained neural network to obtain a deshading image, wherein the neural network is a two-stage subordinate. It includes a connected first-stage network and a second-stage network. The first-stage network receives the processed image and outputs the shaded area mask image, and the second-stage network simultaneously receives the processed image and the shaded area mask image and outputs the shaded area mask image. Provides an image processing method for output.

Optionally, the first stage network includes a first encoder and is connected to a first feature extraction module that extracts features of the image to be processed layer by layer to obtain a first set of feature data, and an output of the first feature extraction module, It includes a first decoder, and includes a shaded area estimation module that estimates a shaded area based on the first set of feature data and outputs a shaded area mask image.

Optionally, the second stage network includes a second encoder, is connected to the output of the first stage network, and receives the image to be processed, as well as receiving the shaded area mask image output from the first stage network to generate a second set of feature data. a second feature extraction module, and a result image output module connected to the output of the second feature extraction module, including a second decoder, and outputting a deshading image based on the second set of feature data.

Optionally, splicing the output of each layer of the first decoder or the second decoder to the output of the corresponding layer of the first encoder or the second encoder along the channel axis through a cross-layer connection, and A multi-scale pyramid pooling module is added to the cross-layer connection of the first encoder or the second encoder, and the multi-scale pyramid pooling module fuses features of different scales.

Optionally, after acquiring the image to be processed including the shaded area, the image processing method downsamples the image to be processed using the image pyramid algorithm, stores the gradient information of the image layer at each stage during downsampling, and performs Laplacian Forming a pyramid, feeding the image layer of the minimum size to the trained neural network to obtain an output image, performing reconstruction from low resolution to high resolution on the output image using the Laplacian pyramid, and producing a deshading image. Includes more to get.

Optionally, the image processing method further includes building an initial neural network, and training the initial neural network using sample data to obtain a trained neural network, wherein the sample data includes a real image and a synthetic shaded image. Then, a synthetic shaded image is synthesized from a simple shaded image and an unshaded image using an image synthesis method.

Optionally, compositing the composite shaded image with a simple shaded image and an unshaded image using an image fusion method may include obtaining a simple shaded image, acquiring a unshaded image, and combining the simple shaded image and the unshaded image. and obtaining a synthetic shaded image based on the

Optionally, compositing the composite shading image with a simple shading image and an unshading image using an image fusion method includes converting the simple shading image and obtaining a composite shading image based on the converted simple shading image and unshading image. Further comprising, the pixel value of the non-shaded area in the converted simple shaded image is collectively set to one fixed value, a, and the pixel value of the shaded area is a value between 0 and a, and a is defined. It is an integer.

Optionally, the initial neural network further includes a module that performs a type judgment on the sample data, and when it is determined that the sample data input to the initial neural network is a real-world image, the labeling data is a de-shaded image collected from the real-world scene, Based on the difference between the shaded image output from the initial neural network and the shaded image as labeling data, the parameters inside the second-stage network are adjusted, and the sample data input to the initial neural network is determined to be a synthetic shaded image. The labeling data includes unshaded images and simple shaded images collected from the real scene. Based on the difference between the shaded area mask image and the simple shaded image, the parameters inside the first stage network are adjusted, and the shading output from the initial neural network is adjusted. Based on the difference between the removed image and the unshaded image, the parameters inside the second-stage network are adjusted.

Optionally, when training the initial neural network using sample data, the loss function includes at least one of pixel loss, feature loss, structural similarity loss, adversarial loss, shaded edge loss, and shaded luminance loss.

Optionally, the pixel loss includes a pixel blocking loss, and if the absolute difference of the corresponding two pixels in the output image from the initial neural network and the label image is greater than a predetermined threshold, calculating the loss of the two pixels; If the absolute difference between the corresponding two pixels in the output image from the initial neural network and the label image is less than or equal to a predetermined threshold, the difference between the two pixels is ignored.

Optionally, the shade luminance loss is achieved by making the difference between the luminance of the area corresponding to the shaded area in the shaded area output from the neural network and the luminance of the shaded area in the input image to be processed greater than 0, thereby creating a shaded area removed image. The luminance of the area corresponding to the shaded area is improved.

Optionally, when the loss function includes shaded edge loss, the image processing method includes performing dilation processing on the shaded area mask image to obtain a dilated image, and performing erosion processing on the shaded area mask image to obtain an erosion image. It includes obtaining an image, and obtaining the difference set between the dilation image and the erosion image as a boundary area between shaded and non-shaded areas, and smoothing it using TVLoss.

According to another aspect of the embodiment of the present application, an image collection unit that acquires an image to be processed including a shaded area, and receives the image to be processed, and processes the image to be processed using a trained neural network to produce a deshaded image. A neural network includes a two-stage cascaded first-stage network and a second-stage network, where the first-stage network receives the image to be processed and outputs a shaded area mask image, and the second-stage network receives the image to be processed. and an image processing device that simultaneously receives a shaded area mask image and outputs a shaded area mask image.

Optionally, the first stage network includes a first encoder and is connected to a first feature extraction module that extracts features of the image to be processed layer by layer to obtain a first set of feature data, and an output of the first feature extraction module, It includes a first decoder, and includes a shaded area estimation module that estimates a shaded area based on the first set of feature data and outputs a shaded area mask image.

Optionally, the second stage network includes a second encoder, is connected to the output of the first stage network, and receives the image to be processed, as well as receiving the shaded area mask image output from the first stage network to generate a second set of feature data. a second feature extraction module, and a result image output module connected to the output of the second feature extraction module, including a second decoder, and outputting a deshading image based on the second set of feature data.

According to another aspect of the embodiment of the present application, a storage medium storing a program, which, when the program is executed, controls a device in which the storage medium is located to execute the image processing method according to any one of the above, is further provided. to provide.

According to another aspect of an embodiment of the present application, there is provided a processor, and a memory storing executable instructions of the processor, wherein the processor executes the executable instructions to process the image according to any one of the above. An electronic device configured to perform the method is further provided.

This application proposes a shadow removal method that is fast, effective, and applicable to mobile terminals such as mobile phones, identifies the characteristics of the physical phenomenon called shadow, synthesizes training data with excellent realism, and reduces various losses. By training in combination with functions or effective network structures and modules, effective shadow removal is implemented, and in response to the characteristics of high resolution of images captured through mobile terminals such as mobile phones, this application proposes downsampling technology and network pruning. Using the technology, it is still possible to achieve fast processing speeds for high-resolution images.

The drawings described herein are intended to aid further understanding of the present application and constitute a part of the present application, and the exemplary embodiments of the present application and their descriptions are intended to interpret the present application and do not inappropriately limit the present application. It's not like that.
1 is a flowchart of a selective image processing method according to an embodiment of the present application.
Figure 2 is a structural diagram of an optional neural network according to an embodiment of the present application.
Figure 3 is a training flowchart of an optional neural network according to an embodiment of the present application.
Figure 4 is a flowchart of a selective image synthesis method according to an embodiment of the present application.
Figures 5(a) and 5(b) are comparative diagrams of the effect of implementing shadow removal using the image processing method of the embodiment of the present application.
Figure 6 is a structural block diagram of an optional image processing device according to an embodiment of the present application.

Hereinafter, in order for those skilled in the art to more easily understand the technical draft of the present application, the technical draft of the embodiments of the present application will be clearly and completely described with reference to the drawings of the embodiments of the present application, and the described embodiments are included in some embodiments of the present application. It is clear that this is limited and not all examples. Based on the embodiments of this application, all other embodiments obtained by a person skilled in the art without creative efforts shall fall within the protection scope of this application.

It should be noted that terms such as “first” and “second” in the specification, claims, and drawings of this application are intended to distinguish similar objects and are not necessarily intended to describe a specific order or priority. Be careful. It is to be understood that the orders used are interchangeable under appropriate circumstances, such that the embodiments of the application described herein can be practiced in orders other than those shown or described herein. Additionally, the terms “comprise” or “have” and any variations thereof are intended to encompass the non-exclusive “including,” e.g., a process, method, system, product or device that includes a series of steps or units. It is not necessarily limited to steps or units explicitly listed, and may include other steps or units that are not explicitly listed or that are unique to their processes, methods, products, or devices.

Hereinafter, a flowchart of an optional image processing method of an embodiment of the present application will be described. The steps shown in the flowcharts of the drawings may be executed by a computer system as a set of computer-executable instructions, and although the flowcharts show a logical order, in some cases the steps shown or described may be executed in a different order than here. It should be noted that this happens.

Referring to FIG. 1, it is a flowchart of a selective image processing method according to an embodiment of the present application. As shown in FIG. 1, the image processing method includes the following steps.

S100: Acquire an image to be processed including a shaded area.

S102: The image to be processed is input to a trained neural network to obtain a shaded image. Here, the neural network includes a two-stage cascaded first-stage network and a second-stage network, the first-stage network receives the image to be processed and outputs a shaded area mask image, and the second-level network receives the image to be processed and the shaded area mask image. simultaneously receives and outputs a shadow-removed image.

Through the above image processing method, an accurate shaded area boundary can be obtained, and the obtained shaded removed image can smoothly transition between shaded and non-shaded.

In one optional embodiment, as shown in FIG. 2, the neural network includes a two-stage cascade-connected first-stage network 20 and a second-stage network 22, where the first-stage network includes a first feature extraction module 200 and a shaded area. It includes an estimation module 202, and the second stage network includes a second feature extraction module 204 and a result image output module 206. Here, the first feature extraction module 200 includes a first encoder, extracts features of the image to be processed layer by layer to obtain a first set of feature data, and the shaded area estimation module 202 outputs the output of the first feature extraction module 200. connected, comprising a first decoder, estimating a shaded area based on the first set of feature data and outputting a shaded area mask image, the second feature extraction module 204 comprising a second encoder, and It is connected to the output, and receives the image to be processed, as well as receiving the shaded area mask image output from the first stage network to obtain a second set of feature data, and the resulting image output module 206 outputs the output of the second feature extraction module 204. It is connected, includes a second decoder, and outputs a shadow removal image based on the second set of feature data. The shadow removal effect can be strengthened through a two-stage cascaded neural network. In one optional embodiment, the first-level network and the second-level network have the same structure except that the number of input channels is different, and may be built based on, for example, a classical partition network UNet.

The output of each layer of the two encoders is spliced to the output of the corresponding layer of the two decoders along the channel axis through cross-layer connection. A multi-scale pyramid pooling module is added to the cross-layer connection of the encoder and decoder. The multi-scale pyramid pooling module includes a plurality of different kernel size pooling layers, convolution layers and interpolation upsampling layers, first extracting features of different scales through the pooling layer, and then through the convolution layer to low-level and/or Alternatively, high-level features are extracted, and further, the outputs of the corresponding layers of the encoder and decoder are adjusted to the same size through an interpolation upsampling layer, and finally, spliced into one feature along the channel axis. Because the degree and area of shading influence vary greatly depending on the image, determination of the shading area not only requires reference to local texture features, but also considers global semantic information. By fusing the features of different scales, the multi-scale pyramid pooling module can enhance the universality of the network and achieve a good effect of the network on shading images of different areas and degrees.

To improve the execution speed of the model on the device, you can prune the model and replace the encoder's convolution layer with a grouping convolution so that each convolution kernel convolves only one channel. This reduces the amount of model calculations and improves processing speed.

In order to better suppress covariance drift and enhance the fitting ability of the network to the data, an instance normalization layer is added after the convolutional layer of the encoder and decoder to normalize the features, thereby improving the effect of shading removal. .

If the image resolution of the image to be processed is high or the amount of data is large, if the image to be processed is directly transmitted to the trained neural network, video memory overflow may occur or the processing time may be too long, affecting the user experience. There is a possibility, and a general interpolation scaling algorithm can be used to solve this problem, but image information is likely to be lost, and the generated image cannot be perfectly enlarged to the original image.

Considering the characteristic that shaded areas usually do not have significant gradient information, in one alternative embodiment, an image pyramid algorithm is used to first downsample the image to be processed, and during downsampling, the image layer at each stage is The gradient information is stored to form a Laplacian pyramid, and then the image layer with the smallest pyramid size is sent to the trained neural network to obtain an output image. Finally, the output image can be reconstructed using the Laplacian pyramid, and the shaded area is Because the gradient information of is weak, even if some of the gradient information of the image to be processed is restored in the reconstruction process, the shadow removal effect is not affected. By performing image reconstruction using the gradient information of each image layer stored during downsampling, shading can be removed without affecting image resolution. By introducing downsampling and image reconstruction, on the one hand, the image processing speed is guaranteed, and on the other hand, the quality before and after image processing is not affected, making it possible to process high-resolution images on devices with low computing power such as mobile phones. It is advantageous.

As shown in Figure 3, in order to obtain a trained neural network, the image processing method further includes the following steps.

S300: Build an initial neural network.

S302: Train the initial neural network using sample data to obtain a trained neural network. Here, the sample data includes a real image and a synthetic shaded image, and the synthetic shaded image is synthesized into a simple shaded image and an unshaded image.

Since there are many types of shading in the images that users always capture, if you distinguish them from the edges of the shading, you can distinguish between clear and sharp shading edges captured when the distance from the light source to the background is close, and sharp shading edges captured when the distance from the light source to the background is long. In addition, if the light source has a different color (e.g., reddish-yellow warm light, blue-colored cool light or daylight), the shading also has a different color. Therefore, considering these characteristics, sample data for training the initial neural network plays a very important role in the entire image processing method, and there are mainly two methods for acquiring sample data: real scene collection and image synthesis.

In a method using real scene collection, the collector selects the corresponding light environment and shooting target according to the type of scene (e.g., different lighting scenes such as warm color light, cold color light, daylight, etc.), and uses a shooting device such as a mobile phone or camera. Fix the camera on a tripod, adjust the appropriate lighting direction and focal length, use the palm, mobile phone or other common objects as a shield to block the light, form a shadow on the object to be photographed and shoot to obtain a shaded image, then remove the shield and start again. By taking pictures, a shadow-free background image is obtained, thereby obtaining paired sample data.

However, real-world collection is usually difficult to ensure high quality of sample data, and on the other hand, due to changes in light rays due to occlusion, differences in luminance and color occur in unshaded areas in the background and shaded images, and at the same time, the shaded images On the one hand, it is difficult to completely match the background image, and on the other hand, changes in light rays or changes in focus produce noise in the shaded image and background image, both of which greatly affect the training of the network.

In response to this, a synthetic shading image that is almost close to the truth may be generated using an image synthesis method and used for training a neural network.

In one alternative embodiment, the image compositing method includes the following steps.

S400: Acquire a simple shaded image.

In one optional embodiment, the data collector, in a preset light environment, lays a sheet of blank paper flat on a table, blocks the light with the palm of his hand, a cell phone, or other common object, and leaves a simple shaded image S on the blank paper, All or part of the area of S is a shaded area.

When acquiring a simple shaded image, there is a possibility that the non-shaded area on the white paper will not be displayed as simple white, so the boundary between the non-shaded area and the shaded area will not be sufficiently clear. Therefore, in another optional embodiment, the simple shading image may be further converted, for example, S'=min(a,S/mean(S)*a), where a is a positive integer. Through the above conversion, the pixel values of the non-shaded area in the converted simple shaded image can be collectively set to one fixed value a (e.g., 255), and the pixel value of the shaded area is a value between 0 and a. , there is a relatively clear boundary between the non-shaded area and the shaded area in the simple shaded image.

S402: Acquire an unshaded image.

In one alternative embodiment, the data collector takes unshaded images B of each type of subject in the same light environment.

S404: Based on the simple shaded image and the unshaded image, a composite shaded image is obtained.

In one alternative embodiment, the simple shaded image S (or the converted simple shaded image S') and the unshaded image B are multiplied pixel-by-pixel to obtain a composite shaded image.

This image synthesis method takes into account the attenuation effect of shadows on light rays and can handle shadows with gentle transitions between edges well, providing an excellent sense of realism.

The sample data is mixed data including a real image and a synthetic shaded image, and the initial neural network further includes a module that performs type judgment on the sample data, so it may have been determined that the sample data input to the initial neural network was a real image. In this case, the labeling data (Ground Truth, GT) is a deshaded image collected from the real scene, and since the mask image of the shaded area of the real image cannot be adjusted, the deshaded image output from the initial neural network and the deshaded image as the labeling data GT Based on the difference between the two, the parameters inside the second-stage network 22 are adjusted, and when it is determined that the sample data input to the initial neural network is a synthetic shaded image, the labeling data (ground truth, GT) is the unshaded image collected from the real scene. It includes an image and a simple shaded image, and adjusts the parameters inside the first stage network 20 based on the difference between the shaded area mask image and the simple shaded image, and the deshaded image output from the initial neural network and the unshaded image as labeling data. Based on the differences between images, the parameters inside the second stage network 22 can be adjusted. Through training using mixed data as sample data, its accurate mask can be obtained for smooth transition shading, ensure the quality of mask segmentation, and improve the effect of shading removal.

In one optional embodiment, the method of acquiring sample data includes random flipping, rotation, and color temperature on the obtained sample data to enrich the sample data and increase the robustness of the network. It may further include performing one or more processing such as adjustment, channel swapping, addition of random noise, etc.

In one alternative embodiment, when training an initial neural network, the loss function includes at least one of a pixel loss, a feature loss, a structural similarity loss, and an adversarial loss.

The pixel loss function is a function that measures the similarity of two images in the pixel dimension of the images, and mainly includes image pixel value loss and gradient loss. In this embodiment, it mainly refers to the weighted sum of the mean square error of the compared pixel values of the output image and the label image from the initial neural network and the L1 norm error of the gradient of the two images. Pixel loss monitors the training process at the pixel level to ensure that the pixel value of each pixel in the output image and label image from the initial neural network is as close as possible. In order to guide the initial neural network to pay attention to the differences between the shadow layer and the background layer in the shaded region rather than the noise in the entire image, in one alternative embodiment, a pixel blocking loss is introduced to block the pixel loss. That is, the loss of two pixels is calculated only when the absolute difference between the two pixels is greater than a predetermined threshold, otherwise, the difference between the two pixels is ignored. After adding pixel blocking loss, noise in the image can be suppressed by guiding the network to pay attention to shaded areas. Not only does the effect of shade removal improve, but the convergence speed of the network is also significantly accelerated.

The feature loss is mainly a weighted sum of the L1 norm errors of the corresponding features of the input image and label image to the initial neural network. In one optional embodiment, a VGG19 network pre-trained on the ImageNet data set is used as a feature extractor, and the output image and label image from the initial neural network are respectively fed to the feature extractor to obtain features of each layer of VGG19. After that, the L1 norm errors of the corresponding features of the input image and the label image are calculated and weighted. The features of each layer of VGG19 are not sensitive to image details or noise and have good semantic characteristics, so even if there are defects such as noise or misalignment in the input image and output image, feature loss still accurately generates the effective shaded area difference. It can compensate for the lack of sensitivity of pixel loss to noise and has good stability.

The structural similarity loss function is a function that measures the similarity of two images based on the global characteristics of the images. In this embodiment, it mainly refers to the global luminance and contrast differences between the output image and the label image from the initial neural network, and by adding the corresponding loss function, the color deviation of the output of the network is effectively suppressed, and the color deviation of the entire image is effectively suppressed. Quality can be improved.

The adversarial loss is mainly the output result of the identifier and the loss value of the true classification of the output image. In the later stages of training, as the difference between the output image from the initial neural network and the label image becomes smaller, the effects of pixel loss, feature loss, and structural similarity loss gradually become smaller, and the convergence of the network slows down. At this time, to support network training, one identifier network is trained synchronously. First, the output image and label image from the initial neural network are sent to the identifier, the identifier determines whether the output image is a label image, the loss is calculated based on the output result of the identifier and the true classification of the output image, and the identifier is The parameters of are updated, and then the identification result of the identifier for the output image is set as the loss of the reality degree of the output image, and the parameters of the identifier are updated with the loss. If the discriminator cannot distinguish between the output image from the initial neural network and the label image, it indicates that training has ended. Adversarial loss can effectively eliminate image side effects caused by network processing (e.g., color mismatch problem in shaded and non-shaded areas, shade residual problem, etc.) and improve the degree of reality of images output from the network.

Threshold cutoff loss. Due to the influence of lighting, paired data collected in real scenes may have subtle luminance differences or color changes even in non-shaded areas, but these differences are within an acceptable range for the user and do not need to be processed. Therefore, in order to prevent the network from paying attention to these global small differences during training, the method introduces a threshold blocking loss, that is, it only detects the difference between the output of the network and GT if it is greater than a certain threshold. It is calculated intensively in the gradient of the total loss calculation parameter, otherwise the loss is considered 0. The loss function allows for small differences between the network's output and GT, and effectively improves the network's removal ability for relatively distinct shading by shifting the center of network learning to areas with large differences.

Shaded edge loss. First, dilation processing is performed on the shaded area mask image to obtain a dilation image, then erosion processing is performed on the shaded area mask image to obtain an erosion image, and then the dilation image and the erosion image are obtained. By obtaining the difference set of the image as the boundary area between shaded and non-shaded areas and smoothing it using TVLoss, it is possible to effectively transition between shaded and non-shaded areas.

Shade luminance loss is caused by making the difference between the luminance of the area corresponding to the shaded area in the shaded area output from the neural network and the luminance of the shaded area in the input image to be processed greater than 0, Improves the luminance of the area corresponding to the shaded area.

In one alternative embodiment, the background layer output module of the initial neural network uses the weighted sum of all losses above as the total loss, and also uses the Wassertein adversarial generative network as the adversarial loss.

The network structure extracts global features and local features of the input image to improve the degree of shading removal and protects non-shaded areas from causing side effects.

Figures 5(a) and 5(b) are comparative diagrams of processing effects realized using the image processing method of the embodiment of the present application, Figure 5(a) is a target image including shading, and Figure 5(a) is a target image including shading. b) is a shadow removal image after processing through an image processing method, and as can be seen from the comparison of the two images, the image processing method according to the present application can effectively remove shadows while not causing significant side effects to the background layer. .

The neural network structure and loss function used in the embodiments of the present application may be applied to application scenes such as shadow removal, rain removal, and fog removal, and are mainly used to process high-resolution images captured through mobile terminals such as mobile phones. , the same applies to processing images of various resolutions on PCs or other embedded devices.

According to another aspect of the embodiment of the present application, comprising a processor and a memory for storing executable instructions of the processor, wherein the processor is configured to execute the image processing method of any one of the above through executing the executable instructions. Provides more electronic devices.

According to another aspect of the embodiment of the present application, a storage medium storing a program is further provided, which controls a device in which the storage medium is located to execute the image processing method of any one of the above when the program is executed.

According to another aspect of the embodiment of the present application, an image processing device is further provided. Referring to FIG. 6, it is a structural block diagram of an optional image processing device according to an embodiment of the present application. As shown in FIG. 6, the image processing device 60 includes an image collection unit 600 and a processing unit 602.

Hereinafter, each unit included in the image processing device 60 will be described in detail.

The image collection unit 600 acquires an image to be processed including a shaded area.

The processing unit 602 receives the image to be processed, and processes the image to be processed using a trained neural network to obtain a deshading image, where the neural network includes a first-stage network and a second-stage network that are two-stage cascade-connected, The image to be processed and the output image from the first-stage network are simultaneously input to the second-stage network.

In one alternative embodiment, the structure of the neural network is as shown in Figure 2 and the related art herein, and the description is not duplicated here.

The numbers of the above examples in this application are for descriptive purposes only and do not indicate superiority or inferiority of the examples.

In the above-mentioned embodiments of the present application, the techniques for each embodiment are all focused on each other, and parts that are not described in detail in one embodiment may refer to related techniques in other embodiments.

It should be understood that in various embodiments according to the present application, the disclosed technical content may be implemented in other ways. Here, the embodiments of the device described above are merely illustrative, and for example, the division of the units may be a division of logical functions, and there may be other division methods in actual implementation, for example, a plurality of units or components may be combined or used in other ways. It may be integrated into the system, or some features may be omitted or not implemented. In addition, the mutual coupling or direct coupling or communication connection disclosed or reviewed may be an indirect coupling or communication connection through some interface, unit or module, and may be electrical or other forms.

The unit described as a separate member may or may not be physically separate, and the member disclosed as a unit may or may not be a physical unit, i.e., may be located in one location. Alternatively, it may be distributed across multiple units. Depending on actual demand, some or all of the units may be selected to implement the purpose of the technical plan of this embodiment.

Additionally, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may physically exist independently, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a single computer-readable storage medium. Based on this understanding, the essential or contributing part of the technical plan of this application, or all or part of the technical plan, may be implemented in the form of a software product, and the computer software product may be implemented on a single computer device (personal computer). , a server or a network device, etc.) are stored in a storage medium containing some instructions for executing all or part of the steps of the method described in each embodiment of the present application. The above-described storage media includes various types of storage media that can store program code, such as USB disks, read-only memory (ROM), random access memory (RAM), removable hard disks, magnetic disks, or optical disks. Includes various media.

The above is only a preferred embodiment of the present application, and those skilled in the art may make some improvements and modifications without departing from the principles of the present application, and these improvements and modifications should also be considered within the scope of protection of the present application. Be careful.

Claims (18)

1. As an image processing method,
acquiring an image to be processed including a shaded area, and
Including inputting the image to be processed into a trained neural network to obtain a deshading image,
The neural network includes a two-stage cascade-connected first-stage network and a second-stage network, wherein the first-stage network receives the image to be processed and outputs a shaded area mask image, and the second-stage network receives the image to be processed and the image to be processed. An image processing method for simultaneously receiving a shaded area mask image and outputting the shadow removal image. According to paragraph 1,
The first stage network is
A first feature extraction module comprising a first encoder and extracting features of the image to be processed layer by layer to obtain a first set of feature data, and
and a shaded area estimation module connected to the output of the first feature extraction module, including a first decoder, and configured to estimate a shaded area based on the first set of feature data and output a shaded area mask image. Image processing method. According to paragraph 1,
The second tier network is
A second feature comprising a second encoder, connected to the output of the first stage network, receiving an image to be processed and receiving a shaded area mask image output from the first stage network to obtain a second set of feature data. an extraction module, and
An image processing method comprising a result image output module coupled to the output of the second feature extraction module, comprising a second decoder, and outputting the deshading image based on the second set of feature data. . According to paragraph 2,
Splicing the output of each layer of the first decoder or the second decoder to the output of a corresponding layer of the first encoder or the second encoder along a channel axis through cross-layer connection, and An image processing method comprising adding a multi-scale pyramid pooling module to a cross-layer connection of a second decoder and the first encoder or the second encoder, wherein the multi-scale pyramid pooling module fuses features of different scales. According to paragraph 1,
After acquiring the image to be processed including the shaded area, the image processing method
Downsampling the image to be processed using an image pyramid algorithm, and storing gradient information of each image layer during downsampling to form a Laplacian pyramid,
Obtaining an output image by feeding the image layer of the minimum size to a trained neural network, and
An image processing method further comprising performing reconstruction from low resolution to high resolution on the output image using a Laplacian pyramid, and obtaining the shadow removal image. According to paragraph 1,
building an initial neural network, and
Further comprising training the initial neural network using sample data to obtain the trained neural network,
The image processing method characterized in that the sample data includes a real image and a synthetic shaded image, and the synthetic shaded image is synthesized into a simple shaded image and an unshaded image using an image synthesis method. According to clause 6,
Combining the synthetic shaded image with a simple shaded image and an unshaded image using an image synthesis method is
acquiring simple shaded images;
acquiring a shadeless image, and
An image processing method comprising obtaining the composite shaded image based on the simple shaded image and the unshaded image. In clause 7,
Compositing the synthetic shaded image with a simple shaded image and an unshaded image using an image synthesis method includes converting the simple shaded image and obtaining the synthesized shaded image based on the converted simple shaded image and the unshaded image. Contains more,
The pixel values of the non-shaded area in the converted simple shaded image are collectively set to one fixed value, a, and the pixel values of the shaded area are set to a number between 0 and a, where a is a positive integer. Image processing method. In clause 7,
The initial neural network further includes a module that performs type judgment on sample data,
When it is determined that the sample data input to the initial neural network is a real image, the labeling data is a deshaded image collected from the real scene, and between the deshaded image output from the initial neural network and the deshaded image as the labeling data. Based on the difference, adjust the parameters inside the second stage network,
When it is determined that the sample data input to the initial neural network is a synthetic shaded image, the labeling data includes the unshaded image and the simple shaded image collected from the real scene, and between the shaded area mask image and the simple shaded image. Based on the difference, adjusting the parameters inside the first-stage network, and adjusting the parameters inside the second-stage network based on the differences between the de-shaded image output from the initial neural network and the unshaded image. Image processing method. According to clause 6,
When training the initial neural network using sample data, the loss function includes at least one of pixel loss, feature loss, structural similarity loss, adversarial loss, shade edge loss, and shade luminance loss. . According to clause 10,
The pixel loss includes pixel blocking loss,
If the absolute difference between the corresponding two pixels in the output image from the initial neural network and the label image is greater than a predetermined threshold, the loss of the two pixels is calculated, and the output image from the initial neural network and the label are calculated. An image processing method, wherein when the absolute difference between two corresponding pixels in an image is less than or equal to the predetermined threshold, the difference between the two pixels is ignored. According to clause 10,
The shade luminance loss is such that the difference between the luminance of an area corresponding to the shaded area in the shaded area output from the neural network and the luminance of the shaded area in the input image to be processed is greater than 0, An image processing method characterized by improving the luminance of an area corresponding to the shaded area in the shaded area. According to clause 10,
When the loss function includes the shaded edge loss, the image processing method includes performing dilation processing on the shaded area mask image to obtain a dilated image, and performing erosion processing on the shaded area mask image to obtain an erotic image. An image processing method comprising obtaining a dilation image, obtaining a set of differences between the dilatation image and the erosion image as a boundary area between shaded and non-shaded areas, and smoothing it using TVLoss. an image collection unit that acquires an image to be processed including a shaded area, and
A processing unit that receives an image to be processed and processes the image to be processed using a trained neural network to obtain a deshading image,
The neural network includes a two-stage cascade-connected first-stage network and a second-stage network, wherein the first-stage network receives the image to be processed and outputs a shaded area mask image, and the second-stage network receives the image to be processed and the image to be processed. An image processing device that simultaneously receives a shaded area mask image and outputs the shaded area mask image. According to clause 14,
The first stage network is
A first feature extraction module comprising a first encoder and extracting features of the image to be processed layer by layer to obtain a first set of feature data, and
and a shaded area estimation module connected to the output of the first feature extraction module, including a first decoder, and configured to estimate a shaded area based on the first set of feature data and output a shaded area mask image. An image processing device that uses According to clause 14,
The second tier network is
A second feature comprising a second encoder, connected to the output of the first stage network, receiving an image to be processed and receiving a shaded area mask image output from the first stage network to obtain a second set of feature data. an extraction module, and
and a result image output module coupled to the output of the second feature extraction module, including a second decoder, and outputting a deshading image based on the second set of feature data. A storage medium in which a program is stored, wherein when the program is executed, a device in which the storage medium is located is controlled to execute the image processing method according to any one of claims 1 to 13. As an electronic device,
processor, and
Includes a memory that stores executable instructions of the processor,
The electronic device, wherein the processor is configured to execute the image processing method according to any one of claims 1 to 13 through executing the executable instructions.