WO2021189321A1 - 一种图像处理方法和装置 - Google Patents

一种图像处理方法和装置 Download PDF

Info

Publication number
WO2021189321A1
WO2021189321A1 PCT/CN2020/081191 CN2020081191W WO2021189321A1 WO 2021189321 A1 WO2021189321 A1 WO 2021189321A1 CN 2020081191 W CN2020081191 W CN 2020081191W WO 2021189321 A1 WO2021189321 A1 WO 2021189321A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
reflection
judgment
input image
input
Prior art date
Application number
PCT/CN2020/081191
Other languages
English (en)
French (fr)
Inventor
王叶斌
赵杨
皮志明
黄枭
陈迪
迟显军
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN202080001382.3A priority Critical patent/CN113728355A/zh
Priority to PCT/CN2020/081191 priority patent/WO2021189321A1/zh
Publication of WO2021189321A1 publication Critical patent/WO2021189321A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • This application relates to the field of image processing technology, and in particular to an image processing method and device.
  • Artificial intelligence is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results.
  • artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a similar way to human intelligence.
  • Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.
  • Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theories.
  • neural networks for example, deep neural networks
  • various media signals such as images, videos, and voices in recent years.
  • AI algorithms can be used to eliminate reflections on the images to reduce the impact of reflections on the picture effect.
  • the existing reflection elimination technology is mainly realized by generating adversarial networks (GAN).
  • GAN adversarial networks
  • the original image with reflection is input into the trained reflection elimination generating network, and the network infers to generate a non-reflective image.
  • the GAN neural network algorithm only has the means to eliminate the reflection, but it does not have the means to identify the reflection. After the elimination of the reflection of the non-reflective image, the image quality will deteriorate.
  • the present application provides an image processing method and device, which can improve the effect of removing light reflection of an image.
  • an image processing device including: a first reflection elimination module, configured to perform reflection elimination processing on an input image to obtain a first image; and a reflection judgment module, configured to obtain a first image based on the first image and the input image Whether the input image has a reflection judgment result; the second reflection elimination module is used to obtain an output image according to the judgment result.
  • the image processing device can judge whether the input image is reflective according to the result of the preliminary reflection elimination, and then perform the reflection elimination processing on the original image with the reflection as the result of the judgment, so as to realize the automatic recognition and elimination of the reflection image.
  • the device may further include a pre-processing module for pre-processing the original image to obtain an input image, the size of the original image is larger than the size of the input image.
  • a pre-processing module for pre-processing the original image to obtain an input image, the size of the original image is larger than the size of the input image.
  • the original image may be used as the input image.
  • the original image is preprocessed to realize the recognition and elimination of the reflection of the original image with a larger size.
  • the second light reflection elimination module is specifically configured to obtain the original image as the output image when the judgment result is that the input image has no light reflection.
  • the original image can be directly output without unnecessary processing of the original image, which avoids the deterioration of the image quality after the reflection of the non-reflective image is eliminated.
  • the second reflection elimination module is specifically configured to generate a small image of the reflective layer according to the first image and the input image, and according to the small reflective layer. Figure and original image to obtain the output image.
  • the second reflection elimination module is specifically configured to generate a small image of the reflective layer according to the first image and the input image, Up-sampling the small image of the reflective layer to obtain the reflective layer, and obtain the output image based on the reflective layer and the original image.
  • the reflective layer is obtained by sampling the small image of the reflective layer, and then the output image is obtained according to the original image and the reflective layer, and finally an output image with the same size and resolution as the original image can be obtained to eliminate the reflection, which improves the output The quality of the image.
  • the reflection judgment module includes a reflection judgment model.
  • the reflection judgment model is a neural network with the first image and the input image as inputs.
  • the output of the reflection judgment model indicates the confidence that the input image has reflection, or the output is the judgment result.
  • the reflection judgment model is a neural network based on the Markov discriminator, and is obtained by training based on a first sample set, where the first sample set includes: original image samples, and original image samples after reflection Whether the erased image sample and the original image sample have reflective labels.
  • the light reflection judgment module includes a light reflection layer intensity judgment unit, the light reflection layer intensity judgment unit is used to generate a small light reflection layer image according to the input image and the first image; the light reflection layer intensity judgment unit is also used to calculate the light reflection layer small image includes The gray value of multiple pixels, and the judgment result is determined according to the gray value of multiple pixels.
  • the light-reflecting layer intensity judgment unit determines the judgment result according to the weighted result of at least one or more of the following information: the maximum value of the gray value of the multiple pixel points, and the gray value is greater than the first The intensity of at least one pixel with a threshold value and the area of at least one pixel with a gray value greater than a second threshold; wherein the intensity of at least one pixel with a gray value greater than the first threshold is that the gray value is greater than the first The sum of the gray values of the pixels of the threshold, or the intensity of at least one pixel with the gray value greater than the first threshold is the average of the gray values of the pixels with the gray value greater than the first threshold; The area of at least one pixel with a gray value greater than the second threshold is the number of pixels with a gray value greater than the second threshold.
  • the reflection judgment module includes a reflection judgment model, a reflection layer intensity judgment unit, and a joint judgment unit.
  • the reflection judgment model is a neural network that uses the first image and the input image as inputs, and the output of the reflection judgment model indicates whether the input image is There is a first judgment value for light reflection;
  • the light reflection layer intensity judgment unit calculates the gray value of the difference between the input image and the first image, and outputs a second judgment value indicating whether the input image has light reflection according to the gray value;
  • the joint judgment unit determines the judgment result according to the first judgment value and the second judgment value.
  • multiple models or modules can be used to determine whether the original image is reflective, which improves the accuracy of the determination result.
  • an image processing method which includes: performing reflection elimination processing on an input image to obtain a first image; judging whether the input image has reflection according to the input image and the first image; and according to whether the input image has reflection To obtain the output image.
  • the image processing device can judge whether the input image is reflective according to the result of the preliminary reflection elimination, and then perform the reflection elimination processing on the original image with the reflection as the result of the judgment, so as to realize the automatic recognition and elimination of the reflection image.
  • the original image may be preprocessed to obtain the input image, and the size of the original image is larger than the size of the input image.
  • the original image is preprocessed to realize the recognition and elimination of the reflection of the original image with a larger size.
  • obtaining the output image according to whether the input image has reflections includes: when the input image has no reflections, the output image is the original image.
  • the original image can be directly output without unnecessary processing of the original image, which avoids the deterioration of the image quality after the reflection of the non-reflective image is eliminated.
  • obtaining the output image includes: when the input image is reflective and the input image is an original image, the method further includes: generating a small reflective layer image based on the input image and the first image; The output image is obtained according to the small image of the reflective layer and the original image.
  • obtaining the output image according to whether the input image is reflective or not includes: when the input image is reflective and the input image is an image preprocessed by the original image, the method further includes: according to the input image and the first image Generate a small image of the reflective layer; up-sample the small image of the reflective layer to obtain a reflective layer; and obtain the output image according to the reflective layer and the original image.
  • the reflective layer is obtained by sampling the small image of the reflective layer, and then the output image is obtained according to the original image and the reflective layer, and finally an output image with the same size and resolution as the original image can be obtained to eliminate the reflection, which improves the output The quality of the image.
  • judging whether the input image is reflective according to the input image and the first image includes: judging whether the input image is reflective according to a reflective judgment model; the reflective judgment model is a neural network that uses the first image and the input image as inputs The output of the reflection judgment model indicates the confidence that the input image has reflection, or the output is the judgment result of whether the input image has reflection.
  • the reflection judgment model is a neural network based on the Markov discriminator, and is obtained by training based on a first sample set, where the first sample set includes: original image samples, and original image samples after reflection Whether the erased image sample and the original image sample have reflective labels.
  • judging whether the input image is reflective according to the input image and the first image includes: judging whether the input image is reflective according to a reflective layer intensity judging unit; the reflective layer intensity judging unit is used to determine whether the input image is reflective A small image of the reflective layer is generated; the intensity determination unit of the reflective layer is also used to calculate the gray values of multiple pixels included in the small image of the reflective layer, and determine the judgment result according to the gray values of the multiple pixels.
  • the reflective layer judgment module determines the judgment result according to the weighted result of at least one or more of the following information: the maximum value among the gray values of the multiple pixels, and the gray value is greater than the first threshold
  • the sum of the gray values of the pixels, or the intensity of at least one pixel with the gray value greater than the first threshold is the average value of the gray values of the pixels with the gray value greater than the first threshold; the gray
  • the area of at least one pixel with a value greater than the second threshold is the number of pixels with a gray value greater than the second threshold.
  • judging whether the input image is reflective based on the input image and the first image includes: judging whether the input image is reflective according to a reflective judgment model, a reflective layer strength judging unit, and a joint judgment unit; the reflective judging model is based on the first A neural network with an image and an input image as the input, the output of the reflection judgment model is the first judgment value indicating whether the input image has reflection; the reflection layer intensity judgment unit calculates the gray value of the difference between the input image and the first image, and according to The gray value output indicates the second judgment value of whether the input image has reflection; the joint judgment unit determines the judgment result according to the first judgment value and the second judgment value.
  • multiple models or modules can be used to determine whether the original image is reflective, which improves the accuracy of the determination result.
  • a computer-readable medium stores program code for device execution, and the program code includes image processing for executing the second aspect or any one of the implementation manners of the second aspect method.
  • a computer program product includes: computer program code, which when the computer program code runs on a computer, causes the computer to execute the second aspect or any one of the implementation manners of the second aspect Image processing methods in.
  • a chip in a fifth aspect, includes a processor and a data interface.
  • the processor reads instructions stored in a memory through the data interface, and executes the second aspect or any one of the implementation manners in the second aspect. Image processing method.
  • the chip may further include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory, and when the instructions are executed, the processor is used to execute the second aspect or the second aspect The image processing method in any one of the implementations.
  • a device including: a processor and a memory, the memory is used to store the computer program code, and when the computer program code runs on the processor, the device executes the second aspect or An image processing method in any implementation of the second aspect.
  • FIG. 1 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present application
  • Figure 2 is a schematic diagram of an application scenario provided by an embodiment of the present application.
  • FIG. 3 is a schematic diagram of another application scenario provided by an embodiment of the present application.
  • FIG. 4 is a schematic diagram of a system architecture provided by an embodiment of the present application.
  • FIG. 5 is a schematic diagram of the chip hardware structure provided by an embodiment of the present application.
  • FIG. 6 is a schematic diagram of the software architecture of the reflection detection and removal system provided by an embodiment of the present application.
  • FIG. 7 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of a reflection judgment model provided by an embodiment of the present application.
  • FIG. 9 is a schematic flowchart of another image processing method provided by an embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of a reflection intensity judgment model provided by an embodiment of the present application.
  • FIG. 11 is a schematic flowchart of another image processing method provided by an embodiment of the present application.
  • FIG. 12 is a schematic diagram of the hardware structure of an image processing device provided by an embodiment of the present application.
  • FIG. 13 is a schematic diagram of the hardware structure of another image processing device provided by an embodiment of the present application.
  • FIG. 14 is a schematic diagram of the hardware structure of another image processing apparatus provided by an embodiment of the present application.
  • Fig. 1 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present application.
  • the image processing method implemented in this application can be applied to a smart terminal.
  • the original image taken by the camera in the smart device can be image processed to obtain a high-definition large image of the original image after removing the reflections in the screen. , Thereby improving the quality of image display.
  • the above-mentioned smart terminal may be mobile or fixed.
  • the smart terminal may be a mobile phone with image processing function, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer, LC), personal digital assistant (PDA), personal computer (PC), camera, video camera, smart watch, augmented reality (AR)/virtual reality (VR), wearable Wearable device (WD) or an autonomous vehicle, etc., which are not limited in the embodiment of the present application.
  • TPC tablet personal computer
  • PDA personal digital assistant
  • PC personal computer
  • camera video camera
  • smart watch augmented reality
  • AR augmented reality
  • VR virtual reality
  • WD wearable Wearable device
  • an autonomous vehicle etc.
  • Application scenario 1 Smart terminal camera field
  • the image processing method of the embodiment of the present application can be applied to the shooting of a smart terminal device (for example, a mobile phone).
  • a smart terminal device for example, a mobile phone.
  • the image method of the embodiment of the present application that is, the method of removing reflection, an output image with higher picture quality and with the reflection area removed from the original image can be obtained.
  • the image processing method of the embodiments of the present application may be used to perform down-sampling processing on the original image obtained when the smart terminal is taking a real-time photograph, and the original image is processed into a smaller image; the processed small image Perform reflection elimination processing, compare the original small image and the small image after the reflection elimination, to determine whether the original image has a reflection area; if the original image has a reflection area, subtract the original small image and the small image after the reflection elimination Up-sampling of the small image of the reflective layer is performed, and the up-sampled small image of the reflective layer is stripped from the original image to obtain a high-quality anti-reflective image, and the output image is displayed on the screen of the smart terminal.
  • the image processing method of the embodiments of the present application may be used to perform down-sampling processing on the original image obtained when the smart terminal is taking a real-time photograph, and the original image is processed into a smaller image; the processed small image Perform reflection elimination processing, compare the original small image and the small image after the reflection elimination, to determine whether the original image has a reflection area; if the original image has a reflection area, subtract the original small image and the small image after the reflection elimination Up-sampling of the small reflective layer image is performed, and the up-sampled small reflective layer image is stripped from the original image to obtain a high-quality anti-reflective image, and the output image is saved in the album of the smart terminal.
  • the image processing method of the embodiment of the present application can be applied to the field of automatic driving.
  • it can be applied to the navigation system of an automatic driving vehicle.
  • the image processing method in this application can make the automatic driving vehicle navigate on the road by processing the image of the glass reflective area in the acquired screen to eliminate The reflective area in the picture can obtain a high-quality non-reflective image, thereby realizing the safety of autonomous vehicles.
  • the present application provides an image processing method, which includes: down-sampling the original image obtained by shooting, and processing the original image into a smaller image; performing reflection elimination processing on the processed small image, and The original small image and the small image after the reflection elimination are compared to determine whether the original image has a reflection area; if the original image has a reflection area, the small image of the reflective layer obtained by subtracting the original small image and the small image after the reflection elimination is performed Sampling, and stripping the up-sampled small image of the reflective layer from the original image to obtain a high-quality road image with no reflections.
  • a neural network can be composed of neural units.
  • a neural unit can refer to an arithmetic unit that takes xs and intercept 1 as inputs.
  • the output of the arithmetic unit can be:
  • s 1, 2,...n, n is a natural number greater than 1
  • Ws is the weight of xs
  • b is the bias of the neural unit.
  • f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal.
  • the output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function.
  • a neural network is a network formed by connecting multiple above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected with the local receptive field of the previous layer to extract the characteristics of the local receptive field.
  • the local receptive field can be a region composed of several neural units.
  • Deep neural network also known as multi-layer neural network
  • the DNN is divided according to the positions of different layers.
  • the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer.
  • the first layer is the input layer
  • the last layer is the output layer
  • the number of layers in the middle are all hidden layers.
  • the layers are fully connected, that is to say, any neuron in the i-th layer must be connected to any neuron in the i+1th layer.
  • DNN looks complicated, it is not complicated in terms of the work of each layer. Simply put, it is the following linear relationship expression: in, Is the input vector, Is the output vector, Is the offset vector, W is the weight matrix (also called coefficient), and ⁇ () is the activation function.
  • Each layer is just the input vector After such a simple operation, the output vector is obtained Due to the large number of DNN layers, the coefficient W and the offset vector The number is also relatively large.
  • DNN The definition of these parameters in DNN is as follows: Take coefficient W as an example: Suppose in a three-layer DNN, the linear coefficients from the fourth neuron in the second layer to the second neuron in the third layer are defined as The superscript 3 represents the number of layers where the coefficient W is located, and the subscript corresponds to the output third-level index 2 and the input second-level index 4.
  • the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as
  • Convolutional neural network (convolutional neuron network, CNN) is a deep neural network with a convolutional structure.
  • the convolutional neural network contains a feature extractor composed of a convolutional layer and a sub-sampling layer.
  • the feature extractor can be regarded as a filter.
  • the convolutional layer refers to the neuron layer that performs convolution processing on the input signal in the convolutional neural network.
  • a neuron can be connected to only part of the neighboring neurons.
  • a convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. Neural units in the same feature plane share weights, and the shared weights here are the convolution kernels.
  • Sharing weight can be understood as the way of extracting image information has nothing to do with location.
  • the convolution kernel can be initialized in the form of a matrix of random size. In the training process of the convolutional neural network, the convolution kernel can obtain reasonable weights through learning. In addition, the direct benefit of sharing weights is to reduce the connections between the layers of the convolutional neural network, and at the same time reduce the risk of overfitting.
  • Generative Adversarial Networks is a deep learning model.
  • the model includes at least two modules: one module is a Generative Model, and the other is a Discriminative Model. Through these two modules, they can learn from each other to produce better output.
  • Both the generative model and the discriminant model can be a neural network, specifically a deep neural network, or a convolutional neural network.
  • GAN Global System for Mobile Communications
  • the goal of generating network G is to generate as real images as possible to deceive the discriminating network D, and the goal of discriminating network D is to distinguish the image generated by G from the real image as much as possible Come.
  • G and D constitute a dynamic "game” process, that is, the "confrontation” in the "generative confrontation network”.
  • an excellent generative model G is obtained, which can be used to generate images.
  • FIG. 4 is a system architecture 100 provided by an embodiment of the present application.
  • the data collection device 160 is used to collect training data.
  • the image processing model can be further trained through training data, that is, the training data collected by the data collection device 160 can be training images.
  • the training data for training the image processing model in the embodiment of the present application may include an original image and a sample image corresponding to the original image.
  • the training data may also include a label, for example, a label indicating whether the original image is reflective.
  • the original image may refer to an image with reflective areas
  • the sample image may refer to an output image that has undergone image reflection elimination processing, for example, it may refer to an image in which the reflective areas are partially or completely eliminated relative to the original image.
  • the data collection device 160 stores the training data in the database 130, and the training device 120 trains based on the training data maintained in the database 130 to obtain the target model/rule 101 (that is, the image processing model in the embodiment of the present application) .
  • the training device 120 inputs the training data into the image processing model until the difference between the predicted image output by the training image processing model and the sample image satisfies a preset condition (for example, the difference between the predicted image and the sample image is less than a certain threshold, or the predicted image and The difference of the sample image remains unchanged or no longer decreases), thereby completing the training of the target model/rule 101.
  • a preset condition for example, the difference between the predicted image and the sample image is less than a certain threshold, or the predicted image and The difference of the sample image remains unchanged or no longer decreases
  • the image processing model used to perform the image processing method in the embodiment of the present application can realize end-to-end training.
  • the image processing model can input the original image and the image after the reflection area corresponding to the input original image (for example, , True value image) to achieve end-to-end training.
  • the target model/rule 101 is obtained by training an image processing model, that is, the image processing model may be a model for judging and eliminating light reflection.
  • the training data maintained in the database 130 may not all come from the collection of the data collection device 160, and may also be received from other devices.
  • the training device 120 does not necessarily perform the training of the target model/rule 101 completely based on the training data maintained by the database 130. It may also obtain training data from the cloud or other places for model training. The above description should not be used as a reference to this application. Limitations of the embodiment. It should also be noted that at least part of the training data maintained in the database 130 may also be used to execute the process of the processing to be processed by the execution device 110.
  • the target model/rule 101 trained according to the training device 120 can be applied to different systems or devices, such as the execution device 110 shown in FIG. 4, which can be a terminal, such as a mobile phone terminal, a tablet computer, Laptops, AR/VR, vehicle-mounted terminals, etc., can also be servers or clouds.
  • the execution device 110 shown in FIG. 4 can be a terminal, such as a mobile phone terminal, a tablet computer, Laptops, AR/VR, vehicle-mounted terminals, etc., can also be servers or clouds.
  • the execution device 110 is configured with an input/output (input/output, I/O) interface 112 for data interaction with external devices.
  • the user can input data to the I/O interface 112 through the client device 140.
  • the input data in this embodiment of the present application may include: a to-be-processed image input by the client device.
  • the preprocessing module 113 and the preprocessing module 114 are used to perform preprocessing according to the input data (such as the image to be processed) received by the I/O interface 112.
  • the preprocessing module 113 and the preprocessing module may not be provided.
  • 114 there may only be one preprocessing module, and the calculation module 111 is directly used to process the input data.
  • the execution device 110 may call data, codes, etc. in the data storage system 150 for corresponding processing .
  • the data, instructions, etc. obtained by corresponding processing may also be stored in the data storage system 150.
  • the I/O interface 112 returns the result of the processing, as described above, to obtain the image of the anti-reflective area of the original image to be processed, and returns the obtained output image to the client device 140 to provide it to the user.
  • the training device 120 can generate corresponding target models/rules 101 based on different training data for different goals or tasks, and the corresponding target models/rules 101 can be used to achieve the above goals or complete The above tasks provide users with the desired results.
  • the user can manually set input data, and the manual setting can be operated through the interface provided by the I/O interface 112.
  • the client device 140 can automatically send input data to the I/O interface 112. If the client device 140 is required to automatically send the input data and the user's authorization is required, the user can set the corresponding authority in the client device 140. The user can view the result output by the execution device 110 on the client device 140, and the specific presentation form may be a specific manner such as display, sound, and action.
  • the client device 140 can also be used as a data collection terminal to collect the input data of the input I/O interface 112 and the output result of the output I/O interface 112 as new sample data, and store it in the database 130 as shown in the figure.
  • the I/O interface 112 directly uses the input data input to the I/O interface 112 and the output result of the output I/O interface 112 as a new sample as shown in the figure.
  • the data is stored in the database 130.
  • FIG. 4 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship among the devices, devices, modules, etc. shown in the figure does not constitute any limitation.
  • the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.
  • the target model/rule 101 is obtained through training according to the training device 120.
  • the target model/rule 101 may be an image processing model in the embodiment of the present application.
  • the image processing model provided in the embodiment of the present application may be Deep neural network, convolutional neural network, or, it can be deep convolutional neural network, etc.
  • FIG. 5 is a hardware structure of a chip provided by an embodiment of the present application.
  • the chip includes a neural network processor 300 (neural-network processing unit, NPU).
  • the chip can be set in the execution device 110 as shown in FIG. 4 to complete the calculation work of the calculation module 111.
  • the chip can also be set in the training device 120 as shown in FIG. 4 to complete the training work of the training device 120 and output the target model/rule 101.
  • the NPU 300 is mounted on a main central processing unit (CPU) as a coprocessor, and the main CPU distributes tasks.
  • the core part of the NPU 300 is the arithmetic circuit 303.
  • the controller 304 controls the arithmetic circuit 303 to extract data from the memory (weight memory or input memory) and perform calculations.
  • the arithmetic circuit 303 includes multiple processing units (process engines, PE).
  • the arithmetic circuit 303 is a two-dimensional systolic array; the arithmetic circuit 303 may also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition.
  • the arithmetic circuit 303 is a general-purpose matrix processor.
  • the arithmetic circuit 303 fetches the corresponding data of the matrix B from the weight memory 302 and caches it on each PE in the arithmetic circuit 303; the arithmetic circuit 303 receives the input
  • the memory 301 takes the data of matrix A and matrix B to perform matrix operations, and the partial results or final results of the obtained matrix are stored in an accumulator 308 (accumulator).
  • the vector calculation unit 307 can perform further processing on the output of the arithmetic circuit 303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, and so on.
  • the vector calculation unit 307 can be used for network calculations in the non-convolutional/non-FC layer of the neural network, such as pooling, batch normalization, local response normalization, etc. .
  • the vector calculation unit 307 can store the processed output vector to the unified memory 306.
  • the vector calculation unit 307 may apply a nonlinear function to the output of the arithmetic circuit 303, such as a vector of accumulated values, to generate the activation value.
  • the vector calculation unit 307 generates a normalized value, a combined value, or both.
  • the processed output vector can be used as an activation input to the arithmetic circuit 303, for example for use in a subsequent layer in a neural network.
  • the unified memory 306 is used to store input data and output data.
  • the weight data directly passes through the storage unit access controller 305 (direct memory access controller, DMAC) to store the input data in the external memory into the input memory 301 and/or the unified memory 306, and store the weight data in the external memory into the weight memory 302 , And store the data in the unified memory 306 into the external memory.
  • DMAC direct memory access controller
  • the bus interface unit 310 (bus interface unit, BIU) is used to implement interaction between the main CPU, the DMAC, and the fetch memory 309 through the bus.
  • the instruction fetch buffer 309 connected to the controller 304 is used to store instructions used by the controller 304; the controller 304 is used to call the instructions buffered in the instruction fetch memory 309 to control the working process of the computing accelerator.
  • unified memory 306, input memory 301, weight memory 302, and fetch memory 309 are all on-chip (On-Chip) memories.
  • Memory double data rate synchronous dynamic random access memory, DDR SDRAM), high bandwidth memory (HBM) or other readable and writable memory.
  • the execution device 110 in FIG. 4 introduced above can execute each step of the image processing method of the embodiment of the present application, and the chip shown in FIG. 5 may also be used to execute each step of the image processing method of the embodiment of the present application.
  • the image processing method provided in the embodiment of the present application can be deployed on the computing node of the related device, and the reflection detection and removal method in the embodiment of the present application can be effectively implemented through software algorithms.
  • the system architecture of the embodiment of the present application will be described in detail below in conjunction with FIG. 6.
  • the image processing method of the embodiment of the present application can be implemented by an image reflection detection and removal system software architecture 500.
  • the reflection detection and removal system software architecture may include: an image preprocessing module 510, a network inference module 520, and an image Post-processing module 530 and software processing module 540. Each module is described in detail below.
  • the image preprocessing module 510 is used for preprocessing the original image, for example, processing the original high-definition large image into a small image of a fixed size.
  • the input of the image preprocessing module 510 is the original image, and the output is a small image of a fixed size.
  • the small image can be used as the input image of the network inference module 520.
  • the size of the input image is equal to the input size of the neural network model of the network inference module 520. . Because the original high-definition large image often has a higher resolution, such as 4K resolution, which exceeds the input size of the image reflection detection and removal neural network model, it is necessary to preprocess the original image so that the processed image can adapt to the image Reflective detection and removal of the input of the neural network model.
  • the original image can be preprocessed by down-sampling, or subtracting the mean, or normalizing.
  • the input image obtained after preprocessing is a small image of 896 ⁇ 896 pixels. It should be understood that the size of the input image obtained after preprocessing may also be another size that meets the input requirements of the image reflection detection and removal neural network model, which is not limited in the embodiment of the present application.
  • the image preprocessing module 510 is an optional module.
  • the size of the original image meets the input size requirement of the neural network model of the network inference module 520. In this case, there is no need to preprocess the original image, and the input image input to the network inference module 520 is the original image.
  • the network reasoning module 520 includes a reflection elimination model, which is used to perform preliminary reflection elimination processing on the input image.
  • the input original image of the anti-reflective model, or the input image obtained by image preprocessing of the input original image is output as an image after anti-reflective on the input image (for example, the first image described below).
  • the network inference module 520 may also include a reflection judgment model for judging whether the input image has reflection.
  • the input of the reflection judgment module is the first image and the input image, and the output is the preliminary reflection judgment result.
  • the judgment result is: the input image has reflection; or, the input image has no reflection.
  • Some anti-reflection models have limitations on the size of the input image. In other words, this type of anti-reflection model can only handle images with lower resolution, such as 896 ⁇ 896 pixels.
  • the original image (for example, 4K resolution) of the high-definition large image can be preprocessed into a small image by the image preprocessing module 510 and input into the reflection model.
  • the processing scheme of the present application enables the problem of reflection of high-definition large images to be dealt with even if a low-resolution reflection elimination model is used.
  • the image post-processing module 530 is used for judging whether the input image has reflection, and processing the original image according to the judgment result, and outputting an image that eliminates the reflection.
  • the network inference module 520 includes a reflection judgment model
  • the image post-processing module 530 is configured to determine whether the input image has reflection according to the judgment result of the reflection judgment model. When it is determined that the input image has reflections, the original image is processed to output an image that eliminates reflections; otherwise, the original image is output.
  • the image post-processing module 530 includes a reflective layer intensity judging unit whose input is the first image and the input image, and the output is a judging value for judging whether the input image is reflective.
  • the image post-processing module 530 is also used to determine whether the input image has light reflection according to the judgment result of the light-reflecting layer intensity judgment unit. When it is determined that the input image has reflections, the original image is processed to output an image that eliminates reflections; otherwise, the original image is output.
  • the network inference module 520 includes a reflection judgment model
  • the image post-processing module 530 includes a reflection layer intensity judgment unit and a joint judgment unit.
  • the image post-processing module 530 is used for judging whether the input image is reflective according to the judgment result of the light reflection judgment model and the judgment result of the light reflection layer intensity judgment unit. When it is determined that the input image has reflections, the original image is processed to output an image that eliminates reflections; otherwise, the original image is output.
  • the input image is the original image.
  • the image post-processing module 530 is used to subtract the first image and the input image to obtain a small image of the reflective layer, and strip the reflective image from the original image. Layer a small image to get the output image.
  • the output image is the image obtained after the original image eliminates reflections.
  • the image post-processing module 530 does not need to process the original image.
  • the original image is output as the output image of the image post-processing module 530.
  • the input image is an image obtained after the original image is preprocessed.
  • the image post-processing module 530 is used to subtract the first image and the input image to obtain a small image of the reflective layer.
  • the image post-processing module 530 is also used for up-sampling the small image of the reflective layer to obtain a reflective layer, wherein the size of the reflective layer is the size of the original image.
  • the image post-processing module 530 peels off the reflective layer from the original image to obtain an output image after the original image has been de-reflected.
  • the image post-processing module 530 does not need to process the original image.
  • the output image is the original image.
  • the network reasoning module 520 determines that the image is non-reflective, which may be an image that has no reflection at all, or it may be an image with very low reflection components that will not affect the image quality (for example, the reflection confidence obtained by the reflection judgment is lower than a certain threshold Image). Performing traditional anti-reflection processing on such an image may actually reduce the image quality.
  • the embodiments of the present application can directly use these original images as output images to improve the effect of eliminating reflections.
  • the software processing module 540 is configured to receive the output image of the image post-processing module 530 and present the output image to the user. Specifically, the software processing module 540 mainly plays a role of scheduling, controlling the device to obtain images from the gallery or camera, etc., controlling and enabling the operation of the image preprocessing module 510, network inference module 520, and image postprocessing module 530, and interacting with the user at the same time. There are no reflection results and no reflection images recommended to users.
  • the image pre-processing module 510, the image post-processing module 530, and the software processing module 540 may be implemented by the main CPU of the chip shown in FIG. 5; the network inference module 520 may be implemented by the neural network of the chip shown in FIG.
  • the processor 300 is implemented.
  • the reflection elimination solution of the present application does not simply perform the elimination operation on all images, but will further determine whether there is reflection in the original image by initially eliminating the reflection of the image, and then provide the final output image according to the judgment result. This can improve the reflection processing effect of the image.
  • the reflection elimination model directly processes high-definition large images, the network complexity of the model and the power consumption of the device will increase significantly, and it is not suitable for some small devices, such as mobile terminals.
  • the reflection elimination model of the embodiment of the application can keep the network complexity small, and through the up-sampling of the network inference module, the reflection elimination effect of the high-definition large image (for example, the image taken by the camera of the mobile terminal) is still better. .
  • FIG. 7 is a schematic diagram of an image processing method 700 according to an embodiment of the present application.
  • This method can be implemented by the software architecture of the reflection detection and removal system shown in FIG. 6.
  • the method may also be executed by an apparatus capable of image processing.
  • the method may be executed by the execution device 110 in FIG. 4.
  • the steps in the method 700 are described in detail below.
  • Step S710 Perform preprocessing on the original image to obtain a first image that meets the input size of the reflection elimination model.
  • Step S710 may be implemented by the image preprocessing module 510 in the software architecture shown in FIG. 6.
  • the original image often has a larger size or a higher resolution, such as 4K resolution. The size of such an original image exceeds the input size of the anti-reflective model, so it is necessary to preprocess the original image.
  • the original image may be processed in a down-sampling manner, and the input image is generated from the high-definition original image sampling.
  • the input image may be a small image of a fixed size, and the size of the input image meets the input size requirement of the reflection elimination model.
  • the size of the input image obtained after preprocessing is 896 ⁇ 896 pixels. It should be understood that the size of the input image obtained after preprocessing may also be another size that meets the input requirements of the reflection elimination model, which is not limited in the embodiment of the present application.
  • the size of the original image meets the input size requirement of the reflection elimination model, so the image preprocessing operation in step S710 may not be performed. In this case, you can directly use the original image as the input image and perform the following steps.
  • Step S720 Input the input image into the anti-reflective model to obtain a first image after anti-reflective.
  • Step S720 can be implemented by the network inference module 520 in the software architecture shown in FIG. 6.
  • the reflection elimination model may be a generative network obtained by generating adversarial networks (GAN), or other neural networks, which is not limited in the embodiment of the present application.
  • GAN adversarial networks
  • Step S730 Input the first image and the input image into the light reflection judgment model, and obtain the judgment result of the light reflection judgment model (for example, the first judgment value described below).
  • Step S730 can be implemented by the network inference module 520 in the software architecture shown in FIG. 6.
  • the reflective judgment neural network model may be a two-class neural network model, which is used to judge whether the original image is a reflective image.
  • the reflection judgment model is a two-class neural network based on a Markov discriminator.
  • FIG. 8 is a schematic diagram of a reflection judgment model according to an embodiment of the present application.
  • the reflection judgment model consists of multiple convolutional layers, which can expand the receptive field of each neuron.
  • the output of the Markov discriminator is a matrix, and each output in the matrix represents a receptive field of the input picture, corresponding to a patch of the input picture.
  • the output of the last convolutional layer of the reflection judgment model is the prediction results of multiple patches.
  • the input of the reflection judgment model is the first image and the input image, and the output is the first judgment value. For example, the first judgment value is that the input image is reflective, or the first judgment value is that the input image is not reflective.
  • a fully connected layer may be added after the last convolutional layer to perform nonlinear mapping on the prediction results of multiple patches to obtain a reflective confidence coefficient.
  • the reflection confidence coefficient can also be called the reflection confidence, the reflection coefficient, etc., and is used to indicate the possibility of the image having a reflection component, or the possibility that the image does not have the reflection component.
  • the reflection confidence coefficient indicates the possibility of the reflection component of the image as an example for description.
  • the first judgment value may be determined according to the value of the reflection confidence coefficient. For example, when the value of the reflection confidence coefficient is greater than a certain threshold, the first judgment value is 1, which means that the input image is reflective; otherwise, the first judgment value is 0, which means that the input image is not reflective. Or, when the value of the confidence coefficient is less than a certain threshold, the first judgment value is 1, which means that the input image is reflective; otherwise, the first judgment value is 0, which means that the input image is not reflective.
  • the corresponding relationship between the specific value of the first judgment value and the presence or absence of reflection in the image is not limited to the above examples, and various reasonable settings can be made as required.
  • the value of the reflection confidence coefficient may be directly used as the first judgment value.
  • the subsequent module or unit of the reflection judgment model further determines whether there is reflection in the image according to the first judgment value.
  • the reflection judgment model can be trained in the following manner:
  • the paired original image sample and the result of the original image sample after the reflection is eliminated, and the label of whether the original image is a reflection image are input to the reflection judgment model for training.
  • the label is 1; if the original image is a non-reflective image, the label is 0.
  • the reflection judgment model is essentially a two-classification network, which is used to distinguish the authenticity of samples. Cross entropy can be used to distinguish the similarity of distributions.
  • the loss function is:
  • N is the number of samples
  • x i represents the i-th pair of samples, that is, the paired original image and the result of the image after the reflection is eliminated;
  • y i represents the label of x i , the positive class is 1, and the negative class is 0.
  • the label can be 1, and if the original image is non-reflective, the label can be 0;
  • D(x i ) represents the probability that the predicted result of the sample x i is the above-mentioned "positive class".
  • Step S740 Perform anti-reflection processing on the original image according to the first judgment value.
  • Step S740 can be implemented by the image post-processing module 530 in the software architecture shown in FIG. 6.
  • the input is the original image, the first image, the input image, and the first judgment value obtained in step 730; the output is the original image after the reflection is eliminated.
  • the reflection judgment result first judgment value
  • the original image can be divided into the following situations:
  • Case 1 The input image is reflective, and the input image is the original image.
  • the input image can be subtracted from the first image to obtain a small image of the reflective layer.
  • the result of the subtraction of the first image and the input image may also be filtered to obtain a small image of the reflective layer.
  • a small image of the reflective layer can be obtained by the following calculation method:
  • M blend is the input image
  • M low_trans is the first image
  • M low_ref is the small image of the reflective layer.
  • the input image and the first image are both three-channel RGB images, and the sizes of the input image and the first image are both 896 ⁇ 896 pixels, so M blend and M low_trans are both 896 ⁇ 896 ⁇ 3 matrices.
  • Each element of the matrix represents the brightness value of each pixel of the image in the three color channels of red (red, R), green (green, G), and blue (blue, B).
  • the M low_ref obtained by subtracting the input image and the first image is also a 896 ⁇ 896 ⁇ 3, where each element in the matrix represents the value of each pixel in the RGB three color channels of the small image of the reflective layer. Brightness value.
  • the small image of the reflective layer is also the size of the original image.
  • the small image of the reflective layer is stripped from the original image to obtain an output image, which is the original image after the reflection is eliminated.
  • the original image can be obtained by the following calculation method after the reflection is eliminated:
  • Morig is the original image
  • M ref is the reflective layer
  • M trans is the result of removing the reflective layer from the original image.
  • the reflective layer M ref is equal to the small reflective layer M low_ref.
  • Morig and M ref are both 896 ⁇ 896 ⁇ 3 matrices. Therefore, the original image obtained by subtracting the original image and the reflective layer from the original image after the reflection is removed M trans is also a 896 ⁇ 896 ⁇ 3 The matrix, where each element in the matrix represents the brightness value of each pixel in the RGB three color channels of the original image after the reflection is eliminated.
  • Case 2 The input image is reflective, and the input image is the preprocessed image of the original image.
  • the input image can be subtracted from the first image to obtain a small image of the reflective layer.
  • the result of the subtraction of the first image and the input image may also be filtered to obtain a small image of the reflective layer.
  • a small image of the reflective layer can be obtained by the following calculation method:
  • M blend is the input image
  • M low_trans is the first image
  • M low_ref is the small image of the reflective layer.
  • the input image and the first image are both three-channel RGB images, and the sizes of the input image and the first image are both 896 ⁇ 896 pixels, so M blend and M low_trans are both 896 ⁇ 896 ⁇ 3 matrices.
  • Each element of the matrix represents the brightness value of each pixel of the image in the three color channels of RGB.
  • the M low_ref obtained by subtracting the input image and the first image is also a 896 ⁇ 896 ⁇ 3, where each element in the matrix represents the value of each pixel in the RGB three color channels of the small image of the reflective layer. Brightness value.
  • the size of the small image of the reflective layer is smaller than the size of the original image.
  • the small image of the reflective layer can be up-sampled to obtain the reflective layer, and the size of the reflective layer is the size of the original image.
  • the small image of the first reflective layer may be up-sampled by interpolation. The reflective layer is stripped from the original image to obtain an output image, which is the original image after the reflection is eliminated.
  • the original image can be obtained by the following calculation method after the reflection is eliminated:
  • Morig is the original image
  • M ref is the reflective layer
  • M trans is the result of removing the reflective layer from the original image.
  • the input image is the image after the original image has been preprocessed, so the size of the small reflective layer is smaller than the size of the original image.
  • the size of the original image is 4K (3000 ⁇ 4000 pixels), and Morig is a 3000 ⁇ 4000 ⁇ 3 matrix.
  • the small reflective layer image M low_ref is up-sampled by difference, and the reflective layer M ref with the same size as the original image can be obtained.
  • M ref is also a 3000 ⁇ 4000 ⁇ 3 matrix, where the matrix is Each element of represents the brightness value of each pixel of the reflective layer in the three color channels of RGB.
  • the image M trans after de-reflecting is also a 3000 ⁇ 4000 ⁇ 3 matrix, where each element in the matrix represents the original image in the image after the de-reflecting The brightness value of each pixel in the RGB three color channels.
  • the output image is the original image.
  • the judging model of the intensity of the reflective layer it can be judged whether there is reflection in the image, and the appropriate reflection processing method can be selected according to the judgment result, which can improve the reflection processing effect of the image.
  • the image processing method provided by the embodiments of the present application can automatically identify whether the original image has reflections, and perform reflection elimination processing on the original image with reflections.
  • the image processing method of the embodiment of the present application when the original image size is large, the original image is first down-sampled to adapt to the input scale of the neural network, and then the small reflective layer image is up-sampled, and from the original image Stripping off the up-sampling reflective layer realizes the elimination of high-resolution images.
  • the image processing method of the embodiment of the present application realizes the function of identifying and eliminating reflections through two lightweight neural networks of the reflection judgment model and the reflection elimination model, which overcomes the problem of the excessively large reflection elimination network required for high-definition images.
  • FIG. 9 is a schematic diagram of an image processing method 900 according to another embodiment of the present application.
  • This method can be implemented by the software architecture of the reflection detection and removal system shown in FIG. 6.
  • the method may also be executed by an apparatus capable of image processing.
  • the method may be executed by the execution device 110 in FIG. 4.
  • the steps in the method 900 are described in detail below.
  • Step S910 Perform preprocessing on the original image to obtain an input image that meets the input size of the reflection elimination model.
  • Step S910 can be implemented by the image preprocessing module 510 in the software architecture shown in FIG. 6.
  • Step S920 Input the input image into the anti-reflective model to obtain a first image after anti-reflective.
  • Step S920 can be implemented by the network inference module 520 in the software architecture shown in FIG. 6.
  • steps S910 to S920 are similar to steps S710 to S720 in the method 700 shown in FIG. 7, and will not be described again.
  • Step S930 The first image and the input image are input to the light-reflective layer intensity judgment unit, and a judgment result of the light-reflective layer intensity judgment unit (for example, the second judgment value described below) is obtained.
  • Step S930 may be implemented by the image post-processing module 530 in the software architecture shown in FIG. 6. In some embodiments, it is possible to determine whether the input image has reflections based on the gray scale of the difference between the first image and the input image.
  • Fig. 10 is a schematic diagram of the judging method of the light-reflecting layer intensity judging unit.
  • the input of the light-reflecting layer intensity judgment unit is the first image and the input image, and the output is the second judgment value: the input image is reflective, or the input image is not reflective.
  • the light-reflecting layer intensity judgment unit calculates the gray value of the difference between the input image and the first image, and the gray value can represent the brightness of the image.
  • the size of the input image and the first image is 896 ⁇ 896 pixels.
  • the input image and the first image can be subtracted to obtain a matrix of 896 ⁇ 896 ⁇ 3 representing the reflective layer. Small picture.
  • Each element of the matrix represents the brightness value of each pixel in the small image of the reflective layer in the three color channels of RGB.
  • the gray value can be calculated according to the following formula:
  • Gray(i,j) represents the gray value of each pixel in the small image of the reflective layer
  • the embodiment of the present application may also calculate the gray value according to other methods, for example, taking the average of the brightness values of the three RGB color channels of each pixel as the gray value of each pixel. This is not limited.
  • At least one or more of the maximum gray value of each pixel in the small image of the reflective layer, the intensity of the pixel with the gray value greater than the threshold TH1, and the area of the pixel with the gray value greater than the threshold TH2 can be selected based on The weighted result of the item determines the second judgment value.
  • the intensity of the pixel with a gray value greater than the threshold TH1 may be the sum of the gray values of the pixels with a gray value greater than the threshold TH1, or it may be the average of the gray values of the pixels with a gray value greater than the threshold TH1 , It can also be other values used to characterize the strength, which is not limited in the embodiment of the present application.
  • the area of the pixel with a gray value greater than the threshold TH2 represents the number of pixels with a gray value greater than the threshold TH2.
  • the input image is reflective according to the intensity of the pixel point whose gray scale is greater than the threshold TH1 is greater than the threshold TH3.
  • the input image is reflective according to the weighted result of the maximum value of the gray scale and the area of the pixel point whose gray scale is greater than the threshold TH2 is greater than the threshold TH3.
  • the input image is reflective based on the weighted result of the maximum value of the grayscale image, the intensity of the pixel with the grayscale greater than the threshold TH1, and the area of the pixel with the grayscale greater than the threshold TH2, which is greater than the threshold TH3.
  • Step S940 Perform anti-reflection processing on the original image according to the second judgment value.
  • Step S940 may be implemented by the image post-processing module 530 in the software architecture shown in FIG. 6.
  • the second judgment value is 1, which means that the input image is reflective; otherwise, the second judgment value is 0, which means that the input image is not reflective.
  • the specific value of the second judgment value and the corresponding relationship between the presence or absence of reflection in the image are not limited to the above examples, and various reasonable settings can be made as required.
  • the original image can be divided into the following situations:
  • Case 1 The input image is reflective, and the input image is the original image.
  • the first image can be subtracted from the input image to obtain a small image of the reflective layer.
  • the result of the subtraction of the first image and the input image may also be filtered to obtain a small image of the reflective layer. Since the size of the input image is the size of the original image, the small image of the reflective layer is also the size of the original image. The small image of the reflective layer is stripped from the original image to obtain an output image, which is the original image after the reflection is eliminated.
  • the calculation method of the small image of the reflective layer and the original image that eliminates the reflection is similar to the calculation method described in case 1 in step S740 in the above method 700, and will not be repeated here.
  • Case 2 The input image is reflective, and the input image is the preprocessed image of the original image.
  • the first image can be subtracted from the input image to obtain a small image of the reflective layer.
  • the result of the subtraction of the first image and the input image may also be filtered to obtain a small image of the reflective layer. Since the input image is a preprocessed image of the original image, the size of the small image of the reflective layer is smaller than the size of the original image.
  • the small image of the reflective layer can be up-sampled to obtain the reflective layer, and the size of the reflective layer is the size of the original image.
  • the small image of the first reflective layer may be up-sampled by interpolation. The reflective layer is stripped from the original image to obtain an output image, which is the original image after the reflection is eliminated.
  • the calculation method of the small reflective layer image, the reflective layer, and the original image of the elimination of reflection is similar to the calculation method described in case 2 in step S740 in the above method 700, and will not be repeated here.
  • the output image is the original image.
  • the light-reflecting layer intensity judgment unit it can be judged whether there is light reflection in the image, and an appropriate light-reflecting treatment method can be selected according to the judgment result, which can improve the effect of the light-reflecting treatment of the image.
  • the image processing method provided by the embodiments of the present application can automatically identify whether the original image has reflections, and perform reflection elimination processing on the original image with reflections.
  • the image processing method of the embodiment of the present application when the original image size is large, the original image is first down-sampled to adapt to the input scale of the neural network, and then the small reflective layer image is up-sampled, and from the original image Stripping off the up-sampling reflective layer realizes the elimination of high-resolution images.
  • the image processing method of the embodiment of the present application realizes the function of identifying and eliminating the reflection through the intensity judgment unit of the reflection layer and the lightweight reflection elimination model, which overcomes the problem of the excessively large reflection elimination network required for high-definition images.
  • FIG. 11 is a schematic diagram of an image processing method 1100 according to another embodiment of the present application.
  • This method can be implemented by the software architecture of the reflection detection and removal system shown in FIG. 6.
  • the method may also be executed by an apparatus capable of image processing.
  • the method may be executed by the execution device 110 in FIG. 4.
  • the steps in the method 1100 are described in detail below.
  • Step S1110 Preprocess the original image to obtain an input image that meets the input size of the reflection elimination model.
  • Step S1110 can be implemented by the image preprocessing module 510 in the software architecture shown in FIG. 6.
  • Step S1120 Input the input image into the anti-reflective model to obtain a first image after anti-reflective.
  • Step S1120 can be implemented by the network inference module 520 in the software architecture shown in FIG. 6.
  • step S1110 to step S1120 are similar to step S710 to step S720 in the method 700 shown in FIG. 7, and will not be described again.
  • Step S1130 The first image and the input image are input to the reflection judgment model to obtain a first judgment value.
  • Step S1130 can be implemented by the network inference module 520 in the software architecture shown in FIG. 6.
  • step S1130 is similar to step S730 in the method 700 shown in FIG. 7, and will not be described again.
  • Step S1140 The first image and the input image are input to the light-reflecting layer intensity judgment unit to obtain a second judgment value.
  • Step S1140 can be implemented by the image post-processing module 530 in the software architecture shown in FIG. 6.
  • step S1140 is similar to step S930 in the method 900 shown in FIG. 9 and will not be described again.
  • Step S1150 Perform a joint judgment based on the first judgment value and the second judgment value to obtain a reflection judgment result.
  • Step S1150 can be implemented by the image post-processing module 530 in the software architecture shown in FIG. 6.
  • the first judgment value may indicate whether the input image is reflective. At this time, when both the first judgment value and the second judgment value are reflective, the reflective judgment result is reflective; when at least one of the first judgment value and the second judgment value is non-reflective, the reflective judgment result is No reflections. Or as another implementation manner, when at least one of the first judgment value and the second judgment value is reflective, the reflective judgment result is reflective; when the first judgment value and the second judgment value are both non-reflective , Reflective judgment result is non-reflective.
  • the first judgment value may indicate the confidence of whether the input image has reflection (that is, the above-mentioned reflection confidence or reflection confidence coefficient). At this time, when the first judgment value is greater than or equal to the specific threshold and the second judgment value indicates that there is reflection, the reflection judgment result output in step S1150 is reflection; on the contrary, when the first judgment value is less than the specific threshold and/or the second judgment When the value indicates that there is no reflection, the reflection judgment result output in step S1150 is no reflection.
  • the reflection judgment result output in step S1150 is reflection; on the contrary, when the first judgment value is less than the specific When the threshold value and the second judgment value indicate that there is no reflection, the reflection judgment result output in step S1150 is no reflection.
  • the first judgment value can be further refined. For example, when the first judgment value is greater than the threshold value A1 and the second judgment value indicates no reflection, the reflection judgment result is reflective; when the first judgment value is less than the threshold value A1 is greater than the threshold A2, and the second judgment value indicates that there is reflection, the reflection judgment result is reflection; in other cases, the reflection judgment result is no reflection, where A1 is greater than A2.
  • Appropriate joint judgment algorithms can be designed according to needs, and these changes are all within the scope of the embodiments of the present application.
  • Step S1160 Perform anti-reflective processing on the original image according to the result of the reflection judgment.
  • Step S1160 can be implemented by the image post-processing module 530 in the software architecture shown in FIG. 6.
  • step S1140 is similar to step S740 in the method 700 shown in FIG. 7 and will not be described again.
  • FIG. 12 is a schematic diagram of the structure of an image processing apparatus according to an embodiment of the present application.
  • the image processing device 1200 of the embodiment of the present application includes a first light reflection elimination module 1210, a light reflection judgment module 1220, and a second light reflection elimination module 1230.
  • the first anti-reflective module 1210 is used to perform anti-reflective processing on the input image to obtain a first image.
  • the first reflection elimination module 1210 may be used to implement step S720 in the method 700 shown in FIG. 7, or step S920 in the method 900 shown in FIG. 9, or step S920 in the method 1100 shown in FIG.
  • the function of S1120 The specific functions and beneficial effects of the first reflection elimination module 1210 can be referred to the description in the above method, and for the sake of brevity, the details are not repeated here.
  • the reflection judging module 1220 is used for judging whether the input image has reflections based on the first image and the input image.
  • the reflection judgment module 1220 may include a reflection judgment model for implementing the function of step S730 in the method 700 shown in FIG. 7.
  • the light reflection judgment module 1220 may include a light reflection layer intensity judgment model for implementing the function of step S930 in the method 900 shown in FIG. 9.
  • the reflection judgment module 1220 may include a reflection judgment model, a reflection layer intensity judgment model, and a joint judgment unit for implementing the functions of step S1130 to step S1150 in the method 1100 shown in FIG. 11.
  • the second light reflection elimination module 1230 is configured to obtain an output image according to the judgment result of the light reflection judgment module.
  • the second reflection elimination module is used to implement the function of step S740 in the method 700 shown in FIG. 7, or step S940 in the method 900 shown in FIG. 9, or step S1160 in the method 1100 shown in FIG. 11 .
  • the specific functions and beneficial effects of the second reflection elimination module 1230 can be referred to the description in the above method, and for the sake of brevity, the details are not repeated here.
  • the image processing apparatus may further include a preprocessing module 1240.
  • the preprocessing module 1240 is used to preprocess the original image to generate an input image that meets the input size of the first reflective output module 1210.
  • the preprocessing module 1240 is used to implement the function of step S710 in the method 700 shown in FIG. 7, or step S910 in the method 900 shown in FIG. 9, or step S1100 in the method 1100 shown in FIG. 11.
  • the specific functions and beneficial effects of the pre-processing module 1240 can be referred to the description in the foregoing method, and for brevity, details are not repeated here.
  • the image processing apparatus shown in FIG. 12 may be implemented by the chip shown in FIG. 5.
  • the neural network processor 300 in the chip shown in FIG. 5 can realize the function of the reflection judgment model in the first reflection elimination module 1210 and the reflection judgment module 1220 of the image processing device shown in FIG. 12;
  • the main CPU in the chip can implement the functions of the reflective layer strength judgment unit, the joint judgment unit, and the second light reflection elimination module 1230 in the light reflection judgment module 1220.
  • the main CPU in the chip shown in FIG. 5 may also implement the function of the preprocessing module 1240.
  • the image processing apparatus shown in FIG. 12 may implement the functions of the software architecture shown in FIG. 6.
  • the first reflection elimination module 1210 can realize the function of the reflection elimination model in the network inference module 520 of the software architecture shown in FIG. 6; the reflection judgment model in the reflection judgment module 1220 can realize the network of the software architecture shown in FIG.
  • the function of the reflection judgment model in the inference module; the reflection layer intensity judgment unit, the joint judgment unit and the second reflection elimination module 1230 in the reflection judgment module 1220 can realize the function of the image post-processing module 530 in the software architecture shown in FIG. 6 .
  • the preprocessing module 1240 can implement the function of the image preprocessing module of the software architecture shown in FIG. 6.
  • the image processing apparatus shown in FIG. 12 may be deployed on the execution device 110 shown in FIG. 4 to realize the functions of various modules.
  • FIG. 13 is a schematic diagram of a hardware structure of an embodiment of the present application. It should be understood that the apparatus in the embodiments of the present application may be one or more chips, or may be included in a terminal or other devices.
  • the processor system 1300 in the embodiment of the present application includes a central processing unit (CPU) 1310, a neural network processor (neutral processing unit, NPU) 1320, and an image signal processing (image signal processing, ISP) 1330.
  • CPU central processing unit
  • NPU neural network processor
  • ISP image signal processing
  • the image pre-processing module 510, the image post-processing module 530, and the software processing module 540 of the software architecture shown in FIG. 6 may be applied to the central processing unit 1310 or implemented by the central processing unit 1310.
  • the central processing unit 1310 can also be used to run general operating system software, and control the neural network processor 1320 and the image signal processor 1330 under the action of the general operating system software.
  • the network inference module 520 of the software architecture shown in FIG. 6 may be applied to the neural network processor 1320 or implemented by the neural network processor 1320.
  • the image signal processor 1330 is used to generate images.
  • the image signal processor can match image sensors of different manufacturers to process the image data output by the front-end image sensor, and generate corresponding image signals based on the image data, such as the original picture.
  • the image processing apparatus shown in Fig. 12 may be implemented by the processor system shown in Fig. 13.
  • the central processing unit 1310 can implement the functions of the reflective layer intensity judging unit and/or the joint judging unit and the second reflective elimination module 1230 in the reflective judging module 1220; in other embodiments, the central processing unit 1310 can also implement preprocessing Function of module 1240.
  • the neural network processor 1320 can realize the function of the reflection judgment model in the first reflection elimination module 1210 and the reflection judgment module 1220 in the image processing device shown in FIG. 12.
  • the processor system shown in FIG. 13 may include one or more chips shown in FIG. 5, and the main CPU in the chip shown in FIG. 5 may implement the central processing unit 1310 of the processor system shown in FIG. Function; the neural network processor 300 in the chip shown in FIG. 5 can implement the functions of the neural network processor 1320 in the processor system shown in FIG. 13.
  • processor system shown in FIG. 13 may also be deployed on the execution device 110 shown in FIG. 4, which is not limited in the embodiment of the present application.
  • FIG. 14 is a schematic diagram of the hardware structure of another image processing apparatus provided by an embodiment of the present application.
  • the image processing apparatus 1400 shown in FIG. 14 includes a memory 1410, a processor 1420, a communication interface 1430, and a bus 1440.
  • the memory 1410, the processor 1420, and the communication interface 1430 implement communication connections between each other through the bus 1440.
  • the memory 1410 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).
  • the memory 1410 may store a program.
  • the processor 1420 is configured to execute each step of the image processing method of the embodiment of the present application, for example, execute each of the steps shown in FIGS. 7 to 11 step.
  • the image processing apparatus shown in the embodiment of the present application may be a server, for example, it may be a server in the cloud, or may also be a chip configured in a server in the cloud; or, the image processing apparatus shown in the embodiment of the present application
  • the device can be a smart terminal or a chip configured in the smart terminal.
  • the image processing method disclosed in the foregoing embodiments of the present application may be applied to the processor 1420 or implemented by the processor 1420.
  • the processor 1420 may be an integrated circuit chip with signal processing capabilities.
  • the steps of the above-mentioned image processing method can be completed by hardware integrated logic circuits in the processor 1420 or instructions in the form of software.
  • the processor 1420 may be a chip including the NPU shown in FIG. 5.
  • the aforementioned processor 1420 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose processor, a digital signal processor (digital signal processor, DSP), and an application specific integrated circuit (application integrated circuit).
  • CPU central processing unit
  • GPU graphics processing unit
  • DSP digital signal processor
  • application integrated circuit application specific integrated circuit
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • the steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor.
  • the software module can be located in random access memory (RAM), flash memory, read-only memory (read-only memory, ROM), programmable read-only memory or electrically erasable programmable memory, registers, etc. mature in the field Storage medium.
  • the storage medium is located in the memory 1410, and the processor 1420 reads the instructions in the memory 1410, and completes the reflection detection and elimination system software architecture shown in Figure 6 in the implementation of this application or the image processing device shown in Figure 12 in combination with its hardware.
  • the communication interface 1430 uses a transceiving device such as but not limited to a transceiver to implement communication between the device 1400 and other devices or a communication network.
  • a transceiving device such as but not limited to a transceiver to implement communication between the device 1400 and other devices or a communication network.
  • the bus 1440 may include a path for transferring information between various components of the image processing apparatus 1400 (for example, the memory 1410, the processor 1420, and the communication interface 1430).
  • image processing device 1400 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the image processing device 1400 may also include other necessary for normal operation. Device. At the same time, according to specific needs, those skilled in the art should understand that the above-mentioned image processing apparatus 1400 may further include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the above-mentioned image processing apparatus 1400 may also include only the components necessary for implementing the embodiments of the present application, and not necessarily all the components shown in FIG. 14.
  • the embodiments of the present application also provide a computer-readable medium, and the computer-readable medium stores a computer program (also called code, or instruction) when it runs on a computer, so that the computer executes any of the above-mentioned method embodiments. In the method.
  • a computer program also called code, or instruction
  • the embodiment of the present application also provides a chip system, including a memory and a processor, the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the image processing device installed with the chip system Execute the method in any of the above method embodiments.
  • the chip system may include an input circuit or interface for sending information or data, and an output circuit or interface for receiving information or data.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in a storage medium, or transmitted from one storage medium to another storage medium.
  • the computer instructions may be transmitted from a website, computer, server, or data center through a cable (such as coaxial cable, optical fiber, etc.).
  • the storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center integrated with one or more available media.
  • the usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, and a magnetic tape), an optical medium (for example, a high-density digital video disc (digital video disc, DVD)), or a semiconductor medium (for example, a solid state disk, SSD)) etc.
  • "at least two” or “multiple” means two or more.
  • the term “and/or” describes the association relationship of the associated objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A alone exists, A and B exist at the same time, and B exists alone, where A , B can be singular or plural.
  • the character “/” generally indicates that the associated objects before and after are in an "or” relationship.
  • the pre-definition in this application can be understood as definition, pre-definition, storage, pre-storage, pre-negotiation, pre-configuration, curing, or pre-fired.
  • one embodiment or “an embodiment” mentioned throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the present application. Therefore, the appearances of "in one embodiment” or “in an embodiment” in various places throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not correspond to the embodiments of the present application. The implementation process constitutes any limitation.
  • component used in this specification are used to denote computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution.
  • the component may be, but is not limited to, a process, a processor, an object, an executable file, an execution thread, a program, and/or a computer running on a processor.
  • the application running on the computing device and the computing device can be components.
  • One or more components may reside in processes and/or threads of execution, and components may be located on one computer and/or distributed among two or more computers.
  • these components can be executed from various computer readable media having various data structures stored thereon.
  • the component can be based on, for example, a signal having one or more data packets (e.g. data from two components interacting with another component in a local system, a distributed system, and/or a network, such as the Internet that interacts with other systems through a signal) Communicate through local and/or remote processes.
  • a signal having one or more data packets (e.g. data from two components interacting with another component in a local system, a distributed system, and/or a network, such as the Internet that interacts with other systems through a signal) Communicate through local and/or remote processes.
  • the disclosed system, device, and method can be implemented in other ways.
  • the device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components may be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

本申请提供了一种图像处理方法及装置，该装置包括：第一反光消除模块，用于对输入图像进行反光消除处理，获取第一图像；反光判断模块，用于根据所述第一图像和所述输入图像，获取所述输入图像是否有反光的判断结果；第二反光消除模块，用于根据所述反光判断模块的判断结果，获取输出图像。本申请实施例提供的图像处理装置，能够改善图像的反光消除效果。

Description

一种图像处理方法和装置 技术领域
本申请涉及图像处理技术领域，尤其涉及一种图像处理方法和装置。
背景技术
人工智能(artificial intelligence,AI)是利用数字计算机或者数字计算机控制的机器模拟、延伸和扩展人的智能，感知环境、获取知识并使用知识获得最佳结果的理论、方法、技术及应用系统。换句话说，人工智能是计算机科学的一个分支，它企图了解智能的实质，并生产出一种新的能以人类智能相似的方式作出反应的智能机器。人工智能也就是研究各种智能机器的设计原理与实现方法，使机器具有感知、推理与决策的功能。人工智能领域的研究包括机器人，自然语言处理，计算机视觉，决策与推理，人机交互，推荐与搜索，AI基础理论等。
随着人工智能技术的快速发展，神经网络(例如，深度神经网络)近年来在图像、视频以及语音等多种媒体信号的处理与分析中取得了很大的成就。在日常的照片拍摄中，拍摄的图像中可能存在反光，为了满足用户对照片智能优化处理的需求，可以利用AI算法对图像进行反光消除操作，减小反光对画面效果的影响。
现有的反光消除技术主要通过生成对抗神经网络(generative adversarial networks，GAN)实现。该技术使用时将带反光原图输入训练好的反光消除生成网络，网络推理生成无反光图像。但是，GAN神经网络算法仅有反光消除手段，却无反光识别手段，对于无反光图像进行反光消除后，图像质量会有恶化。
发明内容
本申请提供一种图像处理方法和装置，能够改善图像的反光消除效果。
第一方面，提供了一种图像处理装置，包括：第一反光消除模块，用于对输入图像进行反光消除处理，获取第一图像；反光判断模块，用于根据第一图像和输入图像，获取该输入图像是否有反光的判断结果；第二反光消除模块，用于根据判断结果，获取输出图像。
上述技术方案中，图像处理装置可以根据初步反光消除的结果判断输入图像是否有反光，再对判断结果为有反光的原始图像进行反光消除处理，实现了对有反光图片的自动识别与反光消除。
可选地，该装置还可以包括预处理模块，用于对原始图像进行预处理，获取输入图像，该原始图像的尺寸大于输入图像的尺寸。
可选地，当原始图像的尺寸符合第一反光消除模块的输入尺寸要求时，可以将原始图像作为输入图像。
上述技术方案中，对原始图像进行预处理，可以实现对尺寸较大的原始图片的反光识别与消除。
在一种实现方式中，第二反光消除模块具体用于，在判断结果为输入图像没有反光时，获取原始图像作为输出图像。
当判断结果为输入图像无反光时，可以直接输出原始图像，不对原始图像做多余的处理，避免了对于无反光图像进行反光消除后，图像质量恶化的情况。
在一种实现方式中，当判断结果为输入图像有反光，并且输入图像是原始图像时，第二反光消除模块具体用于根据第一图像和输入图像生成反光层小图，并根据反光层小图和原始图像获取输出图像。
在一种实现方式中，当判断结果为输入图像有反光，并且输入图像时原始图像预处理后的图像时，第二反光消除模块具体用于根据第一图像和输入图像生成反光层小图，对反光层小图进行上采样以获取反光图层，并根据反光图层和原始图像获取输出图像。
上述技术方案中，对反光层小图上采样得到反光图层，再根据原始图像和反光图层得到输出图像，最终可以得到与原图尺寸、分辨率一致的消除反光的输出图像，提高了输出图像的质量。
可选地，反光判断模块包括反光判断模型，反光判断模型是以第一图像和输入图像作为输入的神经网络，该反光判断模型的输出表示输入图像有反光的置信度，或者输出为判断结果。
在一种实现方式中，反光判断模型是基于马尔科夫判别器的神经网络，并且是基于第一样本集训练得到的，其中第一样本集包括：原始图像样本、原始图像样本经过反光消除后的图像样本、原始图像样本是否有反光的标签。
可选地，反光判断模块包括反光层强度判断单元，该反光层强度判断单元用于根据输入图像和第一图像生成反光层小图；反光层强度判断单元还用于计算反光层小图包括的多个像素点的灰度值，并根据多个像素点的灰度值确定判断结果。
在一种实现方式中，该反光层强度判断单元根据如下信息中的至少一种或者多种的加权结果，确定判断结果：多个像素点的灰度值中的最大值、灰度值大于第一阈值的至少一个像素点的强度、灰度值大于第二阈值的至少一个像素点的面积；其中，所述灰度值大于第一阈值的至少一个像素点的强度为灰度值大于第一阈值的像素点的灰度值之和，或者，所述灰度值大于第一阈值的至少一个像素点的强度为灰度值大于第一阈值的像素点的灰度值的平均值；所述灰度值大于第二阈值的至少一个像素点的面积为灰度值大于第二阈值的像素点的个数。
可选地，反光判断模块包括反光判断模型、反光层强度判断单元和联合判断单元，该反光判断模型是以第一图像和输入图像作为输入的神经网络，反光判断模型的输出为表示输入图像是否有反光的第一判断值；该反光层强度判断单元计算所述输入图像和第一图像之差的灰度值，并根据灰度值输出表示所述输入图像是否有反光的第二判断值；该联合判断单元根据第一判断值和第二判断值，确定判断结果。
上述技术方案中，可以通过多种模型或者模块判断原始图像是否有反光，提高了判断结果的准确性。
第二方面，提供了一种图像处理方法，包括：对输入图像进行反光消除处理，获取第一图像；根据输入图像和第一图像，判断输入图像是否有反光；根据所述输入图像是否有反光，获取输出图像。
上述技术方案中，图像处理装置可以根据初步反光消除的结果判断输入图像是否有反光，再对判断结果为有反光的原始图像进行反光消除处理，实现了对有反光图片的自动识别与反光消除。
在一种实现方式中，可以对原始图像进行预处理，以获取输入图像，该原始图像的尺寸大于输入图像的尺寸。
上述技术方案中，对原始图像进行预处理，可以实现对尺寸较大的原始图片的反光识别与消除。
可选地，根据所述输入图像是否有反光，获取输出图像包括：当输入图像没有反光时，输出图像为原始图像。
当判断结果为输入图像无反光时，可以直接输出原始图像，不对原始图像做多余的处理，避免了对于无反光图像进行反光消除后，图像质量恶化的情况。
可选地，根据所述输入图像是否有反光，获取输出图像包括：当输入图像有反光，并且输入图像为原始图像时，该方法还包括：根据输入图像和第一图像生成反光层小图；根据反光层小图和原始图像获取所述输出图像。
可选地，根据所述输入图像是否有反光，获取输出图像包括：当输入图像有反光时，并且输入图像为原始图像预处理后的图像时，该方法还包括：根据输入图像和第一图像生成反光层小图；对反光层小图进行上采样，获取反光图层；根据反光图层和原始图像获取所述输出图像。
上述技术方案中，对反光层小图上采样得到反光图层，再根据原始图像和反光图层得到输出图像，最终可以得到与原图尺寸、分辨率一致的消除反光的输出图像，提高了输出图像的质量。
可选地，上述根据输入图像和第一图像，判断输入图像是否有反光包括：根据反光判断模型，判断输入图像是否有反光；该反光判断模型是以第一图像和输入图像作为输入的神经网络，反光判断模型的输出表示输入图像有反光的置信度，或者输出为所述输入图像是否有反光的判断结果。
在一种实现方式中，反光判断模型是基于马尔科夫判别器的神经网络，并且是基于第一样本集训练得到的，其中第一样本集包括：原始图像样本、原始图像样本经过反光消除后的图像样本、原始图像样本是否有反光的标签。
可选地，上述根据输入图像和第一图像，判断输入图像是否有反光包括：根据反光层强度判断单元，判断输入图像是否有反光；该反光层强度判断单元用于根据输入图像和第一图像生成反光层小图；反光层强度判断单元还用于计算反光层小图包括的多个像素点的灰度值，并根据多个像素点的灰度值确定判断结果。
在一种实现方式中，反光层判断模块根据如下信息中的至少一种或者多种的加权结果，确定判断结果：多个像素点的灰度值中的最大值、灰度值大于第一阈值的至少一个像素点的强度、灰度值大于第二阈值的至少一个像素点的面积；其中，所述灰度值大于第一阈值的至少一个像素点的强度为灰度值大于第一阈值的像素点的灰度值之和，或者，所述灰度值大于第一阈值的至少一个像素点的强度为灰度值大于第一阈值的像素点的灰度值的平均值；所述灰度值大于第二阈值的至少一个像素点的面积为灰度值大于第二阈值的像素点的个数。
可选地，上述根据输入图像和第一图像，判断输入图像是否有反光包括：根据反光判断模型、反光层强度判断单元和联合判断单元，判断输入图像是否有反光；该反光判断模型是以第一图像和输入图像作为输入的神经网络，反光判断模型的输出为表示输入图像是否有反光的第一判断值；反光层强度判断单元计算输入图像和第一图像之差的灰度值，并根据灰度值输出表示所述输入图像是否有反光的第二判断值；该联合判断单元根据第一判断值和第二判断值，确定判断结果。
上述技术方案中，可以通过多种模型或者模块判断原始图像是否有反光，提高了判断结果的准确性。
第三方面，提供一种计算机可读介质，该计算机可读介质存储用于设备执行的程序代码，该程序代码包括用于执行第二方面或者第二方面的任意一种实现方式中的图像处理方法。
第四方面，提供了一种计算机程序产品，该计算机程序产品包括：计算机程序代码，当所述计算机程序代码在计算机上运行时，使得计算机执行第二方面或者第二方面的任意一种实现方式中的图像处理方法。
第五方面，提供一种芯片，该芯片包括处理器与数据接口，处理器通过所述数据接口读取存储器上存储的指令，执行上述第二方面或第二方面中的任意一种实现方式中的图像处理方法。
可选地，作为一种实现方式，该芯片还可以包括存储器，存储器中存储有指令，处理器用于执行存储器上存储的指令，当指令被执行时，处理器用于执行第二方面或者第二方面中的任意一种实现方式中的图像处理方法。
第六方面，提供了一种装置，包括：处理器和存储器，存储器用于存储所述计算机程序代码，当所述计算机程序代码在所述处理器上运行时，使得该装置执行第二方面或者第二方面的任意一种实现方式中的图像处理方法。
附图说明
图1是本申请实施例提供的图像处理方法的应用场景示意图；
图2是本申请实施例提供的一种应用场景的示意图；
图3是本申请实施例提供的另一种应用场景的示意图；
图4是本申请实施例提供的系统架构示意图；
图5是本申请实施例提供的芯片硬件结构示意图；
图6是本申请实施例提供的反光检测与去除系统软件架构的示意图；
图7是本申请实施例提供的一种图像处理方法的流程示意图；
图8是本申请实施例提供的一种反光判断模型的结构示意图；
图9是本申请实施例提供的另一种图像处理方法的流程示意图；
图10是本申请实施例提供的一种反光强度判断模型的结构示意图；
图11是本申请实施例提供的另一种图像处理方法的流程示意图；
图12是本申请实施例提供的一种图像处理装置的硬件结构示意图；
图13是本申请实施例提供的另一种图像处理装置的硬件结构示意图；
图14是本申请实施例提供的另一种图像处理装置的硬件结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行描述。可以理解的是，所描述的实施例是本申请一部分的实施例，而不是全部的实施例。
图1是本申请实施例提供的图像处理方法的应用场景示意图。
如图1所示，本申请实施的图像处理方法可以应用于智能终端，比如，可以对智能设备中的摄像头拍摄的原始图像进行图像处理，得到该原始图像去除画面中的反光后的高清大图，从而提高图像显示的质量。上述智能终端可以为移动的或固定的，例如，该智能终端可以是具有图像处理功能的移动电话、平板个人电脑(tablet personal computer，TPC)、媒体播放器、智能电视、笔记本电脑(laptop computer，LC)、个人数字助理(personal digital assistant，PDA)、个人计算机(personal computer，PC)、照相机、摄像机、智能手表、增强现实(augmented reality，AR)/虚拟现实(virtual reality，VR)，可穿戴式设备(wearable device，WD)或者自动驾驶的车辆等，本申请实施例对此不作限定。
下面对本申请实施例的具体应用场景进行举例说明。
应用场景一：智能终端拍照领域
在一个实施例中，如图2所示，本申请实施例的图像处理方法可以应用于智能终端设备(例如，手机)的拍摄。在通过智能终端的摄像头对物体进行拍照时，在博物馆、商场等场景下，可能遇到拍摄玻璃、橱窗的画面中存在反光区域的情况。通过本申请实施例的图像方法，即去除反光的方法，可以获得画面质量较高的、对原始图像去除反光区域的输出图像。
示例性地，可以通过本申请实施例的图像处理方法在智能终端进行实时拍照时，对拍摄得到的原始图像进行下采样处理，将原始图像处理成较小的图像；将处理后的小的图像进行反光消除处理，将原始的小图和反光消除后的小图进行对比处理，判断原图像是否存在反光区域；若原图像存在反光区域，对原始小图和反光消除后的小图相减得到的反光层小图进行上采样，并在原始的图像中剥离上采样后的反光层小图，得到高质量的消除反光的图像，将输出图像显示在智能终端的屏幕上。
示例性地，可以通过本申请实施例的图像处理方法在智能终端进行实时拍照时，对拍摄得到的原始图像进行下采样处理，将原始图像处理成较小的图像；将处理后的小的图像进行反光消除处理，将原始的小图和反光消除后的小图进行对比处理，判断原图像是否存在反光区域；若原图像存在反光区域，对原始小图和反光消除后的小图相减得到的反光层小图进行上采样，并在原始的图像中剥离上采样后的反光层小图，得到高质量的消除反光的图像，将输出图像保存至智能终端的相册中。
应用场景二：自动驾驶领域
在一个实施例中，如图3所示，本申请实施例的图像处理方法可以应用于自动驾驶领域。例如，可以应用于自动驾驶车辆的导航系统中，通过本申请中的图像处理方法可以使得自动驾驶车辆在道路行驶的导航过程中，通过对获取的画面中存在玻璃反光区域的图像进行处理，消除图相中的反光区域，得到高质量的无反光图像，从而实现自动驾驶车辆的安全性。
示例性地，本申请提供了一种图像处理方法，包括：对拍摄得到的原始图像进行下采样处理，将原始图像处理成较小的图像；将处理后的小的图像进行反光消除处理，将原始 的小图和反光消除后的小图进行对比处理，判断原图像是否存在反光区域；若原图像存在反光区域，对原始小图和反光消除后的小图相减得到的反光层小图进行上采样，并在原始的图像中剥离上采样后的反光层小图，得到高质量的消除反光的道路图像。
由于本申请实施例涉及大量神经网络的应用，为了便于理解，下面先对本申请实施例可能涉及的神经网络的相关术语和概念进行介绍。
(1)神经网络
神经网络可以是由神经单元组成的，神经单元可以是指以xs和截距1为输入的运算单元，该运算单元的输出可以为：
Figure PCTCN2020081191-appb-000001
其中，s＝1、2、……n，n为大于1的自然数，Ws为xs的权重，b为神经单元的偏置。f为神经单元的激活函数(activation functions)，用于将非线性特性引入神经网络中，来将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入，激活函数可以是sigmoid函数。神经网络是将多个上述单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
(2)深度神经网络
深度神经网络(deep neural network，DNN)，也称多层神经网络，可以理解为具有多层隐含层的神经网络。按照不同层的位置对DNN进行划分，DNN内部的神经网络可以分为三类：输入层，隐含层，输出层。一般来说第一层是输入层，最后一层是输出层，中间的层数都是隐含层。层与层之间是全连接的，也就是说，第i层的任意一个神经元一定与第i+1层的任意一个神经元相连。
虽然DNN看起来很复杂，但是就每一层的工作来说，其实并不复杂，简单来说就是如下线性关系表达式：
Figure PCTCN2020081191-appb-000002
其中，
Figure PCTCN2020081191-appb-000003
是输入向量，
Figure PCTCN2020081191-appb-000004
是输出向量，
Figure PCTCN2020081191-appb-000005
是偏移向量，W是权重矩阵(也称系数)，α()是激活函数。每一层仅仅是对输入向量
Figure PCTCN2020081191-appb-000006
经过如此简单的操作得到输出向量
Figure PCTCN2020081191-appb-000007
由于DNN层数多，系数W和偏移向量
Figure PCTCN2020081191-appb-000008
的数量也比较多。这些参数在DNN中的定义如下所述：以系数W为例：假设在一个三层的DNN中，第二层的第4个神经元到第三层的第2个神经元的线性系数定义为
Figure PCTCN2020081191-appb-000009
上标3代表系数W所在的层数，而下标对应的是输出的第三层索引2和输入的第二层索引4。
综上，第L-1层的第k个神经元到第L层的第j个神经元的系数定义为
Figure PCTCN2020081191-appb-000010
需要注意的是，输入层是没有W参数的。在深度神经网络中，更多的隐含层让网络更能够刻画现实世界中的复杂情形。理论上而言，参数越多的模型复杂度越高，“容量”也就越大，也就意味着它能完成更复杂的学习任务。训练深度神经网络的也就是学习权重矩阵的过程，其最终目的是得到训练好的深度神经网络的所有层的权重矩阵(由很多层的向量W形成的权重矩阵)。
(3)卷积神经网络
卷积神经网络(convolutional neuron network，CNN)是一种带有卷积结构的深度神经网络。卷积神经网络包含了一个由卷积层和子采样层构成的特征抽取器，该特征抽取器可 以看作是滤波器。卷积层是指卷积神经网络中对输入信号进行卷积处理的神经元层。在卷积神经网络的卷积层中，一个神经元可以只与部分邻层神经元连接。一个卷积层中，通常包含若干个特征平面，每个特征平面可以由一些矩形排列的神经单元组成。同一特征平面的神经单元共享权重，这里共享的权重就是卷积核。共享权重可以理解为提取图像信息的方式与位置无关。卷积核可以以随机大小的矩阵的形式初始化，在卷积神经网络的训练过程中卷积核可以通过学习得到合理的权重。另外，共享权重带来的直接好处是减少卷积神经网络各层之间的连接，同时又降低了过拟合的风险。
(4)生成式对抗网络
生成式对抗网络(GAN,Generative Adversarial Networks)是一种深度学习模型。该模型中至少包括两个模块：一个模块是生成模型(Generative Model)，另一个模块是判别模型(Discriminative Model)，通过这两个模块互相博弈学习，从而产生更好的输出。生成模型和判别模型都可以是神经网络，具体可以是深度神经网络，或者卷积神经网络。GAN的基本原理如下：以生成图像的GAN为例，假设有两个网络，G(Generator)和D(Discriminator)，其中G是一个生成图像的网络，它接收一个随机的噪声z，通过这个噪声生成图像，记做G(z)；D是一个判别网络，用于判别一张图像是不是“真实的”。它的输入参数是x，x代表一张图像，输出D(x)代表x为真实图像的概率，如果为1，就代表100％是真实的图像，如果为0，就代表不可能是真实的图像。在对该生成式对抗网络进行训练的过程中，生成网络G的目标就是尽可能生成真实的图像去欺骗判别网络D，而判别网络D的目标就是尽量把G生成的图像和真实的图像区分开来。这样，G和D就构成了一个动态的“博弈”过程，也即“生成式对抗网络”中的“对抗”。最后博弈的结果，在理想的状态下，G可以生成足以“以假乱真”的图像G(z)，而D难以判定G生成的图像究竟是不是真实的，即D(G(z))＝0.5。这样就得到了一个优异的生成模型G，它可以用来生成图像。
(5)损失函数
在训练深度神经网络的过程中，因为希望深度神经网络的输出尽可能的接近真正想要预测的值，所以可以通过比较当前网络的预测值和真正想要的目标值，再根据两者之间的差异情况来更新每一层神经网络的权重向量(当然，在第一次更新之前通常会有初始化的过程，即为深度神经网络中的各层预先配置参数)，比如，如果网络的预测值高了，就调整权重向量让它预测低一些，不断地调整，直到深度神经网络能够预测出真正想要的目标值或与真正想要的目标值非常接近的值。因此，就需要预先定义“如何比较预测值和目标值之间的差异”，这便是损失函数(loss function)或目标函数(objective function)，它们是用于衡量预测值和目标值的差异的重要方程。其中，以损失函数举例，损失函数的输出值(loss)越高表示差异越大，那么深度神经网络的训练就变成了尽可能缩小这个loss的过程。
图4是本申请实施例提供的一种系统架构100。
在图4中，数据采集设备160用于采集训练数据。针对本申请实施例的图像处理方法来说，可以通过训练数据对图像处理模型进行进一步训练，即数据采集设备160采集的训练数据可以是训练图像。
示例性地，在本申请实施例中训练图像处理模型的训练数据可以包括原始图像、与原 始图像对应的样本图像。训练数据还可以包括标签，例如表示原始图像是否有反光的标签。
例如，原始图像可以是指存在反光区域的图像；样本图像可以是指经过图像反光消除处理后的输出图像，比如，可以是指相对于原始图像而言反光区域得到部分或全部消除的图像。
在采集到训练数据之后，数据采集设备160将这些训练数据存入数据库130，训练设备120基于数据库130中维护的训练数据训练得到目标模型/规则101(即本申请实施例中的图像处理模型)。训练设备120将训练数据输入图像处理模型，直到训练图像处理模型输出的预测图像与样本图像之间的差值满足预设条件(例如，预测图像与样本图像差值小于一定阈值，或者预测图像与样本图像的差值保持不变或不再减少)，从而完成目标模型/规则101的训练。
示例性地，本申请实施例中用于执行图像处理方法的图像处理模型可以实现端到端的训练，比如，图像处理模型可以通过输入原始图像与输入原始图像对应的消除反光区域后的图像(例如，真值图像)实现端到端的训练。
在本申请提供的实施例中，该目标模型/规则101是通过训练图像处理模型得到的，即图像处理模型可以是反光判断和消除的模型。需要说明的是，在实际的应用中，所述数据库130中维护的训练数据不一定都来自于数据采集设备160的采集，也有可能是从其他设备接收得到的。
另外需要说明的是，训练设备120也不一定完全基于数据库130维护的训练数据进行目标模型/规则101的训练，也有可能从云端或其他地方获取训练数据进行模型训练，上述描述不应该作为对本申请实施例的限定。还需要说明的是，数据库130中维护的训练数据中的至少部分数据也可以用于执行设备110对待处理处理进行处理的过程。
根据训练设备120训练得到的目标模型/规则101可以应用于不同的系统或设备中，如应用于图4所示的执行设备110，所述执行设备110可以是终端，如手机终端，平板电脑，笔记本电脑，AR/VR，车载终端等，还可以是服务器或者云端等。
在图4中，执行设备110配置输入/输出(input/output，I/O)接口112，用于与外部设备进行数据交互，用户可以通过客户设备140向I/O接口112输入数据，所述输入数据在本申请实施例中可以包括：客户设备输入的待处理图像。
预处理模块113和预处理模块114用于根据I/O接口112接收到的输入数据(如待处理图像)进行预处理，在本申请实施例中，也可以没有预处理模块113和预处理模块114(也可以只有其中的一个预处理模块)，而直接采用计算模块111对输入数据进行处理。
在执行设备110对输入数据进行预处理，或者在执行设备110的计算模块111执行计算等相关的处理过程中，执行设备110可以调用数据存储系统150中的数据、代码等以用于相应的处理，也可以将相应处理得到的数据、指令等存入数据存储系统150中。
最后，I/O接口112将处理结果，如上述得到待处理原始图像的消除反光区域的图像，即将得到的输出图像返回给客户设备140，从而提供给用户。
值得说明的是，训练设备120可以针对不同的目标或称不同的任务，基于不同的训练数据生成相应的目标模型/规则101，该相应的目标模型/规则101即可以用于实现上述目标或完成上述任务，从而为用户提供所需的结果。
在图4中所示情况下，在一种情况下，用户可以手动给定输入数据，该手动给定可以 通过I/O接口112提供的界面进行操作。
另一种情况下，客户设备140可以自动地向I/O接口112发送输入数据，如果要求客户设备140自动发送输入数据需要获得用户的授权，则用户可以在客户设备140中设置相应权限。用户可以在客户设备140查看执行设备110输出的结果，具体的呈现形式可以是显示、声音、动作等具体方式。客户设备140也可以作为数据采集端，采集如图所示输入I/O接口112的输入数据及输出I/O接口112的输出结果作为新的样本数据，并存入数据库130。当然，也可以不经过客户设备140进行采集，而是由I/O接口112直接将如图所示输入I/O接口112的输入数据及输出I/O接口112的输出结果，作为新的样本数据存入数据库130。
值得注意的是，图4仅是本申请实施例提供的一种系统架构的示意图，图中所示设备、器件、模块等之间的位置关系不构成任何限制。例如，在图4中，数据存储系统150相对执行设备110是外部存储器，在其它情况下，也可以将数据存储系统150置于执行设备110中。
如图4所示，根据训练设备120训练得到目标模型/规则101，该目标模型/规则101在本申请实施例中可以是图像处理模型，具体的，本申请实施例提供的图像处理模型可以是深度神经网络，卷积神经网络，或者，可以是深度卷积神经网络等。
图5是本申请实施例提供的一种芯片的硬件结构，该芯片包括神经网络处理器300(neural-network processing unit，NPU)。该芯片可以被设置在如图4所示的执行设备110中，用以完成计算模块111的计算工作。该芯片也可以被设置在如图4所示的训练设备120中，用以完成训练设备120的训练工作并输出目标模型/规则101。
NPU 300作为协处理器挂载到主中央处理器(central processing unit，CPU)上，由主CPU分配任务。NPU 300的核心部分为运算电路303，控制器304控制运算电路303提取存储器(权重存储器或输入存储器)中的数据并进行运算。
在一些实现中，运算电路303内部包括多个处理单元(process engine,PE)。在一些实现中，运算电路303是二维脉动阵列；运算电路303还可以是一维脉动阵列或者能够执行例如乘法和加法这样的数学运算的其它电子线路。在一些实现中，运算电路303是通用的矩阵处理器。
举例来说，假设有输入矩阵A，权重矩阵B，输出矩阵C；运算电路303从权重存储器302中取矩阵B相应的数据，并缓存在运算电路303中每一个PE上；运算电路303从输入存储器301中取矩阵A数据与矩阵B进行矩阵运算，得到的矩阵的部分结果或最终结果，保存在累加器308(accumulator)中。
向量计算单元307可以对运算电路303的输出做进一步处理，如向量乘，向量加，指数运算，对数运算，大小比较等等。例如，向量计算单元307可以用于神经网络中非卷积/非FC层的网络计算，如池化(pooling)，批归一化(batch normalization)，局部响应归一化(local response normalization)等。
在一些实现种，向量计算单元能307将经处理的输出的向量存储到统一存储器306。例如，向量计算单元307可以将非线性函数应用到运算电路303的输出，例如累加值的向量，用以生成激活值。在一些实现中，向量计算单元307生成归一化的值、合并值，或二者均有。
在一些实现中，处理过的输出的向量能够用作到运算电路303的激活输入，例如用于在神经网络中的后续层中的使用。
统一存储器306用于存放输入数据以及输出数据。权重数据直接通过存储单元访问控制器305(direct memory access controller，DMAC)将外部存储器中的输入数据存入到输入存储器301和/或统一存储器306、将外部存储器中的权重数据存入权重存储器302，以及将统一存储器306中的数据存入外部存储器。
总线接口单元310(bus interface unit，BIU)，用于通过总线实现主CPU、DMAC和取指存储器309之间进行交互。
与控制器304连接的取指存储器309(instruction fetch buffer)用于存储控制器304使用的指令；控制器304用于调用取指存储器309中缓存的指令，实现控制该运算加速器的工作过程。
一般地，统一存储器306，输入存储器301，权重存储器302以及取指存储器309均为片上(On-Chip)存储器，外部存储器为该NPU外部的存储器，该外部存储器可以为双倍数据率同步动态随机存储器(double data rate synchronous dynamic random access memory，DDR SDRAM)、高带宽存储器(high bandwidth memory，HBM)或其他可读可写的存储器。
上文中介绍的图4中的执行设备110能够执行本申请实施例的图像处理方法的各个步骤，图5所示的芯片也可以用于执行本申请实施例的图像处理方法的各个步骤。
本申请实施例提供的图像处理方法可部署在相关设备的计算节点上，通过软件算法能够有效的实现本申请实施例中的反光检测和去除的方法。下面结合图6对本申请实施例的系统架构进行详细描述。
如图6所示，本申请实施例的图像处理方法可以由图像反光检测与去除系统软件架构500实现，其中反光检测与去除系统软件架构可以包括：图像预处理模块510，网络推理模块520，图像后处理模块530和软件处理模块540。下面对各个模块进行详细说明。
图像预处理模块510，用于对原始图像进行预处理，例如将原始的高清大图处理成固定大小的小图。图像预处理模块510的输入为原始图像，输出为固定大小的小图，该小图可以作为网络推理模块520的输入图像，该输入图像的尺寸符和网络推理模块520的神经网络模型的输入尺寸。由于原始高清大图的往往具有较高的分辨率，例如4K分辨率，超出了图像反光检测与去除神经网络模型的输入尺寸，因此需要对原始图像进行预处理，使得处理后的图像能够适应图像反光检测与去除神经网络模型的输入。示例性地，可以用下采样，或者减均值，或者归一化等方式对原始图像进行预处理。示例性地，预处理后得到的输入图像为896×896像素的小图。应理解，预处理后得到的输入图像的尺寸还可以是其他符合图像反光检测与去除神经网络模型的输入要求的尺寸，本申请实施例对此不作限定。
图像预处理模块510是可选的模块。在一些实施例中，原始图像的尺寸符合网络推理模块520的神经网络模型的输入尺寸要求，在这种情况下，无需对原始图像进行预处理，输入网络推理模块520的输入图像为原始图像。
网络推理模块520，包括反光消除模型，用于对输入图像做初步反光消除处理。其中，反光消除模型的输入原始图像，或者输入原始图像经过图像预处理得到的输入图像，输出 为对输入图像进行反光消除后的图像(例如，下文所述的第一图像)。
在一些实施例中，网络推理模块520还可以包括反光判断模型，用于判断输入图像是否有反光。其中，反光判断模块的输入为第一图像和输入图像，输出为初步反光判断结果。例如，该判断结果为：输入图像有反光；或者，输入图像无反光。
一些反光消除模型对于输入图像的尺寸有限制。换句话说，此类反光消除模型只能处理分辨率较低的图像，例如896×896像素。在此情况下，可以通过图像预处理模块510将高清大图的原始图像(例如4K分辨率)预处理为小图像输入反光模型中。本申请的这种处理方案，使得即使使用低分辨率的反光消除模型，也能处理高清大图的反光问题。
图像后处理模块530，用于判断输入图像是否有反光，并根据判断结果对原始图像进行处理，输出消除反光的图像。
在一些实施例中，网络推理模块520包括反光判断模型，图像后处理模块530用于根据反光判断模型的判断结果确定输入图像是否有反光。当确定输入图像有反光时，对原始图像进行处理，输出消除反光的图像；否则，输出原始图像。
在一些实施例中，图像后处理模块530包括反光层强度判断单元，该反光层强度判断单元的输入为第一图像和输入图像，输出为判断输入图像是否有反光的判断值。图像后处理模块530还用于根据反光层强度判断单元的判断结果，确定输入图像是否有反光。当确定输入图像有反光时，对原始图像进行处理，输出消除反光的图像；否则，输出原始图像。
在一些实施例中，网络推理模块520包括反光判断模型，图像后处理模块530包括反光层强度判断单元和联合判断单元。图像后处理模块530用于根据反光判断模型的判断结果和反光层强度判断单元的判断结果判断输入图像是否有反光。当确定输入图像有反光时，对原始图像进行处理，输出消除反光的图像；否则，输出原始图像。
在一些实施例中，输入图像是原始图像。在这种情况下，当网络推理模块520的结果为输入图像有反光时，图像后处理模块530用于将第一图像和输入图像相减，得到反光层小图，在原始图像中剥离该反光层小图，得到输出图像。该输出图像是原始图像消除反光后得到的图像。
当网络推理模块520的结果为输入图像无反光时，图像后处理模块530无须对原始图像进行处理，在这种情况下，输出原始图像作为图像后处理模块530的输出图像。
在另一些实施例中，输入图像是原始图像经过上述预处理后得到的图像。
在这种情况下，当网络推理模块520的结果为输入图像有反光时，图像后处理模块530用于将第一图像和输入图像相减，得到反光层小图。图像后处理模块530还用于对该反光层小图进行上采样，得到反光图层，其中该反光图层的尺寸为原始图像的尺寸。图像后处理模块530从原始图像中剥离反光图层，得到原始图像消除反光后的输出图像。
当网络推理模块520的结果为输入图像无反光时，图像后处理模块530无须对原始图像进行处理，在这种情况下，输出图像为原始图像。
应理解，网络推理模块520判断为无反光的图像，可能是完全没有反光的图像，也可能是反光成分非常低从而不会影响图像质量的图像(例如反光判断得到的反光置信度低于一定阈值的图像)。对于这样的图像进行传统的反光消除处理，可能反而会降低图像质量。本申请实施例可以直接将这些原始图像作为输出图像，改善反光消除效果。
软件处理模块540，用于接收图像后处理模块530的输出图像，并向用户呈现该输出 图像。具体地，软件处理模块540主要起调度作用，控制设备从图库或者相机等处获取图像，控制启用图像预处理模块510、网络推理模块520、图像后处理模块530的运行，同时与用户交互，将有无反光结果和无反光图推荐给用户。
在一些实施例中，图像预处理模块510、图像后处理模块530以及软件处理模块540可以由图5所示的芯片的主CPU实现；网络推理模块520可以由图5所示的芯片的神经网络处理器300实现。
本申请的反光消除方案不是简单地对所有图像进行消除操作，而是会通过初步消除反光的图像进一步判断原图像中是否有反光，再根据判断结果提供最终输出图像。这样能够改善图像的反光处理效果。
例如，对于判断为无反光的图像或反光程度较低的图像，可以不须经过反光消除处理，以免引入更多的处理噪声反而降低图像质量。
另一方面，如果反光消除模型直接处理高清大图，模型的网络复杂度和设备的耗电量会显著增加，不适用于一些小型设备，例如移动终端。本申请实施例的反光消除模型可以保持较小的网络复杂度，并且通过网络推理模块的上采样等处理，使得高清大图(例如移动终端的摄像头拍摄得到的图片)的反光消除效果仍然较佳。
下面结合图7至图11详细描述本申请实施例的图像处理方法。
图7是本申请一实施例的图像处理方法700的示意图。该方法可以由图6所示的反光检测和去除系统软件架构实现。该方法还可以由能够进行图像处理的装置执行，例如，该方法可以由图4中的执行设备110执行。下面对该方法700中的步骤进行详细的描述。
S710，对原始图像进行预处理，获得符合反光消除模型输入尺寸的第一图像。步骤S710可以由图6所示的软件架构中的图像预处理模块510实现。具体来说，原始图像往往具有较大的尺寸或者较高的分辨率，例如4K分辨率。这样的原始图像的大小超出了反光消除模型的输入尺寸，因此需要对原始图像进行预处理。
示例性地，可以采用下采样的方式对原始图像进行处理，从高清原始图像采样生成输入图像。该输入图像可以是固定尺寸的小图，并且输入图像的尺寸符合反光消除模型的输入尺寸要求。例如，预处理后得到的输入图像的尺寸是896×896像素。应理解，预处理后得到的输入图像的尺寸还可以是其他符合反光消除模型的输入要求的尺寸，本申请实施例对此不作限定。
在一些实施例中，原始图像的尺寸符合反光消除模型的输入尺寸要求，因此可以不执行步骤S710中的图像预处理操作。在这种情况下，可以直接把原始图像作为输入图像，执行以下的步骤。
S720，将输入图像输入反光消除模型，获得反光消除后的第一图像。步骤S720可以由图6所示的软件架构中的网络推理模块520实现。在步骤720中，反光消除模型可以是通过生成对抗神经网络(generative adversarial networks，GAN)得到的生成网络，或者其他神经网络，本申请实施例对此不作限定。
S730，将第一图像和输入图像输入反光判断模型，获得反光判断模型的判断结果(例如，下文所述的第一判断值)。步骤S730可以由图6所示的软件架构中的网络推理模块520实现。在本申请的实施例中，反光判断神经网络模型可以是二分类神经网络模型，用来判断原始图像是否为有反光的图像。
示例性地，在一些实施例中，反光判断模型为基于马尔科夫判别器的二分类神经网络。图8是本申请一实施例的反光判断模型的示意图。
如图8所示，该反光判断模型由多个卷积层组成，卷积层可以扩大每个神经元的感受野。该马尔科夫判别器的输出为一个矩阵，矩阵中的每一个输出，代表着输入图片的一个感受野，对应了输入图片的一片(patch)。该反光判断模型的最后一个卷积层的输出为多个patch的预测结果。该反光判断模型的输入为第一图像和输入图像，输出为第一判断值。例如，第一判断值为输入图像有反光，或者第一判断值为输入图像无反光。
在一些实施例中，还可以在最后一个卷积层后增加一层全连接层，对多个patch的预测的结果进行非线性映射，得到一个反光置信系数。反光置信系数也可以称为反光置信度、反光系数等，用于表示图像存在反光成分的可能性，或者表示图像不存在反光成分的可能性。下面的实施例中，以反光置信系数表示图像存在反光成分的可能性为例进行说明。
在一些实施例中，可以根据反光置信系数的值确定第一判断值。例如，当反光置信系数的值大于特定阈值时，第一判断值为1，表示输入图像有反光；否则，第一判断值为0，表示输入图像无反光。或者，当置信系数的值小于特定阈值时，第一判断值为1，表示输入图像有反光；否则，第一判断值为0，表示输入图像无反光。第一判断值的具体取值和图像有无反光的对应关系不限于上述实例，可以根据需要进行各种合理的设置。
在另一些实施例中，可以将反光置信系数的值直接作为第一判断值。在此情况下，由反光判断模型的后续模块或单元根据该第一判断值，进一步确定图像中是否存在反光。
在一些实施例中，可以采用如下的方式对反光判断模型进行训练：
将成对的原始图像样本和原始图像样本消除反光后的结果，以及原始图像是否为反光图像的标签(label)输入反光判断模型进行训练。示例性的，如果原始图像为有反光图像，那么标签为1；如果原始图像为无反光图像，那么标签为0。反光判断模型本质上是一个二分类网络，用来区分样本的真伪，可以使用交叉熵来进行判别分布的相似性，损失(loss)函数为：
Figure PCTCN2020081191-appb-000011
其中，N为样本数；
x i表示第i对样本，即成对的原始图像和该图像消除反光后的结果；
y i表示x i的标签，正类为1，负类为0。示例性地，如上文所述，如果原始有反光，那么标签可以是1，如果原始图片无反光，标签可以是0；
D(x i)表示对样本x i预测的结果为上述“正类”的概率。
当loss函数的值小于阈值时，认为反光判断模型训练完成。
S740，根据第一判断值，对原始图像进行消除反光处理。步骤S740可以由图6所示的软件架构中的图像后处理模块530实现。在步骤S740中，输入为原始图像、第一图像、输入图像和步骤730得到的第一判断值；输出为原始图像消除反光后的图像。具体来说，根据反光判断结果(第一判断值)对原始图像进行消除反光处理可以分为以下几种情况：
情况1：输入图像有反光，并且输入图像为原始图像。
在这种情况下，可以将输入图像与第一图像相减，得到反光层小图。在一些实施例中，还可以对第一图像与输入图像相减的结果进行滤波，得到反光层小图。
例如，可以通过以下的计算方式得到反光层小图：
M blend-M low_trans＝M low_ref
其中，M blend为输入图像，M low_trans为第一图像，M low_ref为反光层小图。
示例性地，输入图像、第一图像都是三通道的RGB图，输入图像和第一图像的尺寸均为896×896像素，那么M blend和M low_trans均为896×896×3的矩阵。其中矩阵的每一个元素表示该图像的每一个像素点在红色(red，R)、绿色(green，G)、蓝色(blue，B)三个颜色通道的亮度值。
在这种情况下，输入图像和第一图像相减得到的M low_ref也是一个896×896×3，其中矩阵中的每一个元素表示反光层小图的每一个像素点在RGB三个颜色通道的亮度值。
由于输入图像的尺寸为原始图像的尺寸，因此反光层小图也为原始图像的尺寸。从原始图像中将反光层小图剥离，可以得到输出图像，该输出图像为原始图像消除反光后的图像。
例如，可以通过以下的计算方式得到原始图像消除反光后的图像：
M orig-M ref＝M trans
其中，M orig为原始图像，M ref为反光图层，M trans为原始图像消除反光图层后的结果。
在情况1所述的实施例中，由于输入图像的尺寸为原始图像的尺寸，反光层小图也为原始图像的尺寸，因此反光图层M ref与反光层小图M low_ref相等。在该实施例中，M orig和M ref都是896×896×3的矩阵，因此，将原始图像和反光图层相减得到的原始图像去除反光后的图像M trans也是一个896×896×3的矩阵，其中矩阵中的每一个元素表示原始图像消除反光后的图像中的每一个像素点在RGB三个颜色通道的亮度值。
情况2：输入图像有反光，并且输入图像为原始图像经过预处理后的图像。
在这种情况下，可以将输入图像与第一图像相减，得到反光层小图。在一些实施例中，还可以对第一图像与输入图像相减的结果进行滤波，得到反光层小图。
例如，可以通过以下的计算方式得到反光层小图：
M blend-M low_trans＝M low_ref
其中，M blend为输入图像，M low_trans为第一图像，M low_ref为反光层小图。
示例性地，输入图像、第一图像都是三通道的RGB图，输入图像和第一图像的尺寸均为896×896像素，那么M blend和M low_trans均为896×896×3的矩阵。其中矩阵的每一个元素表示图像的每一个像素点在RGB三个颜色通道的亮度值。
在这种情况下，输入图像和第一图像相减得到的M low_ref也是一个896×896×3，其中矩阵中的每一个元素表示反光层小图的每一个像素点在RGB三个颜色通道的亮度值。
由于输入图像是原始图像经过预处理后的图像，因此反光层小图的尺寸小于原始图像的尺寸。在这种情况下，可以对该反光层小图进行上采样，得到反光图层，该反光图层的尺寸为原始图像的尺寸。在一些实施例中，可以采用插值的方式对第一反光层小图进行上采样。从原始图像中将反光图层剥离，可以得到输出图像，该输出图像为原始图像消除反光后的图像。
例如，可以通过以下的计算方式得到原始图像消除反光后的图像：
M orig-M ref＝M trans
其中，M orig为原始图像，M ref为反光图层，M trans为原始图像消除反光图层后的结果。
在情况2所述的实施例中，输入图像为原始图像经过预处理后的图像，因此反光层小 图的尺寸小于原始图像的尺寸。
示例性地，原始图像的尺寸为4K(3000×4000像素)，M orig是一个3000×4000×3的矩阵。在这种情况下对反光层小图M low_ref采用差值的方式进行上采样，可以得到与原始图像尺寸一致的反光图层M ref，M ref也是一个3000×4000×3的矩阵，其中矩阵中的每一个元素表示反光图层的每一个像素点在RGB三个颜色通道的亮度值。
将原始图像M orig和反光图层相减M ref得到的原始图像消除反光后的图像M trans也是一个3000×4000×3的矩阵，其中矩阵中的每一个元素表示原始图像消除反光后的图像中的每一个像素点在RGB三个颜色通道的亮度值。
情况3：输入图像无反光。
在这种情况下，由于输入图像无反光，无需对原始图像进行处理。因此输出的输出图像为原始图像。
根据反光层强度判断模型，可以判断出图像中是否有反光，从而根据判断结果选择合适的反光处理方式，能够改善图像的反光处理效果。
本申请实施例提供的图像处理方法，一方面，可以自动识别原始图像是否有反光，并对有反光的原始图像进行反光消除处理。另一方面，本申请实施例的图像处理方法，在原始图像尺寸较大时，先通过原图下采样以适应神经网络的输入规模，又对反光层小图进行上采样，并从原图中剥离上采样后的反光图层，实现了高分辨率图像的反光消除。
此外，本申请实施例的图像处理方法，通过反光判断模型和反光消除模型两个轻量级神经网络实现反光的识别和消除功能，克服了高清图像所需的反光消除网络过大的问题。
图9是本申请另一实施例的图像处理方法900的示意图。该方法可以由图6所示的反光检测和去除系统软件架构实现。该方法还可以由能够进行图像处理的装置执行，例如，该方法可以由图4中的执行设备110执行。下面对该方法900中的步骤进行详细的描述。
S910，对原始图像进行预处理，获得符合反光消除模型输入尺寸的输入图像。步骤S910可以由图6所示的软件架构中的图像预处理模块510实现。
S920，将输入图像输入反光消除模型，获得反光消除后的第一图像。步骤S920可以由图6所示的软件架构中的网络推理模块520实现。
在方法900中，步骤S910至步骤S920与图7所示的方法700中的步骤S710至步骤S720相似，不再赘述。
S930，第一图像和输入图像输入反光层强度判断单元，获得反光层强度判断单元的判断结果(例如，下文所述的第二判断值)。步骤S930可以由图6所示的软件架构中的图像后处理模块530实现。在一些实施例中，可以通过第一图像与输入图像差值的灰度，判断输入图像是否有反光。
图10是反光层强度判断单元的判断方法示意图。
如图10所示，反光层强度判断单元的输入为第一图像和输入图像，输出为第二判断值：输入图像有反光，或者输入图像无反光。
反光层强度判断单元计算输入图像与第一图像之差的灰度值，该灰度值可以表征图像的亮度。
示例性地，输入图像和第一图像的尺寸为896×896像素，如上述方法700中步骤S740中所述，输入图像和第一图像相减可以得到一个896×896×3的矩阵表示反光层小图。其 中矩阵的每一个元素表示反光层小图的每一个像素点在RGB三个颜色通道的亮度值。
在一些实施例中，该灰度值可以根据如下公式计算得到：
Gray(i,j)＝0.299×R(i,j)+0.587×G(i,j)+0.144×B(i,j)
其中，Gray(i,j)表示反光层小图的每一个像素点的灰度值；
R(i,j)表示反光层小图每一个像素点的红色通道亮度值的绝对值，R(i,j)＝|M low_ref(i,j,0)|；
G(i,j)表示反光层小图每一个像素点的绿色通道亮度值的绝对值，G(i,j)＝|M low_ref(i,j,1)|；
B(i,j)表示反光层小图每一个像素点的蓝色通道亮度值的绝对值，B(i,j)＝|M low_ref(i,j,2)|。
应理解，本申请实施例还可以根据其他方法计算灰度值，例如取每一个像素点的RGB三个颜色通道亮度值的平均值作为每个像素点的灰度值等，本申请实施例对此不做限定。
在一些实施例中，可以根据反光层小图各个像素点灰度值的最大值、灰度值大于阈值TH1的像素点强度、灰度值大于阈值TH2的像素点面积中的至少一项或者多项的加权结果确定第二判断值。
其中，灰度值大于阈值TH1的像素点强度可以是灰度值大于阈值TH1的像素点的灰度值之和，或者，可以是灰度值大于阈值TH1的像素点的灰度值的平均值，还可以是其他用于表征强度的数值，本申请实施例对此不做限定。
其中，灰度值大于阈值TH2的像素点面积表示灰度值大于阈值TH2的像素点的个数。
示例性地，可以根据灰度大于阈值TH1的像素点强度大于阈值TH3，确定输入图像有反光。
示例性地，可以根据灰度的最大值与灰度大于阈值TH2的像素点面积的加权结果大于阈值TH3，确定输入图像有反光。
示例性地，可以根据灰度图的最大值、灰度大于阈值TH1的像素点强度、灰度大于阈值TH2的像素点面积的加权结果大于阈值TH3，确定输入图像有反光。
S940，根据第二判断值，对原始图像进行消除反光处理。步骤S940可以由图6所示的软件架构中的图像后处理模块530实现。
例如，假设第二判断值为1，表示输入图像有反光；否则，第二判断值为0，表示输入图像无反光。第二判断值的具体取值和图像有无反光的对应关系不限于上述实例，可以根据需要进行各种合理的设置。
具体来说，根据反光判断结果(第二判断值)对原始图像进行消除反光处理可以分为以下几种情况：
情况1：输入图像有反光，并且输入图像为原始图像。
在这种情况下，可以将第一图像与输入图像相减，得到反光层小图。在一些实施例中，还可以对第一图像与输入图像相减的结果进行滤波，得到反光层小图。由于输入图像的尺寸为原始图像的尺寸，因此反光层小图也为原始图像的尺寸。从原始图像中将反光层小图剥离，可以得到输出图像，该输出图像为原始图像消除反光后的图像。
在这种情况下，反光层小图、消除反光的原始图像的计算方式与上述方法700中步骤 S740中的情况1所述的计算方式相似，在此不再赘述。
情况2：输入图像有反光，并且输入图像为原始图像经过预处理后的图像。
在这种情况下，可以将第一图像与输入图像相减，得到反光层小图。在一些实施例中，还可以对第一图像与输入图像相减的结果进行滤波，得到反光层小图。由于输入图像是原始图像经过预处理后的图像，因此反光层小图的尺寸小于原始图像的尺寸。在这种情况下，可以对该反光层小图进行上采样，得到反光图层，该反光图层的尺寸为原始图像的尺寸。在一些实施例中，可以采用插值的方式对第一反光层小图进行上采样。从原始图像中将反光图层剥离，可以得到输出图像，该输出图像为原始图像消除反光后的图像。
在这种情况下，反光层小图、反光图层、消除反光的原始图像的计算方式与上述方法700中步骤S740中的情况2所述的计算方式相似，在此不再赘述。
情况3：输入图像无反光。
在这种情况下，由于输入图像无反光，无需对原始图像进行处理。因此输出的输出图像为原始图像。
根据反光层强度判断单元，可以判断出图像中是否有反光，从而根据判断结果选择合适的反光处理方式，能够改善图像的反光处理效果。
本申请实施例提供的图像处理方法，一方面，可以自动识别原始图像是否有反光，并对有反光的原始图像进行反光消除处理。另一方面，本申请实施例的图像处理方法，在原始图像尺寸较大时，先通过原图下采样以适应神经网络的输入规模，又对反光层小图进行上采样，并从原图中剥离上采样后的反光图层，实现了高分辨率图像的反光消除。
此外，本申请实施例的图像处理方法，通过反光层强度判断单元和轻量级的反光消除模型实现反光的识别和消除功能，克服了高清图像所需的反光消除网络过大的问题。
图11是本申请另一实施例的图像处理方法1100的示意图。该方法可以由图6所示的反光检测和去除系统软件架构实现。该方法还可以由能够进行图像处理的装置执行，例如，该方法可以由图4中的执行设备110执行。下面对该方法1100中的步骤进行详细的描述。
S1110，对原始图像进行预处理，获得符合反光消除模型输入尺寸的输入图像。步骤S1110可以由图6所示的软件架构中的图像预处理模块510实现。
S1120，将输入图像输入反光消除模型，获得反光消除后的第一图像。步骤S1120可以由图6所示的软件架构中的网络推理模块520实现。
在方法1100中，步骤S1110至步骤S1120与图7所示的方法700中的步骤S710至步骤S720相似，不再赘述。
S1130，第一图像和输入图像输入反光判断模型，获得第一判断值。步骤S1130可以由图6所示的软件架构中的网络推理模块520实现。
在方法1100中，步骤S1130与图7所示的方法700中的步骤S730相似，不再赘述。
S1140，第一图像和输入图像输入反光层强度判断单元，获得第二判断值。步骤S1140可以由图6所示的软件架构中的图像后处理模块530实现。
在方法1100中，步骤S1140与图9所示的方法900中的步骤S930相似，不再赘述。
S1150，根据第一判断值和第二判断值进行联合判断，获得反光判断结果。步骤S1150可以由图6所示的软件架构中的图像后处理模块530实现。
在一些实施例中，例如第一判断值可以表示输入图像是否有反光。此时，当第一判断 值和第二判断值都为有反光时，反光判断结果为有反光；当第一判断值和第二判断值中至少有一个结果为无反光时，反光判断结果为无反光。或者作为另一种实现方式，当第一判断值和第二判断值中至少有一个结果为有反光时，反光判断结果为有反光；当第一判断值和第二判断值均为无反光时，反光判断结果为无反光。
在另一些实施例中，例如第一判断值可以表示输入图像是否有反光的置信度(即上述反光置信度或反光置信系数)。此时，当第一判断值大于或等于特定阈值且第二判断值表示有反光时，步骤S1150输出的反光判断结果为有反光；反之，当第一判断值小于特定阈值和/或第二判断值表示无反光时，步骤S1150输出的反光判断结果为无反光。或者作为另一种实现方式，当第一判断值大于或等于特定阈值和/或第二判断值表示有反光时，步骤S1150输出的反光判断结果为有反光；反之，当第一判断值小于特定阈值且第二判断值表示无反光时，步骤S1150输出的反光判断结果为无反光。
或者，还可以对第一判断值进行进一步的区间细化，例如，当第一判断值大于阈值A1且第二判断值表示无反光时，反光判断结果为有反光；当第一判断值小于阈值A1且大于阈值A2，且第二判断值表示有反光时，反光判断结果为有反光；其他情况下，反光判断结果为无反光，其中A1大于A2。可以根据需要设计合适的联合判断算法，这些变化均在本申请实施例的范围内。
S1160，根据反光判断结果，对原始图像进行消除反光处理。步骤S1160可以由图6所示的软件架构中的图像后处理模块530实现。
在方法1100中，步骤S1140与图7所示的方法700中的步骤S740相似，不再赘述。
使用多种判断模型和单元联合判断的方式，可以比单一模型更准确地判断图像的反光情况，改善图像的反光处理效果。应理解，如果有其他合适的判断模型和/或判断单元，同样可以结合到本申请的反光判断模块或联合判断模式中。
图12是本申请实施例的图像处理装置的结构示意图。如图12所示，本申请实施例的图像处理装置1200包括第一反光消除模块1210、反光判断模块1220和第二反光消除模块1230。
第一反光消除模块1210，用于对输入图像进行反光消除处理，以得到第一图像。在一些实施例中，第一反光消除模块1210可以用于实现图7所示的方法700中的步骤S720、或者图9所示的方法900中步骤S920、或者图11所示的方法1100中步骤S1120的功能。第一反光消除模块1210的具体功能和有益效果可以参见上述方法中的描述，为了简洁，在此不再赘述。
反光判断模块1220，用于根据所述第一图像和所述输入图像，判断所述输入图像是否有反光。
在一些实施例中，反光判断模块1220可以包括反光判断模型，用于实现图7所示的方法700中的步骤S730的功能。
在另一些实施例中，反光判断模块1220可以包括反光层强度判断模型，用于实现图9所示的方法900中步骤S930的功能。
在另一些实施例中，反光判断模块1220可以包括反光判断模型、反光层强度判断模型和联合判断单元，用于实现图11所示的方法1100中步骤S1130至步骤S1150的功能。
反光判断模块1220的具体功能和有益效果可以参见上述方法中的描述，为了简洁， 在此不再赘述。
第二反光消除模块1230，用于根据所述反光判断模块的判断结果，获取输出图像。在一些实施例中，第二反光消除模块用于实现图7所示地方法700中步骤S740、或者图9所示的方法900中步骤S940、或者图11所示的方法1100中步骤S1160的功能。第二反光消除模块1230的具体功能和有益效果可以参见上述方法中的描述，为了简洁，在此不再赘述。
在一些实施例中，图像处理装置还可以包括预处理模块1240。预处理模块1240用于对原始图像进行预处理，生成符合第一反光输出模块1210的输入尺寸的输入图像。在一些实施例中，预处理模块1240用于实现图7所示的方法700中步骤S710、或者图9所示的方法900中步骤S910、或者图11所示的方法1100中步骤S1100的功能。预处理模块1240的具体功能和有益效果可以参见上述方法中的描述，为了简洁，在此不再赘述。
应理解，在一些实施例中，图12所示的图像处理装置可以由图5所示的芯片实现。其中，图5所示的芯片中的神经网络处理器300可以实现图12所示的图像处理装置的第一反光消除模块1210和反光判断模块1220中的反光判断模型的功能；图5所示的芯片中的主CPU可以实现反光判断模块1220中的反光层强度判断单元、联合判断单元以及第二反光消除模块1230的功能。在另一些实施例中，图5所示的芯片中的主CPU还可以实现预处理模块1240的功能。
在一些实施例中，图12所示的图像处理装置可以实现图6所示的软件架构的功能。其中，第一反光消除模块1210可以实现图6所示的软件架构的网络推理模块520中的反光消除模型的功能；反光判断模块1220中的反光判断模型可以实现图6所示的软件架构的网络推理模块中的反光判断模型的功能；反光判断模块1220中的反光层强度判断单元、联合判断单元以及第二反光消除模块1230可以实现图6所示的软件架构中的图像后处理模块530的功能。在另一些实施例中，预处理模块1240可以实现图6所示的软件架构的图像预处理模块的功能。
在一些实施例中，图12所示的图像处理装置可以部署在图4所示的执行设备110上，以实现各个模块的功能。
图13是本申请实施例的一种硬件结构示意图。应理解，本申请实施例的装置可以是一个或多个芯片，或者可以包括在终端或其他设备内。
如图13所示，本申请实施例的处理器系统1300包括中央处理器(central processing unit，CPU)1310、神经网络处理器(neutral processing unit，NPU)1320和图像信号处理器(image signal processing，ISP)1330。
上述方法中，图6所示的软件架构的图像预处理模块510、图像后处理模块530以及软件处理模块540可以应用于中央处理器1310，或者中央处理器1310实现。中央处理器1310还可以用于运行通用操作系统软件，并在通用操作系统软件的作用下控制神经网络处理器1320和图像信号处理器1330。上述方法中，图6所示的软件架构的网络推理模块520可以应用于神经网络处理器1320，或者由神经网络处理器1320实现。
图像信号处理器1330用于生成图像，该图像信号处理器可以匹配不同厂商的图像传感器，以用来对前端图像传感器输出的图像数据进行处理，根据图像数据生成对应的图像信号，例如上述的原始图片。在一些实施例中，图12所示的图像处理装置可以由图13所 示的处理器系统实现。其中，中央处理器1310可以实现反光判断模块1220中反光层强度判断单元和/或联合判断单元、第二反光消除模块1230的功能；在另一些实施例中，中央处理器1310还可以实现预处理模块1240的功能。
其中，神经网络处理器1320可以实现图12所示的图像处理装置中第一反光消除模块1210和反光判断模块1220中的反光判断模型的功能。
应理解，图13所示的处理器系统可以包括一个或多个图5所示的芯片，图5所示的芯片中的主CPU可以实现图13所示的处理器系统的中央处理器1310的功能；图5所示的芯片中的神经网络处理器300可以实现图13所示的处理器系统中的神经网络处理器1320的功能。
应理解，图13所示的处理器系统还可以部署在图4所示的执行设备110上，本申请实施例对此不做限定。
图14是本申请实施例提供的另一种图像处理装置的硬件结构示意图。如图14所示的图像处理装置1400包括存储器1410、处理器1420、通信接口1430以及总线1440。其中，存储器1410、处理器1420、通信接口1430通过总线1440实现彼此之间的通信连接。
存储器1410可以是只读存储器(read only memory，ROM)，静态存储设备，动态存储设备或者随机存取存储器(random access memory，RAM)。存储器1410可以存储程序，当存储器1410中存储的程序被处理器1420执行时，处理器1420用于执行本申请实施例的图像处理方法的各个步骤，例如，执行图7至图11所示的各个步骤。
应理解，本申请实施例所示的图像处理装置可以是服务器，例如，可以是云端的服务器，或者，也可以是配置于云端的服务器中的芯片；或者，本申请实施例所示的图像处理装置可以是智能终端，也可以是配置于智能终端中的芯片。
上述本申请实施例揭示的图像处理方法可以应用于处理器1420中，或者由处理器1420实现。处理器1420可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述图像处理方法的各步骤可以通过处理器1420中的硬件的集成逻辑电路或者软件形式的指令完成。例如，处理器1420可以是包含图5所示的NPU的芯片。
上述的处理器1420可以是中央处理器(central processing unit，CPU)、图形处理器(graphics processing unit，GPU)、通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现成可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存取存储器(random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)、可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器1410，处理器1420读取存储器1410中的指令，结合其硬件完成本申请实施中图6所示的的反光检测与消除系统软件架构或图12所示的图像处理装置中包括的模块所需执行的功能，或者，执行本申请方法实施例的图7至图11所示的图像处理方法的各个步骤。
通信接口1430使用例如但不限于收发器一类的收发装置，来实现装置1400与其他设 备或通信网络之间的通信。
总线1440可包括在图像处理装置1400各个部件(例如，存储器1410、处理器1420、通信接口1430)之间传送信息的通路。
应注意，尽管上述图像处理装置1400仅仅示出了存储器、处理器、通信接口，但是在具体实现过程中，本领域的技术人员应当理解，图像处理装置1400还可以包括实现正常运行所必须的其他器件。同时，根据具体需要本领域的技术人员应当理解，上述图像处理装置1400还可包括实现其他附加功能的硬件器件。此外，本领域的技术人员应当理解，上述图像处理装置1400也可仅仅包括实现本申请实施例所必须的器件，而不必包括图14中所示的全部器件。
本申请实施例还提供了一种计算机可读介质，该计算机可读介质存储有计算机程序(也可以称为代码，或指令)当其在计算机上运行时，使得计算机执行上述任一方法实施例中的方法。
本申请实施例还提供了一种芯片系统，包括存储器和处理器，该存储器用于存储计算机程序，该处理器用于从存储器中调用并运行该计算机程序，使得安装有该芯片系统的图像处理装置执行上述任一方法实施例中的方法。
其中，该芯片系统可以包括用于发送信息或数据的输入电路或者接口，以及用于接收信息或数据的输出电路或者接口。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在存储介质中，或者从一个存储介质向另一个存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(digital subscriber line，DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述存储介质可以是计算机能够存取的任何可用介质或者是包括一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质(例如，软盘、硬盘、磁带)、光介质(例如，高密度数字视频光盘(digital video disc，DVD))、或者半导体介质(例如，固态硬盘(solid state disk，SSD))等。
本申请中，“至少两种”或“多种”是指两种或两种以上。术语“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B的情况，其中A,B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。
本申请中的预定义可以理解为定义、预先定义、存储、预存储、预协商、预配置、固化、或预烧制。
应理解，说明书通篇中提到的“一个实施例”或“一实施例”意味着与实施例有关的特定特征、结构或特性包括在本申请的至少一个实施例中。因此，在整个说明书各处出现的“在一个实施例中”或“在一实施例中”未必一定指相同的实施例。此外，这些特定的特征、结构或特性可以任意适合的方式结合在一个或多个实施例中。应理解，在本申请的 各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
在本说明书中使用的术语“部件”、“模块”、“系统”等用于表示计算机相关的实体、硬件、固件、硬件和软件的组合、软件、或执行中的软件。例如，部件可以是但不限于，在处理器上运行的进程、处理器、对象、可执行文件、执行线程、程序和/或计算机。通过图示，在计算设备上运行的应用和计算设备都可以是部件。一个或多个部件可驻留在进程和/或执行线程中，部件可位于一个计算机上和/或分布在2个或更多个计算机之间。此外，这些部件可从在上面存储有各种数据结构的各种计算机可读介质执行。部件可例如根据具有一个或多个数据分组(例如来自与本地系统、分布式系统和/或网络间的另一部件交互的二个部件的数据，例如通过信号与其它系统交互的互联网)的信号通过本地和/或远程进程来通信。
本领域普通技术人员可以意识到，结合本文中所公开的实施例中描述的各方法步骤，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各实施例的步骤。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。本领域普通技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各种说明性逻辑块(illustrative logical block)和步骤(step)，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，可以理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (18)

  1. 一种图像处理装置，其特征在于，包括：
    第一反光消除模块，用于对输入图像进行反光消除处理，获取第一图像；
    反光判断模块，用于根据所述第一图像和所述输入图像，获取所述输入图像是否有反光的判断结果；
    第二反光消除模块，用于根据所述判断结果，获取输出图像。
  2. 根据权利要求1所述的装置，其特征在于，所述装置还包括：
    预处理模块，用于对原始图像进行预处理，以获取所述输入图像，其中所述原始图像的尺寸大于所述输入图像的尺寸。
  3. 根据权利要求1或2所述的装置，其特征在于，所述第二反光消除模块具体用于在所述判断结果为所述输入图像没有反光时，获取原始图像作为所述输出图像。
  4. 根据权利要求2所述的装置，其特征在于，所述第二反光消除模块具体用于在所述判断结果为所述输入图像有反光时，根据所述第一图像和所述输入图像生成反光层小图，对所述反光层小图进行上采样以获取反光图层，并根据所述反光图层和所述原始图像获取所述输出图像。
  5. 根据权利要求1至4中任一项所述的装置，其特征在于，所述反光判断模块包括反光判断模型，所述反光判断模型是以所述第一图像和所述输入图像作为输入的神经网络，所述反光判断模型的输出表示所述输入图像有反光的置信度，或者所述输出为所述判断结果。
  6. 根据权利要求5所述的装置，其特征在于，所述反光判断模型是基于马尔科夫判别器的神经网络，并且是基于第一样本集训练得到的，其中所述第一样本集包括：原始图像样本、所述原始图像样本经过反光消除后的图像样本、所述原始图像样本是否有反光的标签。
  7. 根据权利要求1至4中任一项所述的装置，其特征在于，所述反光判断模块包括反光层强度判断单元，所述反光层强度判断单元用于根据所述输入图像和所述第一图像生成反光层小图；所述反光层强度判断单元还用于计算所述反光层小图包括的多个像素点的灰度值，并根据所述多个像素点的灰度值确定所述判断结果。
  8. 根据权利要求7所述的装置，其特征在于，所述反光层强度判断单元根据如下信息中的至少一种或者多种的加权结果，确定所述判断结果：
    所述多个像素点的灰度值中的最大值、灰度值大于第一阈值的至少一个像素点的强度、灰度值大于第二阈值的至少一个像素点的面积；
    其中，所述灰度值大于第一阈值的至少一个像素点的强度为灰度值大于第一阈值的像素点的灰度值之和，或者，
    所述灰度值大于第一阈值的至少一个像素点的强度为灰度值大于第一阈值的像素点的灰度值的平均值；
    所述灰度值大于第二阈值的至少一个像素点的面积为灰度值大于第二阈值的像素点的个数。
  9. 根据权利要求1至4中任一项所述的装置，其特征在于，所述反光判断模块包括反光判断模型、反光层强度判断单元和联合判断单元，
    所述反光判断模型是以所述第一图像和所述输入图像作为输入的神经网络，所述反光判断模型的输出为表示所述输入图像是否有反光的第一判断值；
    所述反光层强度判断单元计算所述输入图像和所述第一图像之差的灰度值，并根据所述灰度值输出表示所述输入图像是否有反光的第二判断值；
    所述联合判断单元根据所述第一判断值和所述第二判断值，确定所述判断结果。
  10. 一种图像处理方法，其特征在于，包括：
    对输入图像进行反光消除处理，获取第一图像；
    根据所述输入图像和所述第一图像，判断所述输入图像是否有反光；
    根据所述输入图像是否有反光，获取输出图像。
  11. 根据权利要求10所述的方法，其特征在于，所述方法还包括：
    对原始图像进行预处理，以获取所述输入图像，所述原始图像的尺寸大于所述输入图像的尺寸。
  12. 根据权利要求10或11所述的方法，其特征在于，所述根据所述输入图像是否有反光，获取输出图像包括：当所述输入图像没有反光时，获取原始图像作为所述输出图像。
  13. 根据权利要求11所述的方法，其特征在于，所述根据所述输入图像是否有反光，获取输出图像包括：当所述输入图像有反光时，根据所述输入图像和所述第一图像生成反光层小图；
    对所述反光层小图进行上采样，获取反光图层；
    根据所述反光图层和所述原始图像获取所述输出图像。
  14. 根据权利要求10至13中任一项所述的方法，其特征在于，所述根据所述输入图像和所述第一图像，判断所述输入图像是否有反光包括：
    根据反光判断模型，判断所述输入图像是否有反光；
    所述反光判断模型是以所述第一图像和所述输入图像作为输入的神经网络，所述反光判断模型的输出表示所述输入图像有反光的置信度，或者所述输出为所述输入图像是否有反光的判断结果。
  15. 根据权利要求14所述的方法，其特征在于，所述反光判断模型是基于马尔科夫判别器的神经网络，并且是基于第一样本集训练得到的，其中所述第一样本集包括：原始图像样本、所述原始图像样本经过反光消除后的图像样本、所述原始图像样本是否有反光的标签。
  16. 根据权利要求10至13中任一项所述的方法，其特征在于，所述根据所述输入图像和所述第一图像，判断所述输入图像是否有反光包括：
    根据反光层强度判断单元，判断所述输入图像是否有反光；
    所述反光层强度判断单元用于计算根据所述输入图像和所述第一图像生成反光层小图；所述反光层强度判断单元还用于计算所述反光层小图包括的多个像素点的灰度值，并根据所述多个像素点的灰度值确定所述判断结果。
  17. 根据权利要求16所述的方法，其特征在于，所述反光层判断单元根据如下信息中的至少一种或者多种的加权结果，确定所述判断结果：
    所述多个像素点的灰度值中的最大值、灰度值大于第一阈值的至少一个像素点的强度、灰度值大于第二阈值的至少一个像素点的面积；
    其中，所述灰度值大于第一阈值的至少一个像素点的强度为灰度值大于第一阈值的像素点的灰度值之和，或者，
    所述灰度值大于第一阈值的至少一个像素点的强度为灰度值大于第一阈值的像素点的灰度值的平均值；
    其中，所述灰度值大于第二阈值的至少一个像素点的面积为灰度值大于第二阈值的像素点的个数。
  18. 根据权利要求10至13中任一项所述的方法，其特征在于，所述根据所述输入图像和所述第一图像，判断所述输入图像是否有反光包括：
    根据反光判断模型、反光层强度判断单元和联合判断单元，判断所述输入图像是否有反光；
    所述反光判断模型是以所述第一图像和所述输入图像作为输入的神经网络，所述反光判断模型的输出为表示所述输入图像是否有反光的第一判断值；
    所述反光层强度判断单元计算所述输入图像和所述第一图像之差的灰度值，并根据所述灰度值输出表示所述输入图像是否有反光的第二判断值；
    所述联合判断单元根据所述第一判断值和所述第二判断值，确定判断结果。
PCT/CN2020/081191 2020-03-25 2020-03-25 一种图像处理方法和装置 WO2021189321A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202080001382.3A CN113728355A (zh) 2020-03-25 2020-03-25 一种图像处理方法和装置
PCT/CN2020/081191 WO2021189321A1 (zh) 2020-03-25 2020-03-25 一种图像处理方法和装置

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/081191 WO2021189321A1 (zh) 2020-03-25 2020-03-25 一种图像处理方法和装置

Publications (1)

Publication Number Publication Date
WO2021189321A1 true WO2021189321A1 (zh) 2021-09-30

Family

ID=77891499

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/081191 WO2021189321A1 (zh) 2020-03-25 2020-03-25 一种图像处理方法和装置

Country Status (2)

Country Link
CN (1) CN113728355A (zh)
WO (1) WO2021189321A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115082477A (zh) * 2022-08-23 2022-09-20 山东鲁芯之光半导体制造有限公司 一种基于去反光效果的半导体晶圆加工质量检测方法

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019201336A1 (en) * 2018-04-19 2019-10-24 Shanghaitech University Light field based reflection removal
CN110827217A (zh) * 2019-10-30 2020-02-21 维沃移动通信有限公司 图像处理方法、电子设备及计算机可读存储介质

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019201336A1 (en) * 2018-04-19 2019-10-24 Shanghaitech University Light field based reflection removal
CN110827217A (zh) * 2019-10-30 2020-02-21 维沃移动通信有限公司 图像处理方法、电子设备及计算机可读存储介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115082477A (zh) * 2022-08-23 2022-09-20 山东鲁芯之光半导体制造有限公司 一种基于去反光效果的半导体晶圆加工质量检测方法
CN115082477B (zh) * 2022-08-23 2022-10-28 山东鲁芯之光半导体制造有限公司 一种基于去反光效果的半导体晶圆加工质量检测方法

Also Published As

Publication number Publication date
CN113728355A (zh) 2021-11-30

Similar Documents

Publication Publication Date Title
WO2021043168A1 (zh) 行人再识别网络的训练方法、行人再识别方法和装置
WO2021043273A1 (zh) 图像增强方法和装置
WO2021164731A1 (zh) 图像增强方法以及图像增强装置
US20220335583A1 (en) Image processing method, apparatus, and system
WO2021155792A1 (zh) 一种处理装置、方法及存储介质
CN110222717B (zh) 图像处理方法和装置
WO2021013095A1 (zh) 图像分类方法、图像分类模型的训练方法及其装置
WO2022001372A1 (zh) 训练神经网络的方法、图像处理方法及装置
EP4006776A1 (en) Image classification method and apparatus
WO2021073311A1 (zh) 图像识别方法、装置、计算机可读存储介质及芯片
CN113065645B (zh) 孪生注意力网络、图像处理方法和装置
WO2021018251A1 (zh) 图像分类方法及装置
WO2021047587A1 (zh) 手势识别方法、电子设备、计算机可读存储介质和芯片
CN112598597A (zh) 一种降噪模型的训练方法及相关装置
CN113326930A (zh) 数据处理方法、神经网络的训练方法及相关装置、设备
CN113011562A (zh) 一种模型训练方法及装置
WO2021175278A1 (zh) 一种模型更新方法以及相关装置
WO2024002211A1 (zh) 一种图像处理方法及相关装置
CN114359289A (zh) 一种图像处理方法及相关装置
CN113191489A (zh) 二值神经网络模型的训练方法、图像处理方法和装置
CN113284055A (zh) 一种图像处理的方法以及装置
WO2022179606A1 (zh) 一种图像处理方法及相关装置
CN115601820A (zh) 一种人脸伪造图像检测方法、装置、终端及存储介质
WO2021189321A1 (zh) 一种图像处理方法和装置
WO2021042774A1 (zh) 图像恢复方法、图像恢复网络训练方法、装置和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20926659

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20926659

Country of ref document: EP

Kind code of ref document: A1