CN113728355A

CN113728355A - Image processing method and device

Info

Publication number: CN113728355A
Application number: CN202080001382.3A
Authority: CN
Inventors: 王叶斌; 赵杨; 皮志明; 黄枭; 陈迪; 迟显军
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2020-03-25
Filing date: 2020-03-25
Publication date: 2021-11-30
Also published as: WO2021189321A1

Abstract

The application provides an image processing method and device, and the device comprises: the first reflection eliminating module is used for carrying out reflection eliminating processing on the input image to obtain a first image; the light reflection judging module is used for acquiring a judgment result of whether the input image has light reflection according to the first image and the input image; and the second reflection eliminating module is used for acquiring an output image according to the judgment result of the reflection judging module. The image processing device provided by the embodiment of the application can improve the reflection eliminating effect of the image.

Description

Image processing method and device

Technical Field

The present application relates to the field of image processing technologies, and in particular, to an image processing method and apparatus.

Background

Artificial Intelligence (AI) is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision and reasoning, human-computer interaction, recommendation and search, AI basic theory, and the like.

With the rapid development of artificial intelligence technology, neural networks (e.g., deep neural networks) have achieved great success in processing and analyzing various media signals such as images, videos, and voices in recent years. In daily photo shooting, reflection possibly exists in a shot image, in order to meet the requirement of a user on photo intelligent optimization processing, reflection eliminating operation can be carried out on the image by using an AI algorithm, and the influence of the reflection on a picture effect is reduced.

The existing light reflection eliminating technology is mainly realized by generating a antagonistic neural network (GAN). When the technology is used, original images with reflection are input into a trained reflection elimination generation network, and the network reasoning generates a non-reflection image. However, the GAN neural network algorithm only has a reflection elimination means but does not have a reflection identification means, and after the reflection elimination is performed on a non-reflection image, the image quality is deteriorated.

Disclosure of Invention

The application provides an image processing method and device, which can improve the reflection elimination effect of an image.

In a first aspect, an image processing apparatus is provided, including: the first reflection eliminating module is used for carrying out reflection eliminating processing on the input image to obtain a first image; the light reflection judging module is used for acquiring a judgment result whether the input image has light reflection or not according to the first image and the input image; and the second reflection eliminating module is used for acquiring an output image according to the judgment result.

Among the above-mentioned technical scheme, image processing apparatus can judge whether the input image has the reflection of light according to the result of preliminary reflection of light elimination, carries out reflection of light elimination processing to the original image that the judged result is for there being the reflection of light again, has realized having the automatic identification of reflection of light picture and reflection of light elimination.

Optionally, the apparatus may further include a preprocessing module, configured to preprocess an original image to obtain an input image, where a size of the original image is larger than a size of the input image.

Alternatively, when the size of the original image meets the input size requirement of the first reflection elimination module, the original image may be used as the input image.

According to the technical scheme, the original image is preprocessed, and the reflection identification and elimination of the original image with a large size can be realized.

In one implementation, the second reflection eliminating module is specifically configured to, when the determination result is that the input image is not reflected, obtain the original image as the output image.

When the judgment result is that the input image has no reflection, the original image can be directly output without performing redundant processing on the original image, and the situation that the image quality is deteriorated after the reflection of the reflection-free image is eliminated is avoided.

In an implementation manner, when the determination result is that the input image has reflection and the input image is an original image, the second reflection elimination module is specifically configured to generate a reflection layer small image according to the first image and the input image, and acquire an output image according to the reflection layer small image and the original image.

In an implementation manner, when the determination result is that the input image has a reflection and the image is an image obtained by preprocessing an original image when the image is input, the second reflection elimination module is specifically configured to generate a small reflection layer image according to the first image and the input image, perform upsampling on the small reflection layer image to obtain a reflection layer, and obtain an output image according to the reflection layer and the original image.

According to the technical scheme, the reflection layer is obtained by sampling the reflection layer small image, the output image is obtained according to the original image and the reflection layer, the reflection-eliminated output image which is consistent with the original image in size and resolution can be finally obtained, and the quality of the output image is improved.

Optionally, the reflection determination module includes a reflection determination model, the reflection determination model is a neural network that takes the first image and the input image as input, and an output of the reflection determination model indicates a confidence level that the input image has reflection, or an output of the reflection determination model is a determination result.

In one implementation, the glistening decision model is a markov arbiter-based neural network trained based on a first set of samples, wherein the first set of samples includes: the original image sample, the image sample after the original image sample is subjected to reflection elimination, and whether the original image sample has a reflective label.

Optionally, the reflection determination module includes a reflection layer strength determination unit, and the reflection layer strength determination unit is configured to generate a reflection layer small image according to the input image and the first image; the light reflecting layer strength judging unit is also used for calculating the gray values of a plurality of pixel points included in the light reflecting layer small image and determining the judging result according to the gray values of the plurality of pixel points.

In one implementation, the light-reflecting layer strength determination unit determines the determination result according to a weighted result of at least one or more of the following information: the maximum value of the gray values of the plurality of pixel points, the intensity of at least one pixel point with the gray value larger than a first threshold value and the area of at least one pixel point with the gray value larger than a second threshold value; the intensity of at least one pixel point with the gray value larger than the first threshold is the sum of the gray values of the pixel points with the gray value larger than the first threshold, or the intensity of at least one pixel point with the gray value larger than the first threshold is the average value of the gray values of the pixel points with the gray value larger than the first threshold; the area of at least one pixel point with the gray value larger than the second threshold is the number of the pixel points with the gray value larger than the second threshold.

Optionally, the reflection determination module includes a reflection determination model, a reflection layer strength determination unit, and a joint determination unit, where the reflection determination model is a neural network that takes the first image and the input image as inputs, and an output of the reflection determination model is a first determination value indicating whether the input image has reflection; the light reflecting layer strength judging unit calculates a gray value of the difference between the input image and the first image and outputs a second judging value for indicating whether the input image has light reflection or not according to the gray value; the joint judgment unit determines a judgment result according to the first judgment value and the second judgment value.

According to the technical scheme, whether the original image has reflection or not can be judged through various models or modules, and the accuracy of the judgment result is improved.

In a second aspect, an image processing method is provided, including: performing reflection elimination processing on an input image to obtain a first image; judging whether the input image has reflection or not according to the input image and the first image; and acquiring an output image according to whether the input image has reflection.

In one implementation, an original image may be pre-processed to obtain an input image, the original image having a size larger than a size of the input image.

Optionally, acquiring an output image according to whether the input image has reflection includes: when the input image is not backlit, the output image is the original image.

Optionally, acquiring an output image according to whether the input image has reflection includes: when the input image has reflection and is the original image, the method further comprises: generating a small image of a light reflecting layer according to the input image and the first image; and acquiring the output image according to the small image of the light reflecting layer and the original image.

Optionally, acquiring an output image according to whether the input image has reflection includes: when the input image has the light reflection and the input image is an image preprocessed by the original image, the method further comprises the following steps: generating a small image of a light reflecting layer according to the input image and the first image; the small reflective layer image is up-sampled to obtain a reflective layer; and acquiring the output image according to the reflection layer and the original image.

Optionally, the determining whether the input image reflects light according to the input image and the first image includes: judging whether the input image has reflection according to the reflection judgment model; the light reflection judgment model is a neural network taking a first image and an input image as input, and the output of the light reflection judgment model represents the confidence coefficient of the input image that the input image has light reflection or the output is the judgment result of whether the input image has light reflection.

Optionally, the determining whether the input image reflects light according to the input image and the first image includes: judging whether the input image has reflection according to the reflection layer strength judging unit; the reflecting layer strength judging unit is used for generating a reflecting layer small image according to the input image and the first image; the light reflecting layer strength judging unit is also used for calculating the gray values of a plurality of pixel points included in the light reflecting layer small image and determining the judging result according to the gray values of the plurality of pixel points.

In one implementation, the light-reflecting layer determining module determines the determination result according to a weighted result of at least one or more of the following information: the maximum value of the gray values of the plurality of pixel points, the intensity of at least one pixel point with the gray value larger than a first threshold value and the area of at least one pixel point with the gray value larger than a second threshold value; the intensity of at least one pixel point with the gray value larger than the first threshold is the sum of the gray values of the pixel points with the gray value larger than the first threshold, or the intensity of at least one pixel point with the gray value larger than the first threshold is the average value of the gray values of the pixel points with the gray value larger than the first threshold; the area of at least one pixel point with the gray value larger than the second threshold is the number of the pixel points with the gray value larger than the second threshold.

Optionally, the determining whether the input image reflects light according to the input image and the first image includes: judging whether the input image has reflection according to the reflection judgment model, the reflection layer strength judgment unit and the joint judgment unit; the light reflection judgment model is a neural network taking a first image and an input image as input, and the output of the light reflection judgment model is a first judgment value representing whether the input image has light reflection or not; the light reflecting layer strength judging unit calculates a gray value of the difference between the input image and the first image and outputs a second judging value for indicating whether the input image has light reflection or not according to the gray value; the joint judgment unit determines a judgment result according to the first judgment value and the second judgment value.

In a third aspect, a computer readable medium is provided, which stores program code for execution by a device, the program code comprising instructions for performing the second aspect or the image processing method in any one of the implementations of the second aspect.

In a fourth aspect, there is provided a computer program product comprising: computer program code for causing a computer to perform the second aspect or the image processing method in any of its implementations when said computer program code is run on a computer.

In a fifth aspect, a chip is provided, where the chip includes a processor and a data interface, and the processor reads instructions stored in a memory through the data interface to execute the image processing method in any implementation manner of the second aspect or the second aspect.

Optionally, as an implementation manner, the chip may further include a memory, the memory stores instructions, and the processor is configured to execute the instructions stored on the memory, and when the instructions are executed, the processor is configured to execute the image processing method in the second aspect or any one of the implementations of the second aspect.

In a sixth aspect, an apparatus is provided, comprising: a processor and a memory for storing the computer program code, which when run on the processor causes the apparatus to perform the second aspect or the image processing method in any of the implementations of the second aspect.

Drawings

Fig. 1 is a schematic view of an application scenario of an image processing method provided in an embodiment of the present application;

fig. 2 is a schematic diagram of an application scenario provided in an embodiment of the present application;

fig. 3 is a schematic diagram of another application scenario provided in an embodiment of the present application;

FIG. 4 is a schematic diagram of a system architecture provided by an embodiment of the present application;

FIG. 5 is a schematic diagram of a chip hardware structure provided in an embodiment of the present application;

FIG. 6 is a diagram illustrating a software architecture of a reflection detection and removal system according to an embodiment of the present disclosure;

fig. 7 is a schematic flowchart of an image processing method according to an embodiment of the present application;

fig. 8 is a schematic structural diagram of a reflection determination model according to an embodiment of the present disclosure;

FIG. 9 is a schematic flowchart of another image processing method provided in the embodiments of the present application;

fig. 10 is a schematic structural diagram of a reflection intensity determination model according to an embodiment of the present application;

FIG. 11 is a schematic flowchart of another image processing method provided in the embodiments of the present application;

fig. 12 is a schematic diagram of a hardware structure of an image processing apparatus according to an embodiment of the present application;

fig. 13 is a schematic diagram of a hardware structure of another image processing apparatus according to an embodiment of the present application;

fig. 14 is a schematic hardware configuration diagram of another image processing apparatus according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are examples of a part of this application and not of all embodiments.

Fig. 1 is a schematic view of an application scenario of an image processing method according to an embodiment of the present application.

As shown in fig. 1, the image processing method implemented by the present application may be applied to an intelligent terminal, for example, the method may perform image processing on an original image shot by a camera in an intelligent device to obtain a high-definition large image of the original image without light reflection in a picture, so as to improve the quality of image display. The smart terminal may be mobile or fixed, for example, the smart terminal may be a mobile phone with an image processing function, a Tablet Personal Computer (TPC), a media player, a smart tv, a Laptop Computer (LC), a Personal Digital Assistant (PDA), a Personal Computer (PC), a camera, a camcorder, a smart watch, an Augmented Reality (AR)/Virtual Reality (VR), a Wearable Device (WD), or an autonomous vehicle, and the present application is not limited thereto.

Specific application scenarios of the embodiments of the present application are illustrated below.

The application scene one: intelligent terminal photographing field

In one embodiment, as shown in fig. 2, the image processing method of the embodiment of the present application may be applied to shooting of a smart terminal device (e.g., a mobile phone). When an object is photographed through a camera of the intelligent terminal, under the scenes of museums, shopping malls and the like, the situation that a light reflecting area exists in the pictures of shooting glass and a show window is likely to happen. By the image method, namely the method for removing the reflection, the output image which is high in picture quality and removes the reflection area from the original image can be obtained.

Exemplarily, when the intelligent terminal performs real-time photographing through the image processing method of the embodiment of the application, down-sampling processing is performed on an original image obtained by photographing, and the original image is processed into a smaller image; performing reflection elimination processing on the processed small image, comparing the original small image with the reflection eliminated small image, and judging whether the original image has a reflection area; if the original image has a light reflection region, the original small image and the small image with the eliminated reflection are subjected to up-sampling to obtain a small image of a reflection layer, the small image of the reflection layer subjected to up-sampling is stripped from the original image to obtain a high-quality image with the eliminated reflection, and the output image is displayed on a screen of the intelligent terminal.

Exemplarily, when the intelligent terminal performs real-time photographing through the image processing method of the embodiment of the application, down-sampling processing is performed on an original image obtained by photographing, and the original image is processed into a smaller image; performing reflection elimination processing on the processed small image, comparing the original small image with the reflection eliminated small image, and judging whether the original image has a reflection area; if the original image has a light reflection region, the original small image and the small image with the eliminated reflection are subjected to up-sampling to obtain a small image of a reflection layer, the small image of the reflection layer subjected to up-sampling is stripped from the original image to obtain a high-quality image with the eliminated reflection, and the output image is stored in an album of the intelligent terminal.

Application scenario two: field of autonomous driving

In one embodiment, as shown in fig. 3, the image processing method of the embodiment of the present application may be applied to the field of automatic driving. For example, the method can be applied to a navigation system of an automatic driving vehicle, and the image processing method in the application can process the acquired image with the glass light reflecting area in the picture to eliminate the light reflecting area in the picture to obtain a high-quality non-light reflecting image in the navigation process of road driving of the automatic driving vehicle, so that the safety of the automatic driving vehicle is realized.

Illustratively, the present application provides an image processing method comprising: carrying out down-sampling processing on the original image obtained by shooting, and processing the original image into a smaller image; performing reflection elimination processing on the processed small image, comparing the original small image with the reflection eliminated small image, and judging whether the original image has a reflection area; if the original image has a light reflection region, the original small image and the small image with the reflection eliminated are subtracted to obtain a small image of the reflection layer, the small image of the reflection layer which is obtained by subtracting the original small image and the small image with the reflection eliminated is up-sampled, and the small image of the reflection layer which is up-sampled is stripped from the original image, so that a high-quality road image with the reflection eliminated is obtained.

Since the embodiments of the present application relate to the application of a large number of neural networks, for the sake of understanding, the following description will be made first of all with respect to terms and concepts of the neural networks to which the embodiments of the present application may relate.

(1) Neural network

The neural network may be composed of neural units, the neural units may refer to operation units with xs and intercept 1 as inputs, and the output of the operation units may be:

where s is 1, 2, … … n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is an activation function (activation functions) of the neural unit for introducing a nonlinear characteristic into the neural network to convert an input signal in the neural unit into an output signal. The output signal of the activation function may be used as an input for the next convolutional layer, and the activation function may be a sigmoid function. A neural network is a network formed by a plurality of the above-mentioned single neural units being joined together, i.e. the output of one neural unit may be the input of another neural unit. The input of each neural unit can be connected with the local receiving domain of the previous layer to extract the characteristics of the local receiving domain, and the local receiving domain can be a region composed of a plurality of neural units.

(2) Deep neural network

Deep Neural Networks (DNNs), also called multi-layer neural networks, can be understood as neural networks with multiple hidden layers. The DNNs are divided according to the positions of different layers, and the neural networks inside the DNNs can be divided into three categories: input layer, hidden layer, output layer. Generally, the first layer is an input layer, the last layer is an output layer, and the middle layers are hidden layers. The layers are all connected, that is, any neuron of the ith layer is necessarily connected with any neuron of the (i + 1) th layer.

Although DNN appears complex, it is not really complex in terms of the work of each layer, simply the following linear relational expression:

wherein the content of the first and second substances,

is the input vector of the input vector,

is the output vector of the output vector,

is an offset vector, W is a weight matrix (also called coefficient), and α () is an activation function. Each layer is only for the input vector

Obtaining the output vector through such simple operation

Due to the large number of DNN layers, the coefficient W and the offset vector

The number of the same is also large. The definition of these parameters in DNN is as follows: taking coefficient W as an example: assume that in a three-layer DNN, the linear coefficients of the 4 th neuron of the second layer to the 2 nd neuron of the third layer are defined as

The superscript 3 represents the number of layers in which the coefficient W is located, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

In summary, the coefficients from the kth neuron at layer L-1 to the jth neuron at layer L are defined as

Note that the input layer is without the W parameter. In deep neural networks, more hidden layers make the network more able to depict complex situations in the real world. Theoretically, the more parameters the higher the model complexity, the larger the "capacity", which means that it can accomplish more complex learning tasks. The final goal of the process of training the deep neural network, i.e., learning the weight matrix, is to obtain the weight matrix (the weight matrix formed by the vectors W of many layers) of all the layers of the deep neural network that is trained.

(3) Convolutional neural network

A Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. The convolutional neural network comprises a feature extractor consisting of convolutional layers and sub-sampling layers, which can be regarded as a filter. The convolutional layer is a neuron layer for performing convolutional processing on an input signal in a convolutional neural network. In convolutional layers of convolutional neural networks, one neuron may be connected to only a portion of the neighbor neurons. In a convolutional layer, there are usually several characteristic planes, and each characteristic plane may be composed of several neural units arranged in a rectangular shape. The neural units of the same feature plane share weights, where the shared weights are convolution kernels. Sharing weights may be understood as the way image information is extracted is location independent. The convolution kernel can be initialized in the form of a matrix of random size, and can be learned to obtain reasonable weights in the training process of the convolutional neural network. In addition, sharing weights brings the direct benefit of reducing connections between layers of the convolutional neural network, while reducing the risk of overfitting.

(4) Generative countermeasure network

Generative Adaptive Networks (GAN) is a deep learning model. The model comprises at least two modules: one module is a Generative Model (Generative Model), the other module is a discriminant Model (discriminant Model), and the two modules are used for mutually game learning, so that better output is generated. The generative model and the discriminant model may be both neural networks, specifically, deep neural networks, or convolutional neural networks. The basic principle of GAN is as follows: taking the GAN for generating an image as an example, assume that there are two networks, G (generator) and d (discriminator), where G is a network for generating an image that receives a random noise z from which the image is generated, denoted G (z); d is a discrimination network for discriminating whether an image is "real". The input parameter is x, x represents an image, and the output D (x) represents the probability that x is a real image, if the probability is 1, 100% of the real image is represented, and if the probability is 0, the probability is not possible to be the real image. In the course of training the generative countermeasure network, the goal of the generative network G is to generate as real images as possible to deceive the discrimination network D, and the goal of the discrimination network D is to distinguish the images generated by G from the real images as much as possible. Thus, G and D constitute a dynamic "gaming" process, i.e., "play" in a "generative play network". As a result of the last game, in an ideal state, G can generate enough "spurious" images G (z), and D has difficulty in determining whether the images generated by G are true or not, i.e., D (G (z)) is 0.5. This results in an excellent generative model G which can be used to generate images.

(5) Loss function

In the process of training the deep neural network, because the output of the deep neural network is expected to be as close to the value really expected to be predicted as possible, the weight vector of each layer of the neural network can be updated according to the difference between the predicted value of the current network and the really expected target value (of course, an initialization process is usually carried out before the first updating, namely parameters are preset for each layer in the deep neural network), for example, if the predicted value of the network is high, the weight vector is adjusted to be lower, and the adjustment is continuously carried out until the deep neural network can predict the really expected target value or the value which is very close to the really expected target value. Therefore, it is necessary to define in advance "how to compare the difference between the predicted value and the target value", which are loss functions (loss functions) or objective functions (objective functions), which are important equations for measuring the difference between the predicted value and the target value. Taking the loss function as an example, if the higher the output value (loss) of the loss function indicates the larger the difference, the training of the deep neural network becomes the process of reducing the loss as much as possible.

Fig. 4 is a system architecture 100 according to an embodiment of the present application.

In fig. 4, a data acquisition device 160 is used to acquire training data. For the image processing method of the embodiment of the present application, the image processing model may be further trained by training data, that is, the training data acquired by the data acquisition device 160 may be a training image.

For example, the training data for training the image processing model in the embodiment of the present application may include a raw image, and a sample image corresponding to the raw image. The training data may also include labels, such as labels indicating whether the original image has glistenings.

For example, the original image may refer to an image in which a light reflection region exists; the sample image may refer to an output image subjected to image reflection elimination processing, for example, an image in which a reflection area is partially or completely eliminated with respect to an original image.

After the training data is collected, the data collection device 160 stores the training data in the database 130, and the training device 120 trains the target model/rule 101 (i.e., the image processing model in the embodiment of the present application) based on the training data maintained in the database 130. The training apparatus 120 inputs training data into the image processing model until a difference between a prediction image output by the training image processing model and a sample image satisfies a preset condition (for example, the difference between the prediction image and the sample image is less than a certain threshold, or the difference between the prediction image and the sample image remains unchanged or no longer decreases), thereby completing training of the target model/rule 101.

For example, the image processing model for performing the image processing method in the embodiment of the present application may implement end-to-end training, for example, the image processing model may implement end-to-end training by inputting an original image and an image (e.g., a true value image) of the input original image corresponding to the reflection region eliminated.

In the embodiments provided herein, the target model/rule 101 is obtained by training an image processing model, i.e., the image processing model may be a model for determination and elimination of light reflection. It should be noted that, in practical applications, the training data maintained in the database 130 may not necessarily all come from the acquisition of the data acquisition device 160, and may also be received from other devices.

It should be noted that, the training device 120 does not necessarily perform the training of the target model/rule 101 based on the training data maintained by the database 130, and may also obtain the training data from the cloud or other places for performing the model training. It should also be noted that at least a portion of the training data maintained in database 130 may also be used to perform the process of device 110 on the to-be-processed process.

The target model/rule 101 obtained by training according to the training device 120 may be applied to different systems or devices, for example, the execution device 110 shown in fig. 4, where the execution device 110 may be a terminal, such as a mobile phone terminal, a tablet computer, a notebook computer, an AR/VR, a vehicle-mounted terminal, and may also be a server or a cloud.

In fig. 4, the execution device 110 configures an input/output (I/O) interface 112 for data interaction with an external device, and a user may input data to the I/O interface 112 through the client device 140, where the input data may include: the image to be processed is input by the client device.

The preprocessing module 113 and the preprocessing module 114 are configured to perform preprocessing according to input data (such as an image to be processed) received by the I/O interface 112, and in this embodiment of the application, the preprocessing module 113 and the preprocessing module 114 may not be provided (or only one of the preprocessing modules may be provided), and the computing module 111 may be directly used to process the input data.

In the process that the execution device 110 preprocesses the input data or in the process that the calculation module 111 of the execution device 110 executes the calculation or other related processes, the execution device 110 may call the data, the code, and the like in the data storage system 150 for corresponding processes, and may store the data, the instruction, and the like obtained by corresponding processes in the data storage system 150.

Finally, the I/O interface 112 returns the processing result, i.e., the image with the reflection-eliminated area of the original image to be processed, to the client apparatus 140, so as to provide the user with the resultant output image.

It should be noted that the training device 120 may generate corresponding target models/rules 101 for different targets or different tasks based on different training data, and the corresponding target models/rules 101 may be used to achieve the targets or complete the tasks, so as to provide the user with the required results.

In the case shown in fig. 4, in one case, the user may manually give the input data, which may be operated through an interface provided by the I/O interface 112.

Alternatively, the client device 140 may automatically send the input data to the I/O interface 112, and if the client device 140 is required to automatically send the input data to obtain authorization from the user, the user may set the corresponding permissions in the client device 140. The user can view the result output by the execution device 110 at the client device 140, and the specific presentation form can be display, sound, action, and the like. The client device 140 may also serve as a data collection terminal, collecting input data of the input I/O interface 112 and output results of the output I/O interface 112 as new sample data, and storing the new sample data in the database 130. Of course, the input data inputted to the I/O interface 112 and the output result outputted from the I/O interface 112 as shown in the figure may be directly stored in the database 130 as new sample data by the I/O interface 112 without being collected by the client device 140.

It should be noted that fig. 4 is only a schematic diagram of a system architecture provided in the embodiment of the present application, and the position relationship between the devices, modules, and the like shown in the diagram does not constitute any limitation. For example, in FIG. 4, the data storage system 150 is an external memory with respect to the execution device 110, in other cases, the data storage system 150 may be disposed in the execution device 110.

As shown in fig. 4, a target model/rule 101 is obtained by training according to a training device 120, where the target model/rule 101 may be an image processing model in this embodiment, and specifically, the image processing model provided in this embodiment may be a deep neural network, a convolutional neural network, or may be a deep convolutional neural network.

Fig. 5 is a hardware structure of a chip provided in an embodiment of the present application, where the chip includes a neural-Network Processing Unit (NPU) 300. The chip may be provided in the execution device 110 as shown in fig. 4 to complete the calculation work of the calculation module 111. The chip may also be disposed in the training apparatus 120 as shown in fig. 4 to complete the training work of the training apparatus 120 and output the target model/rule 101.

The NPU 300 is mounted as a coprocessor on a main processing unit (CPU), and tasks are allocated by the main CPU. The core portion of the NPU 300 is an arithmetic circuit 303, and the controller 304 controls the arithmetic circuit 303 to extract data in a memory (weight memory or input memory) and perform an operation.

In some implementations, the arithmetic circuitry 303 includes a plurality of processing units (PEs) internally. In some implementations, the operational circuitry 303 is a two-dimensional systolic array; the arithmetic circuit 303 may also be a one-dimensional systolic array or other electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry 303 is a general-purpose matrix processor.

For example, assume there are an input matrix a, a weight matrix B, and an output matrix C; the arithmetic circuit 303 fetches the data corresponding to the matrix B from the weight memory 302 and buffers the data on each PE in the arithmetic circuit 303; the arithmetic circuit 303 takes the matrix a data from the input memory 301 and performs matrix operation with the matrix B, and a partial result or a final result of the obtained matrix is stored in an accumulator 308 (accumulator).

The vector calculation unit 307 may further process the output of the operation circuit 303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, magnitude comparison, and the like. For example, the vector calculation unit 307 may be used for network calculation of a non-convolution/non-FC layer in a neural network, such as pooling (Pooling), batch normalization (batch normalization), local response normalization (local response normalization), and the like.

In some implementations, the vector calculation unit 307 can store the processed output vector to the unified memory 306. For example, the vector calculation unit 307 may apply a non-linear function to the output of the arithmetic circuit 303, such as a vector of accumulated values, to generate the activation value. In some implementations, the vector calculation unit 307 generates normalized values, combined values, or both.

In some implementations, the vector of processed outputs can be used as activation inputs to the arithmetic circuitry 303, for example, for use in subsequent layers in a neural network.

The unified memory 306 is used to store input data as well as output data. The weight data directly passes through a memory unit access controller 305 (DMAC) to store the input data in the external memory into the input memory 301 and/or the unified memory 306, store the weight data in the external memory into the weight memory 302, and store the data in the unified memory 306 into the external memory.

A bus interface unit 310 (BIU) for implementing interaction between the main CPU, the DMAC, and the instruction fetch memory 309 through a bus.

An instruction fetch buffer 309(instruction fetch buffer) connected to the controller 304 is used for storing instructions used by the controller 304; the controller 304 is configured to call an instruction cached in the instruction fetch memory 309, so as to control the operation process of the operation accelerator.

Generally, the unified memory 306, the input memory 301, the weight memory 302, and the instruction fetch memory 309 are On-Chip (On-Chip) memories, the external memory is a memory outside the NPU, and the external memory may be a double data rate synchronous dynamic random access memory (DDR SDRAM), a High Bandwidth Memory (HBM), or other readable and writable memories.

The execution device 110 in fig. 4 described above is capable of executing the steps of the image processing method according to the embodiment of the present application, and the chip shown in fig. 5 may also be used for executing the steps of the image processing method according to the embodiment of the present application.

The image processing method provided by the embodiment of the application can be deployed on the computing nodes of the related equipment, and the method for detecting and removing the reflected light in the embodiment of the application can be effectively realized through a software algorithm. The system architecture of the embodiment of the present application is described in detail below with reference to fig. 6.

As shown in fig. 6, the image processing method according to the embodiment of the present application may be implemented by an image reflection detection and removal system software architecture 500, where the image reflection detection and removal system software architecture may include: an image pre-processing module 510, a network inference module 520, an image post-processing module 530, and a software processing module 540. Each block will be described in detail below.

An image preprocessing module 510, configured to perform preprocessing on the original image, for example, process an original high-definition large image into a small image with a fixed size. The input of the image preprocessing module 510 is a raw image, and the output is a small graph with a fixed size, which can be used as an input image of the network inference module 520, and the size of the input image corresponds to the input size of the neural network model of the network inference module 520. Because the original high-definition large image often has a higher resolution, for example, a resolution of 4K, which exceeds the input size of the image reflection detection and neural network model removal, the original image needs to be preprocessed, so that the processed image can adapt to the input of the image reflection detection and neural network model removal. Illustratively, the original image may be preprocessed by down-sampling, or by averaging, or by normalization, etc. Illustratively, the input image resulting after preprocessing is a small graph of 896 × 896 pixels. It should be understood that the size of the input image obtained after the preprocessing may also be other sizes that meet the input requirement of the image reflection detection and neural network model removal, which is not limited in the embodiment of the present application.

The image pre-processing module 510 is an optional module. In some embodiments, the raw image size conforms to the input size requirements of the neural network model of the network inference module 520, in which case the input image to the network inference module 520 is the raw image without pre-processing the raw image.

The network inference module 520 includes a reflection elimination model for performing preliminary reflection elimination processing on the input image. The input original image of the reflection elimination model or the input image obtained by preprocessing the input original image is output as an image (for example, a first image described below) obtained by reflection elimination of the input image.

In some embodiments, the network inference module 520 may further include a reflection determination model for determining whether the input image has reflection. The input of the reflection judgment module is a first image and an input image, and the output is a preliminary reflection judgment result. For example, the determination result is: the input image has reflection; alternatively, the input image is non-reflective.

Some reflection elimination models have limitations on the size of the input image. In other words, such a reflection elimination model can only process images with a lower resolution, for example, 896 × 896 pixels. In this case, the raw image (e.g., 4K resolution) of the high-definition large image may be preprocessed into a small image by the image preprocessing module 510 and input into the glistening model. The processing scheme of the application enables the problem of reflection of high-definition large images to be processed even if a low-resolution reflection elimination model is used.

And an image post-processing module 530, configured to determine whether the input image has reflection, process the original image according to the determination result, and output an image with reflection removed.

In some embodiments, the network inference module 520 includes a reflection determination model, and the image post-processing module 530 is configured to determine whether the input image has reflection according to a determination result of the reflection determination model. When the input image is determined to have reflection, processing the original image and outputting an image with reflection eliminated; otherwise, the original image is output.

In some embodiments, the image post-processing module 530 includes a reflective layer strength determining unit, which inputs the first image and the input image and outputs a determination value for determining whether the input image has reflection. The image post-processing module 530 is further configured to determine whether the input image has reflection according to the determination result of the reflection layer strength determination unit. When the input image is determined to have reflection, processing the original image and outputting an image with reflection eliminated; otherwise, the original image is output.

In some embodiments, the network inference module 520 includes a glistening judgment model, and the image post-processing module 530 includes a glistening layer strength judgment unit and a joint judgment unit. The image post-processing module 530 is configured to determine whether the input image has reflection according to the determination result of the reflection determination model and the determination result of the reflection layer strength determination unit. When the input image is determined to have reflection, processing the original image and outputting an image with reflection eliminated; otherwise, the original image is output.

In some embodiments, the input image is an original image. In this case, when the result of the network inference module 520 is that the input image has a reflection, the image post-processing module 530 is configured to subtract the first image from the input image to obtain a reflection layer thumbnail, and strip the reflection layer thumbnail from the original image to obtain an output image. The output image is the original image after reflection is eliminated.

When the result of the network inference module 520 is that the input image is not back-illuminated, the image post-processing module 530 does not need to process the original image, in which case the original image is output as the output image of the image post-processing module 530.

In other embodiments, the input image is an original image that has been pre-processed as described above.

In this case, when the result of the network inference module 520 is that the input image has a reflection, the image post-processing module 530 is configured to subtract the first image from the input image to obtain a reflection layer small map. The image post-processing module 530 is further configured to perform upsampling on the small reflective layer image to obtain a reflective layer, where the size of the reflective layer is the size of the original image. The image post-processing module 530 strips the reflection layer from the original image to obtain an output image of the original image after reflection is eliminated.

When the result of the network inference module 520 is that the input image is not back-illuminated, the image post-processing module 530 does not need to process the original image, in which case the output image is the original image.

It should be appreciated that the network inference module 520 may determine that the image is non-reflective, may be an image without total reflection, or may be an image with a very low reflective component that does not affect the image quality (e.g., an image with a reflective confidence below a certain threshold value obtained by the reflective determination). Conventional reflection removal processing for such an image may adversely degrade image quality. According to the embodiment of the application, the original images can be directly used as output images, and the reflection elimination effect is improved.

A software processing module 540 for receiving the output image of the image post-processing module 530 and presenting the output image to the user. Specifically, the software processing module 540 mainly plays a role in scheduling, controls the device to acquire images from a gallery or a camera, and the like, controls the starting of the operations of the image preprocessing module 510, the network reasoning module 520 and the image post-processing module 530, interacts with the user, and recommends the result of existence of glistenings and the glistenings to the user.

In some embodiments, the image pre-processing module 510, the image post-processing module 530, and the software processing module 540 may be implemented by the main CPU of the chip shown in fig. 5; the network inference module 520 may be implemented by the neural network processor 300 of the chip shown in fig. 5.

According to the reflection eliminating scheme, not all images are simply eliminated, whether reflection exists in the original image or not can be further judged through the image which preliminarily eliminates reflection, and then a final output image is provided according to the judgment result. This can improve the effect of the light reflection processing of the image.

For example, for an image determined to be non-reflective or less reflective, the reflection elimination process is not required, so as to avoid introducing more processing noise and reducing the image quality.

On the other hand, if the reflection elimination model directly processes a high-definition large image, the network complexity of the model and the power consumption of the device are significantly increased, and the model is not suitable for some small devices such as mobile terminals. The reflection elimination model of the embodiment of the application can keep smaller network complexity, and the reflection elimination effect of a high-definition large image (such as a picture obtained by shooting through a camera of a mobile terminal) is still better through processing such as up-sampling of the network reasoning module.

The image processing method according to the embodiment of the present application is described in detail below with reference to fig. 7 to 11.

Fig. 7 is a diagram illustrating an image processing method 700 according to an embodiment of the present application. The method may be implemented by the reflectance detection and removal system software architecture shown in fig. 6. The method may also be performed by an apparatus capable of image processing, for example, the method may be performed by the execution device 110 in fig. 4. The steps in the method 700 are described in detail below.

S710, preprocessing the original image to obtain a first image which accords with the input size of the reflection elimination model. Step S710 may be implemented by the image preprocessing module 510 in the software architecture shown in fig. 6. In particular, the original image tends to have a larger size or a higher resolution, for example, 4K resolution. The size of such an original image exceeds the input size of the reflection elimination model, and thus the original image needs to be preprocessed.

For example, the original image may be processed in a downsampling manner to generate the input image from the high-definition original image samples. The input image may be a small image of fixed size, and the size of the input image conforms to the input size requirements of the reflection elimination model. For example, the size of the input image obtained after the preprocessing is 896 × 896 pixels. It should be understood that the size of the input image obtained after the preprocessing may also be other sizes that meet the input requirement of the reflection elimination model, which is not limited in this embodiment of the application.

In some embodiments, the size of the original image conforms to the input size requirement of the reflection elimination model, and thus the image preprocessing operation in step S710 may not be performed. In this case, the following steps may be performed directly using the original image as an input image.

S720, inputting the input image into the reflection elimination model to obtain a first image after reflection elimination. Step S720 may be implemented by the network inference module 520 in the software architecture shown in fig. 6. In step 720, the reflection elimination model may be a generated network obtained by generating a countermeasure neural network (GAN), or other neural networks, which is not limited in this embodiment of the present application.

S730, the first image and the input image are input into the reflection determination model, and a determination result (for example, a first determination value described below) of the reflection determination model is obtained. Step S730 may be implemented by the network inference module 520 in the software architecture shown in fig. 6. In an embodiment of the present application, the reflection determination neural network model may be a two-class neural network model, and is used to determine whether the original image is a reflected image.

Illustratively, in some embodiments, the glint determination model is a Markov arbiter-based two-class neural network. Fig. 8 is a schematic diagram of a reflection determination model according to an embodiment of the present application.

As shown in fig. 8, the glistening judgment model is composed of a plurality of convolutional layers, which can enlarge the receptive field of each neuron. The output of the markov discriminator is a matrix, each output of the matrix representing a receptive field of the input picture corresponding to a slice (patch) of the input picture. The output of the last convolution layer of the reflection determination model is the prediction results of a plurality of patches. The input of the reflection judgment model is a first image and an input image, and the output is a first judgment value. For example, the first determination value is that the input image has reflection, or the first determination value is that the input image has no reflection.

In some embodiments, a fully connected layer may be added after the last convolutional layer, and the results of the predictions of the plurality of latches are non-linearly mapped to obtain a reflective confidence coefficient. The reflection confidence coefficient may also be referred to as reflection confidence, reflection coefficient, etc. and is used to indicate the possibility of the image having the reflection component or the possibility of the image having no reflection component. In the following examples, the probability that the image has a reflective component is represented by the reflective confidence coefficient.

In some embodiments, the first determination value may be determined based on the value of the reflective confidence coefficient. For example, when the value of the reflection confidence coefficient is greater than a specific threshold, the first judgment value is 1, which indicates that the input image has reflection; otherwise, the first judgment value is 0, which indicates that the input image has no reflection. Or when the value of the confidence coefficient is smaller than a specific threshold value, the first judgment value is 1, and the input image is reflected; otherwise, the first judgment value is 0, which indicates that the input image has no reflection. The specific value of the first determination value and the correspondence between the presence or absence of reflection in the image are not limited to the above examples, and various reasonable settings may be performed as needed.

In other embodiments, the value of the reflective confidence coefficient may be directly used as the first determination value. In this case, whether or not there is reflection in the image is further determined by a subsequent module or unit of the reflection determination model based on the first determination value.

In some embodiments, the glistening judgment model may be trained as follows:

and inputting the result of the original image sample and the original image sample in pair after reflection is eliminated and a label (label) of whether the original image is a reflection image into a reflection judgment model for training. Illustratively, if the original image is a reflected image, the label is 1; if the original image is a non-reflective image, the label is 0. The reflection judgment model is essentially a two-classification network, is used for distinguishing the authenticity of a sample, can use cross entropy to judge the similarity of distribution, and has a loss (loss) function as follows:

wherein N is the number of samples;

x _ito representThe ith pair of samples is the paired original image and the result of the image after the reflection is eliminated;

y _idenotes x_iThe positive class is 1, and the negative class is 0. Illustratively, as described above, if the original picture had glistenings, the label may be 1, and if the original picture had no glistenings, the label may be 0;

D(x _i) Represents to the sample x_iThe result of the prediction is the probability of the "positive class" described above.

And when the value of the loss function is smaller than the threshold value, the training of the reflection judgment model is considered to be finished.

And S740, according to the first judgment value, carrying out reflection elimination processing on the original image. Step S740 may be implemented by the image post-processing module 530 in the software architecture shown in fig. 6. In step S740, the original image, the first image, the input image, and the first determination value obtained in step 730 are input; and outputting the image with the reflection eliminated for the original image. Specifically, the reflection elimination processing of the original image according to the reflection determination result (first determination value) may be classified into the following cases:

case 1: the input image has reflection, and the input image is an original image.

In this case, the input image may be subtracted from the first image to obtain the retroreflective layer thumbnail. In some embodiments, the result of subtracting the input image from the first image may be filtered to obtain the reflection layer small map.

For example, the retroreflective layer small map can be obtained by the following calculation:

M _blend-M _{low_trans}＝M _{low_ref}

wherein M is_blendFor inputting an image, M_{low_trans}For the first image, M_{low_ref}Is a small picture of a light reflecting layer.

Illustratively, the input image, the first image are both three-channel RGB maps, the input image and the first image are both 896 × 896 pixels in size, then M is_blendAnd M_{low_trans}Are all 896X 896 x 3 matrix. Each element of the matrix represents the brightness value of each pixel point of the image in three color channels of red (R), green (G) and blue (B).

In this case, M is obtained by subtracting the input image and the first image_{low_ref}And is also 896 × 896 × 3, where each element in the matrix represents the brightness value of each pixel point of the reflection layer small graph in the RGB three color channels.

Since the size of the input image is the size of the original image, the retroreflective layer thumbnail is also the size of the original image. And stripping the small image of the light reflecting layer from the original image to obtain an output image, wherein the output image is the image of the original image after the light reflection is eliminated.

For example, the image of the original image after reflection is eliminated can be obtained through the following calculation method:

M _orig-M _ref＝M _trans

wherein M is_origFor the original image, M_refIs a reflective layer, M_transAnd eliminating the reflecting layer for the original image.

In the embodiment described in case 1, since the size of the input image is the size of the original image, and the size of the reflection layer small image is also the size of the original image, the reflection layer M_refAnd a reflection layer small picture M_{low_ref}Are equal. In this embodiment, M_origAnd M_refAre all 896 × 896 × 3 matrices, and therefore, the original image obtained by subtracting the reflection layer from the original image is subjected to reflection removal to obtain the image M_transThe color image is also an 896 × 896 × 3 matrix, wherein each element in the matrix represents the brightness value of each pixel point in the original image after the reflection of the light is eliminated in the RGB three color channels.

Case 2: the input image has reflection, and the input image is an image of an original image after preprocessing.

M _blend-M _{low_trans}＝M _{low_ref}

Illustratively, the input image, the first image are both three-channel RGB maps, the input image and the first image are both 896 × 896 pixels in size, then M is_blendAnd M_{low_trans}Are all 896 × 896 × 3 matrices. Each element of the matrix represents the brightness value of each pixel point of the image in three RGB color channels.

Since the input image is the preprocessed image of the original image, the size of the small image of the light reflecting layer is smaller than that of the original image. In this case, the small reflection layer image may be up-sampled to obtain a reflection layer, where the size of the reflection layer is the size of the original image. In some embodiments, the first light reflection layer small map may be upsampled by interpolation. And stripping the reflection layer from the original image to obtain an output image, wherein the output image is the image of the original image after reflection is eliminated.

M _orig-M _ref＝M _trans

wherein M is_origFor the original image, M_refIs a reflective layer, M_transIs an original pictureLike the result after eliminating the reflection layer.

In the embodiment described in case 2, the input image is a pre-processed image of the original image, and therefore the size of the reflection layer small image is smaller than that of the original image.

Illustratively, the size of the original image is 4K (3000 × 4000 pixels), M_origIs a 3000 x 4000 x 3 matrix. In this case, a small image M of the light-reflecting layer_{low_ref}The up-sampling is carried out by adopting a difference value mode, so that a reflecting layer M with the same size as the original image can be obtained_ref，M _refThe color conversion layer is also a 3000 × 4000 × 3 matrix, wherein each element in the matrix represents the brightness value of each pixel point of the reflection layer in three RGB color channels.

The original image M_origSubtracting from reflection layer M_refThe obtained image M with the original image subjected to reflection elimination_transThe color image is also a 3000 × 4000 × 3 matrix, wherein each element in the matrix represents the brightness value of each pixel point in the original image after reflection of the original image is eliminated in three RGB color channels.

Case 3: the input image is non-reflective.

In this case, since the input image is non-reflective, there is no need to process the original image. The output image is thus the original image.

According to the model for judging the intensity of the light reflecting layer, whether light is reflected in the image can be judged, so that a proper light reflecting processing mode is selected according to the judgment result, and the light reflecting processing effect of the image can be improved.

On one hand, the image processing method provided by the embodiment of the application can automatically identify whether the original image has reflection or not and carry out reflection elimination processing on the original image with reflection. On the other hand, when the original image size is large, the image processing method of the embodiment of the application firstly performs down-sampling on the original image to adapt to the input scale of the neural network, performs up-sampling on the small image of the reflective layer, and peels off the up-sampled reflective layer from the original image, thereby realizing reflection elimination of the high-resolution image.

In addition, according to the image processing method, the functions of identifying and eliminating the reflected light are achieved through the light-weight neural networks of the reflected light judging model and the reflected light eliminating model, and the problem that the reflected light eliminating network required by a high-definition image is too large is solved.

Fig. 9 is a diagram illustrating an image processing method 900 according to another embodiment of the present application. The method may be implemented by the reflectance detection and removal system software architecture shown in fig. 6. The method may also be performed by an apparatus capable of image processing, for example, the method may be performed by the execution device 110 in fig. 4. The steps in the method 900 are described in detail below.

S910, preprocessing the original image to obtain an input image which accords with the input size of the reflection elimination model. Step S910 may be implemented by the image preprocessing module 510 in the software architecture shown in fig. 6.

S920, inputting the input image into a reflection elimination model to obtain a first image after reflection elimination. Step S920 may be implemented by the network inference module 520 in the software architecture shown in fig. 6.

In the method 900, steps S910 to S920 are similar to steps S710 to S720 in the method 700 shown in fig. 7, and are not repeated.

S930, the first image and the input image are input to the retroreflective layer strength determining unit, and the determination result (for example, a second determination value described below) by the retroreflective layer strength determining unit is obtained. Step S930 may be implemented by the image post-processing module 530 in the software architecture shown in fig. 6. In some embodiments, whether the input image has reflection or not may be determined by the gray scale of the difference between the first image and the input image.

Fig. 10 is a schematic view of a determination method of the light-reflecting layer strength determination unit.

As shown in fig. 10, the input of the light-reflecting layer strength determination unit is the first image and the input image, and the output is the second determination value: the input image has reflection or the input image has no reflection.

The light reflecting layer strength judging unit calculates the gray value of the difference between the input image and the first image, and the gray value can represent the brightness of the image.

Illustratively, the size of the input image and the first image is 896 × 896 pixels, and subtracting the input image and the first image as described above in step S740 of method 700 results in an 896 × 896 × 3 matrix representation of the retroreflective layer patch. Wherein each element of the matrix represents the brightness value of each pixel point of the small image of the reflective layer in the RGB three color channels.

In some embodiments, the gray value may be calculated according to the following formula:

Gray(i,j)＝0.299×R(i,j)+0.587×G(i,j)+0.144×B(i,j)

wherein Gray (i, j) represents the Gray value of each pixel point of the small image of the reflective layer;

r (i, j) represents the absolute value of the red channel brightness value of each pixel point of the small image of the reflecting layer, and R (i, j) ═ M_{low_ref}(i,j,0)|；

G (i, j) represents the absolute value of the brightness value of the green channel of each pixel point of the small image of the light reflecting layer, and G (i, j) ═ M_{low_ref}(i,j,1)|；

B (i, j) represents the absolute value of the blue channel brightness value of each pixel point of the small image of the reflecting layer, and B (i, j) ═ M_{low_ref}(i,j,2)|。

It should be understood that, in the embodiment of the present application, the gray value may also be calculated according to other methods, for example, an average value of luminance values of three RGB color channels of each pixel point is taken as the gray value of each pixel point, and the like, which is not limited in the embodiment of the present application.

In some embodiments, the second determination value may be determined according to a weighted result of at least one or more of a maximum gray value of each pixel point of the reflection layer small image, a pixel point intensity of which the gray value is greater than the threshold TH1, and a pixel point area of which the gray value is greater than the threshold TH 2.

The intensity of the pixel point with the gray value greater than the threshold TH1 may be the sum of the gray values of the pixel points with the gray value greater than the threshold TH1, or may be the average of the gray values of the pixel points with the gray value greater than the threshold TH1, or may be other numerical values for representing the intensity, which is not limited in the embodiment of the present application.

The area of the pixel points with the gray value larger than the threshold TH2 represents the number of the pixel points with the gray value larger than the threshold TH 2.

For example, it may be determined that the input image has reflection according to the fact that the intensity of the pixel point with the gray scale greater than the threshold TH1 is greater than the threshold TH 3.

For example, it may be determined that the input image has reflection according to the fact that the weighted result of the maximum value of the gray scale and the area of the pixel point with the gray scale larger than the threshold TH2 is larger than the threshold TH 3.

For example, it may be determined that the input image has reflection according to the maximum value of the gray map, the intensity of the pixel point with the gray level greater than the threshold TH1, and the weighted result of the area of the pixel point with the gray level greater than the threshold TH2 that is greater than the threshold TH 3.

And S940, according to the second judgment value, the original image is subjected to reflection elimination processing. Step S940 may be implemented by the image post-processing module 530 in the software architecture shown in fig. 6.

For example, assuming that the second determination value is 1, it indicates that the input image has reflection; otherwise, the second judgment value is 0, which indicates that the input image has no reflection. The specific value of the second determination value and the correspondence between the presence or absence of reflection in the image are not limited to the above examples, and various reasonable settings may be performed as needed.

Specifically, the reflection elimination processing of the original image according to the reflection determination result (second determination value) may be classified into the following cases:

In this case, the first image may be subtracted from the input image to obtain the retroreflective layer thumbnail. In some embodiments, the result of subtracting the input image from the first image may be filtered to obtain the reflection layer small map. Since the size of the input image is the size of the original image, the retroreflective layer thumbnail is also the size of the original image. And stripping the small image of the light reflecting layer from the original image to obtain an output image, wherein the output image is the image of the original image after the light reflection is eliminated.

In this case, the calculation method of the reflection layer small image and the reflection-eliminated original image is similar to the calculation method described in the case 1 in the step S740 in the method 700, and is not repeated here.

In this case, the first image may be subtracted from the input image to obtain the retroreflective layer thumbnail. In some embodiments, the result of subtracting the input image from the first image may be filtered to obtain the reflection layer small map. Since the input image is the preprocessed image of the original image, the size of the small image of the light reflecting layer is smaller than that of the original image. In this case, the small reflection layer image may be up-sampled to obtain a reflection layer, where the size of the reflection layer is the size of the original image. In some embodiments, the first light reflection layer small map may be upsampled by interpolation. And stripping the reflection layer from the original image to obtain an output image, wherein the output image is the image of the original image after reflection is eliminated.

In this case, the calculation method of the small reflection layer image, the reflection layer image, and the original reflection-eliminated image is similar to the calculation method described in the case 2 in the step S740 in the method 700, and details are not repeated here.

Case 3: the input image is non-reflective.

According to the reflecting layer strength judging unit, whether the image has reflection or not can be judged, so that a proper reflecting processing mode is selected according to the judging result, and the reflecting processing effect of the image can be improved.

In addition, according to the image processing method provided by the embodiment of the application, the functions of identifying and eliminating the reflected light are realized through the reflecting layer strength judging unit and the light-weight reflected light eliminating model, and the problem that a reflected light eliminating network required by a high-definition image is too large is solved.

Fig. 11 is a diagram illustrating an image processing method 1100 according to another embodiment of the present application. The method may be implemented by the reflectance detection and removal system software architecture shown in fig. 6. The method may also be performed by an apparatus capable of image processing, for example, the method may be performed by the execution device 110 in fig. 4. The steps in this method 1100 are described in detail below.

S1110, preprocessing the original image to obtain an input image according with the input size of the reflection elimination model. Step S1110 may be implemented by the image preprocessing module 510 in the software architecture shown in fig. 6.

And S1120, inputting the input image into the reflection elimination model to obtain a first image after reflection elimination. Step S1120 may be implemented by the network inference module 520 in the software architecture shown in fig. 6.

In the method 1100, steps S1110 to S1120 are similar to steps S710 to S720 in the method 700 shown in fig. 7, and are not repeated.

S1130, the first image and the input image are input into the light reflection judgment model, and a first judgment value is obtained. Step S1130 may be implemented by the network inference module 520 in the software architecture shown in fig. 6.

In the method 1100, step S1130 is similar to step S730 in the method 700 shown in fig. 7, and is not repeated.

And S1140, inputting the first image and the input image into a reflecting layer strength judging unit to obtain a second judging value. Step S1140 may be implemented by the image post-processing module 530 in the software architecture shown in fig. 6.

In the method 1100, step S1140 is similar to step S930 in the method 900 shown in fig. 9 and is not repeated.

S1150, carrying out combined judgment according to the first judgment value and the second judgment value to obtain a light reflection judgment result. Step S1150 may be implemented by the image post-processing module 530 in the software architecture shown in fig. 6.

In some embodiments, for example, the first determination value may indicate whether the input image has a reflection. At this time, when the first judgment value and the second judgment value are both reflective, the reflective judgment result is reflective; and when at least one of the first judgment value and the second judgment value is no reflection, the reflection judgment result is no reflection. Or as another implementation mode, when at least one of the first judgment value and the second judgment value is judged to be reflected, the reflection judgment result is that reflection exists; and when the first judgment value and the second judgment value are both non-reflective, the reflective judgment result is non-reflective.

In other embodiments, for example, the first determination value may indicate a confidence level of whether the input image has reflection (i.e., the reflection confidence level or the reflection confidence coefficient). At this time, when the first determination value is greater than or equal to the specific threshold and the second determination value indicates that there is reflection, the reflection determination result output in step S1150 is that there is reflection; on the contrary, when the first determination value is smaller than the specific threshold and/or the second determination value indicates no reflection, the reflection determination result output in step S1150 is no reflection. Or as another implementation manner, when the first determination value is greater than or equal to the specific threshold and/or the second determination value indicates that there is reflection, the reflection determination result output in step S1150 is that there is reflection; on the contrary, when the first determination value is smaller than the specific threshold and the second determination value indicates no reflection, the reflection determination result output in step S1150 is no reflection.

Alternatively, the first determination value may be further refined, for example, when the first determination value is greater than the threshold value a1 and the second determination value indicates no reflection, the reflection determination result is reflection; when the first judgment value is smaller than a threshold A1 and larger than a threshold A2 and the second judgment value indicates that the light is reflected, the light reflection judgment result is that the light is reflected; in other cases, the light reflection judgment result is no light reflection, wherein A1 is larger than A2. Suitable joint judgment algorithms can be designed as required, and these variations are within the scope of the embodiments of the present application.

And S1160, performing reflection elimination processing on the original image according to the reflection judgment result. Step S1160 may be implemented by the image post-processing module 530 in the software architecture shown in fig. 6.

In the method 1100, step S1140 is similar to step S740 in the method 700 shown in fig. 7, and is not repeated.

By using a plurality of judgment models and a unit joint judgment mode, the light reflection condition of the image can be judged more accurately than that of a single model, and the light reflection processing effect of the image is improved. It is understood that other suitable judging models and/or judging units, if any, can be also incorporated into the glistening judging module or the joint judging mode of the present application.

Fig. 12 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present application. As shown in fig. 12, the image processing apparatus 1200 according to the embodiment of the present application includes a first reflection elimination module 1210, a reflection determination module 1220, and a second reflection elimination module 1230.

The first reflection eliminating module 1210 is configured to perform reflection eliminating processing on an input image to obtain a first image. In some embodiments, the first reflection eliminating module 1210 may be used to implement the function of step S720 in the method 700 shown in fig. 7, or step S920 in the method 900 shown in fig. 9, or step S1120 in the method 1100 shown in fig. 11. The specific functions and advantages of the first reflection eliminating module 1210 can be described in the above method, and for brevity, are not described in detail herein.

The reflection determination module 1220 is configured to determine whether the input image has reflection according to the first image and the input image.

In some embodiments, the reflection determination module 1220 may include a reflection determination model for implementing the function of step S730 in the method 700 shown in fig. 7.

In other embodiments, the reflection determination module 1220 may include a reflection layer strength determination model for implementing the function of step S930 in the method 900 shown in fig. 9.

In other embodiments, the reflection determination module 1220 may include a reflection determination model, a reflection layer strength determination model, and a joint determination unit, and is configured to implement the functions of steps S1130 to S1150 in the method 1100 shown in fig. 11.

The detailed functions and advantages of the reflection determination module 1220 can be referred to the description of the above method, and are not described herein for brevity.

And a second reflection eliminating module 1230, configured to obtain an output image according to the determination result of the reflection determining module. In some embodiments, the second reflection eliminating module is used to implement the function of step S740 in the method 700 shown in fig. 7, or step S940 in the method 900 shown in fig. 9, or step S1160 in the method 1100 shown in fig. 11. The specific functions and advantages of the second reflection eliminating module 1230 can be described in the above method, and for brevity, are not described in detail herein.

In some embodiments, the image processing apparatus may further include a preprocessing module 1240. The pre-processing module 1240 is used for pre-processing the original image to generate an input image that conforms to the input size of the first reflective output module 1210. In some embodiments, the preprocessing module 1240 is used to implement the functions of step S710 of the method 700 shown in fig. 7, or step S910 of the method 900 shown in fig. 9, or step S1100 of the method 1100 shown in fig. 11. The specific functions and advantages of the preprocessing module 1240 can be referred to the descriptions of the above methods, and are not described herein again for brevity.

It should be understood that in some embodiments, the image processing apparatus shown in fig. 12 may be implemented by the chip shown in fig. 5. The neural network processor 300 in the chip shown in fig. 5 may implement the functions of the reflection determination models in the first reflection elimination module 1210 and the reflection determination module 1220 of the image processing apparatus shown in fig. 12; the main CPU in the chip shown in fig. 5 may implement the functions of the reflective layer strength determining unit, the joint determining unit and the second reflective elimination module 1230 in the reflective determining module 1220. In other embodiments, the main CPU in the chip shown in fig. 5 may also implement the functions of the pre-processing module 1240.

In some embodiments, the image processing apparatus shown in fig. 12 may implement the functionality of the software architecture shown in fig. 6. The first reflection elimination module 1210 may implement the function of the reflection elimination model in the network inference module 520 of the software architecture shown in fig. 6; the reflection determination model in the reflection determination module 1220 may implement the function of the reflection determination model in the network inference module of the software architecture shown in fig. 6; the reflective layer strength determining unit, the joint determining unit and the second reflective elimination module 1230 in the reflective determining module 1220 may implement the function of the image post-processing module 530 in the software architecture shown in fig. 6. In other embodiments, the pre-processing module 1240 may implement the functionality of the image pre-processing module of the software architecture shown in FIG. 6.

In some embodiments, the image processing apparatus shown in fig. 12 may be disposed on the execution device 110 shown in fig. 4 to implement the functions of the respective modules.

Fig. 13 is a hardware configuration diagram according to an embodiment of the present application. It should be understood that the apparatus of the embodiments of the present application may be one or more chips or may be included in a terminal or other device.

As shown in fig. 13, a processor system 1300 according to an embodiment of the present disclosure includes a Central Processing Unit (CPU) 1310, a neural Network Processor (NPU) 1320, and an Image Signal Processing (ISP) 1330.

In the above method, the image pre-processing module 510, the image post-processing module 530 and the software processing module 540 of the software architecture shown in fig. 6 can be applied to the central processor 1310, or implemented by the central processor 1310. The central processor 1310 may also be used to run general-purpose operating system software and control the neural network processor 1320 and the image signal processor 1330 under the control of the general-purpose operating system software. In the above method, the network inference module 520 of the software architecture shown in fig. 6 may be applied to the neural network processor 1320, or implemented by the neural network processor 1320.

The image signal processor 1330 is used to generate images, and may be matched with image sensors of different manufacturers to process the image data output by the front-end image sensor and generate corresponding image signals, such as the original pictures mentioned above, according to the image data. In some embodiments, the image processing apparatus shown in fig. 12 may be implemented by the processor system shown in fig. 13. The central processor 1310 may implement the functions of the reflective layer strength determining unit and/or the joint determining unit in the reflective determining module 1220 and the second reflective eliminating module 1230; in other embodiments, the central processor 1310 may also implement the functionality of the pre-processing module 1240.

The neural network processor 1320 may implement the functions of the reflection determination model in the first reflection elimination module 1210 and the reflection determination module 1220 in the image processing apparatus shown in fig. 12.

It should be understood that the processor system shown in FIG. 13 may include one or more of the chips shown in FIG. 5, and that the main CPU in the chip shown in FIG. 5 may implement the functions of the central processor 1310 of the processor system shown in FIG. 13; the neural network processor 300 in the chip shown in fig. 5 may implement the functions of the neural network processor 1320 in the processor system shown in fig. 13.

It should be understood that the processor system shown in fig. 13 may also be disposed on the execution device 110 shown in fig. 4, which is not limited by the embodiment of the present application.

Fig. 14 is a schematic hardware configuration diagram of another image processing apparatus according to an embodiment of the present application. The image processing apparatus 1400 shown in fig. 14 includes a memory 1410, a processor 1420, a communication interface 1430, and a bus 1440. Wherein, the memory 1410, the processor 1420, and the communication interface 1430 are communicatively coupled to each other via a bus 1440.

The memory 1410 may be a Read Only Memory (ROM), a static memory device, a dynamic memory device, or a Random Access Memory (RAM). The memory 1410 may store a program, and when the program stored in the memory 1410 is executed by the processor 1420, the processor 1420 is configured to perform the steps of the image processing method according to the embodiment of the present application, for example, the steps shown in fig. 7 to 11.

It should be understood that the image processing apparatus shown in the embodiment of the present application may be a server, for example, a server in a cloud, or may also be a chip configured in the server in the cloud; alternatively, the image processing apparatus shown in the embodiment of the present application may be an intelligent terminal, or may be a chip configured in an intelligent terminal.

The image processing method disclosed in the embodiment of the present application may be applied to the processor 1420, or implemented by the processor 1420. Processor 1420 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the image processing method may be implemented by integrated logic circuits of hardware or instructions in the form of software in the processor 1420. For example, the processor 1420 may be a chip containing the NPU shown in fig. 5.

The processor 1420 may be a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules may be located in a Random Access Memory (RAM), a flash memory, a read-only memory (ROM), a programmable ROM, an electrically erasable programmable memory, a register, or other storage media that are well known in the art. The storage medium is located in the memory 1410, and the processor 1420 reads the instructions in the memory 1410, and completes the functions required to be executed by the software architecture of the reflection detection and elimination system shown in fig. 6 or the modules included in the image processing apparatus shown in fig. 12 in the application of the present application, or executes the steps of the image processing methods shown in fig. 7 to 11 in the embodiments of the method of the present application, in combination with the hardware of the processor.

Communication interface 1430 enables communication between apparatus 1400 and other devices or communication networks using transceiver devices such as, but not limited to, transceivers.

Bus 1440 may include a pathway to transfer information between various components of image processing device 1400 (e.g., memory 1410, processor 1420, communication interface 1430).

It should be noted that although the image processing apparatus 1400 just shows a memory, a processor and a communication interface, in a specific implementation process, a person skilled in the art should understand that the image processing apparatus 1400 may also include other devices necessary for normal operation. Meanwhile, it will be understood by those skilled in the art that the image processing apparatus 1400 may further include hardware components for implementing other additional functions according to specific needs. Furthermore, it should be understood by those skilled in the art that the image processing apparatus 1400 may also include only the components necessary to implement the embodiments of the present application, and not necessarily all of the components shown in fig. 14.

The present embodiments also provide a computer-readable medium storing a computer program (also referred to as code, or instructions) which, when run on a computer, causes the computer to perform the method in any of the above-described method embodiments.

The embodiment of the present application further provides a chip system, which includes a memory and a processor, where the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that an image processing apparatus in which the chip system is installed executes the method in any of the above method embodiments.

The system-on-chip may include, among other things, input circuitry or interfaces for transmitting information or data, and output circuitry or interfaces for receiving information or data.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on a storage medium or transmitted from one storage medium to another storage medium, for example, from one website, computer, server, or data center to another website, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The storage medium may be any available medium that can be accessed by a computer or a data storage device including one or more available media integrated servers, data centers, and the like. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a Digital Video Disk (DVD)), or a semiconductor medium (e.g., a Solid State Disk (SSD)), among others.

In the present application, "at least two" or "plural" means two or more. The term "and/or" describes an associative relationship of associated objects, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone, wherein A and B can be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

Predefinition in this application may be understood as defining, predefining, storing, pre-negotiating, pre-configuring, curing, or pre-firing.

It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in the various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

As used in this specification, the terms "component," "module," "system," and the like are intended to refer to a computer-related entity, either hardware, firmware, a combination of hardware and software, or software in execution. For example, a component may be, but is not limited to being, a process running on a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a computing device and the computing device can be a component. One or more components can reside within a process and/or thread of execution and a component can be localized on one computer and/or distributed between 2 or more computers. In addition, these components can execute from various computer readable media having various data structures stored thereon. The components may communicate by way of local and/or remote processes such as in accordance with a signal having one or more data packets (e.g., data from two components interacting with another component in a local system, distributed system, and/or across a network such as the internet with other systems by way of the signal).

Those of ordinary skill in the art will appreciate that the various method steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the steps of the embodiments have been described above generally in terms of their functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps (step) described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it is understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

An image processing apparatus characterized by comprising:

the first reflection eliminating module is used for carrying out reflection eliminating processing on the input image to obtain a first image;

the light reflection judging module is used for acquiring a judgment result of whether the input image has light reflection according to the first image and the input image;

and the second reflection eliminating module is used for acquiring an output image according to the judgment result.
The apparatus of claim 1, further comprising:

the preprocessing module is used for preprocessing an original image to obtain the input image, wherein the size of the original image is larger than that of the input image.
The apparatus according to claim 1 or 2, wherein the second reflection elimination module is specifically configured to, when the determination result is that the input image is not reflected, acquire an original image as the output image.
The apparatus according to claim 2, wherein the second reflection elimination module is specifically configured to, when the determination result is that the input image has reflection, generate a reflection layer small map according to the first image and the input image, perform upsampling on the reflection layer small map to obtain a reflection layer, and obtain the output image according to the reflection layer and the original image.
The apparatus according to any one of claims 1 to 4, wherein the reflection determination module includes a reflection determination model, the reflection determination model is a neural network having the first image and the input image as input, and an output of the reflection determination model indicates a confidence that the input image has reflection, or the output is the determination result.
The apparatus of claim 5, wherein the glistening decision model is a Markov arbiter-based neural network and is trained based on a first set of samples, wherein the first set of samples comprises: the image processing method comprises the steps of obtaining an original image sample, obtaining the image sample after the original image sample is subjected to reflection elimination, and judging whether the original image sample has a reflection label.
The apparatus according to any one of claims 1 to 4, wherein the reflection determination module comprises a reflection layer intensity determination unit configured to generate a reflection layer small map from the input image and the first image; the reflective layer strength judging unit is further configured to calculate gray values of a plurality of pixel points included in the reflective layer small graph, and determine the judging result according to the gray values of the plurality of pixel points.
The apparatus according to claim 7, wherein the light reflection layer strength judgment unit determines the judgment result based on a weighted result of at least one or more of the following information:

the maximum value of the gray values of the plurality of pixel points, the intensity of at least one pixel point with the gray value larger than a first threshold value and the area of at least one pixel point with the gray value larger than a second threshold value;

wherein, the intensity of at least one pixel point with the gray value larger than the first threshold is the sum of the gray values of the pixel points with the gray value larger than the first threshold, or,

the intensity of at least one pixel point with the gray value larger than the first threshold value is the average value of the gray values of the pixel points with the gray values larger than the first threshold value;

the area of at least one pixel point with the gray value larger than the second threshold is the number of the pixel points with the gray value larger than the second threshold.
The apparatus according to any one of claims 1 to 4, wherein the reflection determination means comprises a reflection determination model, a reflection layer strength determination unit, and a joint determination unit,

the reflection judgment model is a neural network taking the first image and the input image as input, and the output of the reflection judgment model is a first judgment value representing whether the input image has reflection;

the light reflecting layer strength judging unit calculates a gray value of a difference between the input image and the first image, and outputs a second judgment value representing whether the input image has light reflection according to the gray value;

the joint judgment unit determines the judgment result according to the first judgment value and the second judgment value.
An image processing method, comprising:

performing reflection elimination processing on an input image to obtain a first image;

judging whether the input image has reflection or not according to the input image and the first image;

and acquiring an output image according to whether the input image has reflection.
The method of claim 10, further comprising:

preprocessing an original image to obtain the input image, wherein the size of the original image is larger than that of the input image.
The method according to claim 10 or 11, wherein the obtaining an output image according to whether the input image has reflection comprises: when the input image is not back-lighted, an original image is acquired as the output image.
The method of claim 11, wherein obtaining an output image based on whether the input image has reflections comprises: when the input image has the reflection, generating a reflection layer small image according to the input image and the first image;

the small reflective layer image is up-sampled to obtain a reflective layer;

and acquiring the output image according to the reflection layer and the original image.
The method according to any one of claims 10 to 13, wherein the determining whether the input image has glistenings according to the input image and the first image comprises:

judging whether the input image has reflection or not according to a reflection judgment model;

the light reflection judgment model is a neural network which takes the first image and the input image as input, and the output of the light reflection judgment model represents the confidence coefficient of the input image that the input image has light reflection, or the output is the judgment result of whether the input image has light reflection.
The method of claim 14, wherein the glistening decision model is a markov discriminator-based neural network trained based on a first set of samples, wherein the first set of samples comprises: the image processing method comprises the steps of obtaining an original image sample, obtaining the image sample after the original image sample is subjected to reflection elimination, and judging whether the original image sample has a reflection label.
The method according to any one of claims 10 to 13, wherein the determining whether the input image has glistenings according to the input image and the first image comprises:

judging whether the input image has reflection according to a reflection layer strength judging unit;

the reflecting layer strength judging unit is used for calculating a reflecting layer small image generated according to the input image and the first image; the reflective layer strength judging unit is further configured to calculate gray values of a plurality of pixel points included in the reflective layer small graph, and determine the judging result according to the gray values of the plurality of pixel points.
The method according to claim 16, wherein the light-reflective layer determination unit determines the determination result based on a weighted result of at least one or more of the following information:

the maximum value of the gray values of the plurality of pixel points, the intensity of at least one pixel point with the gray value larger than a first threshold value and the area of at least one pixel point with the gray value larger than a second threshold value;

wherein, the intensity of at least one pixel point with the gray value larger than the first threshold is the sum of the gray values of the pixel points with the gray value larger than the first threshold, or,

the intensity of at least one pixel point with the gray value larger than the first threshold value is the average value of the gray values of the pixel points with the gray values larger than the first threshold value;

and the area of at least one pixel point with the gray value larger than the second threshold is the number of the pixel points with the gray value larger than the second threshold.
The method according to any one of claims 10 to 13, wherein the determining whether the input image has glistenings according to the input image and the first image comprises:

judging whether the input image has reflection according to the reflection judgment model, the reflection layer strength judgment unit and the joint judgment unit;

the reflection judgment model is a neural network taking the first image and the input image as input, and the output of the reflection judgment model is a first judgment value representing whether the input image has reflection;

the light reflecting layer strength judging unit calculates a gray value of a difference between the input image and the first image, and outputs a second judgment value representing whether the input image has light reflection according to the gray value;

and the joint judgment unit determines a judgment result according to the first judgment value and the second judgment value.