TWI749593B - Method, electronic device and computer readable storage medium for removing reflection in image - Google Patents

Abstract

The invention relates to a method for removing light reflection in an image, an electronic device and a computer-readable storage medium. The method includes: acquiring a to-be-processed image; acquiring multiple polarization maps corresponding to the to-be-processed image, and polarization information corresponding to the to-be-processed image, wherein the multiple polarizations corresponding to the to-be-processed image The figure is formed by polarizing plates with different angles; according to the multiple polarization maps corresponding to the image to be processed and the polarization information corresponding to the image to be processed, the reflection prediction map corresponding to the image to be processed is determined; according to The multiple polarization maps corresponding to the image to be processed and the light reflection prediction map corresponding to the image to be processed determine the image after the light reflection corresponding to the image to be processed is removed.

Description

Method, electronic device and computer readable storage medium for removing reflection in image

This application requires the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 202010144325.2, and the application title is "Method and device for removing light reflection in images, electronic equipment and storage medium" on March 4, 2020. The entire content is incorporated into this application by reference.

The present invention relates to the field of image technology, and in particular to a method for removing light reflection in an image, an electronic device and a computer-readable storage medium.

In real life and work, using a camera to take photos requires shooting objects through glass in some cases. For example, taking pictures of still life outside through the window, taking pictures of people wearing glasses, taking pictures of exhibits in glass cabinets in museums, taking pictures of illegal vehicles on traffic roads, etc. Due to the different lighting conditions on both sides of the glass, reflections on the surface of the glass are certain.

The present invention provides a technical solution for removing light reflection in an image.

According to one aspect of the present invention, there is provided a method for removing reflections in an image, including: acquiring an image to be processed; acquiring multiple polarization maps corresponding to the image to be processed, and Polarization information, wherein the multiple polarization images corresponding to the image to be processed are formed by polarizing plates with different angles; according to the multiple polarization images corresponding to the image to be processed and the polarization corresponding to the image to be processed Message, determine the reflection prediction map corresponding to the image to be processed; determine the reflection prediction map corresponding to the image to be processed according to the multiple polarization maps corresponding to the image to be processed and the reflection prediction map corresponding to the image to be processed The image after the reflection is removed.

In a possible implementation, the acquiring multiple polarization images corresponding to the image to be processed and the polarization information corresponding to the image to be processed includes: pixels belonging to different polarizer angles in the image to be processed Points are separated to obtain multiple polarization maps corresponding to the image to be processed; corresponding pixel points in the multiple polarization maps corresponding to the image to be processed are processed to obtain polarization information corresponding to the image to be processed .

In a possible implementation, the polarization information corresponding to the image to be processed includes a first polarization information image corresponding to the image to be processed, a second polarization information image corresponding to the image to be processed, and The third polarization information map corresponding to the processed image At least one of the fourth polarization information images corresponding to the image to be processed, wherein the first polarization information image corresponding to the image to be processed is used to represent multiple polarization images corresponding to the image to be processed The second polarization information image corresponding to the image to be processed is used to indicate the polarization degree of the multiple polarization images corresponding to the image to be processed, and the third polarization information image corresponding to the image to be processed Used to indicate the polarization angles of the light of the multiple polarization images corresponding to the image to be processed, and the fourth polarization information image corresponding to the image to be processed is used to indicate the removal of the multiple polarization images corresponding to the image to be processed Message after overexposure.

In a possible implementation manner, before the obtaining the image to be processed, the method further includes: obtaining a plurality of polarization maps corresponding to the training image, and polarization information corresponding to the training image, wherein the The multiple polarization images corresponding to the training image are formed through polarizers with different angles; the multiple polarization images corresponding to the training image and the polarization information corresponding to the training image are input into the first subnet of the neural network Path, output the reflection prediction map corresponding to the training image via the first subnet; input the multiple polarization maps corresponding to the training image and the reflection prediction map corresponding to the training image into the neural network The second sub-network of the road, through the second sub-network, outputs the transmission light prediction map corresponding to the training image; at least according to the transmission light prediction map corresponding to the training image, the training station The first subnet and the second subnet.

In a possible implementation manner, the acquiring multiple polarization maps corresponding to the training image and the polarization information corresponding to the training image includes: separating pixels belonging to different polarizer angles in the training image; Obtaining multiple polarization maps corresponding to the training image; processing corresponding pixels in the multiple polarization maps corresponding to the training image to obtain polarization information corresponding to the training image.

In a possible implementation, the polarization information corresponding to the training image includes a first polarization information image corresponding to the training image, a second polarization information image corresponding to the training image, and a second polarization information image corresponding to the training image. At least one of the third polarization information image corresponding to the training image and the fourth polarization information image corresponding to the training image, wherein the first polarization information image corresponding to the training image is used to indicate the multiple information corresponding to the training image. The polarization intensity of a polarization image, the second polarization information image corresponding to the training image is used to indicate the polarization degree of the multiple polarization images corresponding to the training image, and the third polarization information image corresponding to the training image Used to represent the polarization angles of the light of the multiple polarization images corresponding to the training image, and the fourth polarization information image corresponding to the training image is used to represent the multiple polarization images corresponding to the training image after the overexposure is removed Message.

In a possible implementation, the training the first subnet and the second subnet at least according to the transmitted light prediction map corresponding to the training image includes: Determine the value of the first loss function according to the transmitted light prediction map corresponding to the training image and the reflection prediction map corresponding to the training image; train the first subnet at least according to the value of the first loss function And the second subnet.

l

P，P表示所述第一預設網路的總層數；將所述歸一化的反光預測圖輸入所述第一預設網路，經由所述第一預設網路的第l層輸出所述歸一化的反光預測圖對應的第l層的特徵圖；根據所述歸一化的透射光預測圖對應的第l層的特徵圖與所述歸一化的反光預測圖對應的第l層的特徵圖之間的歸一化互相關值，確定第一損失函數的值。 In a possible implementation manner, the determining the value of the first loss function according to the transmitted light prediction map corresponding to the training image and the reflection prediction map corresponding to the training image includes: The corresponding transmitted light prediction image and the reflection prediction image corresponding to the training image are respectively normalized to obtain a normalized transmission light prediction image and a normalized reflection prediction image corresponding to the training image; The normalized transmitted light prediction map is input into a first preset network, and the characteristic map of the first layer corresponding to the normalized transmitted light prediction map is output through the first layer of the first preset network , Where 1

l

P, P represents the total number of layers of the first preset network; input the normalized reflection prediction map into the first preset network, and pass through the first layer of the first preset network Output the feature map of the first layer corresponding to the normalized reflection prediction map; according to the feature map of the first layer corresponding to the normalized transmitted light prediction map corresponding to the normalized reflection prediction map The normalized cross-correlation value between the feature maps of the first layer determines the value of the first loss function.

In a possible implementation, the training of the first sub-network and the second sub-network at least according to the transmitted light prediction map corresponding to the training image includes: according to the training image and The difference between the reflective real image corresponding to the training image is used to obtain the transmitted light target image corresponding to the training image; the second loss is determined according to the transmitted light prediction image corresponding to the training image and the transmitted light target image The value of the function; training the first sub-network and the second sub-network at least according to the value of the second loss function.

k

Q，Q表示所述第二預設網路的總層數；將所述去除過曝的透射光目標圖輸入所述第二預設網路，經由所述第二預設網路的第k層輸出所述去除過曝的透射光目標圖對應的第k層的特徵圖；根據所述去除過曝的透射光預測圖對應的第k層的特徵圖與所述去除過曝的透射光目標圖對應的第k層的特徵圖之間的差值，確定第二損失函數的值。 In a possible implementation manner, the determining the value of the second loss function according to the transmitted light prediction map corresponding to the training image and the transmitted light target map includes: according to the transmitted light corresponding to the training image The product of the prediction image and the pixel value of the corresponding pixel in the fourth polarization information image corresponding to the training image is obtained to obtain the transmission light prediction image corresponding to the training image without overexposure, wherein, in the training image In the corresponding fourth polarization information image, the pixel value of the overexposed pixel is the first preset value, and the pixel value of the non-overexposed pixel is the second preset value, wherein the first preset value is less than The second preset value; according to the product of the transmitted light target image and the pixel value of the corresponding pixel in the fourth polarization information image corresponding to the training image, obtain the overexposed removal corresponding to the training image Transmitted light target map; input the overexposed transmitted light predicted map into a second preset network, and output the corresponding to the removed overexposed transmitted light predicted map through the kth layer of the second preset network The feature map of the k-th layer, where 1

k

Q, Q represents the total number of layers of the second preset network; input the overexposed transmitted light target map into the second preset network, and pass through the kth of the second preset network The layer outputs the feature map of the k-th layer corresponding to the overexposed transmitted light target image; according to the feature map of the k-th layer corresponding to the overexposed transmitted light prediction image and the removed overexposed transmitted light target The difference between the feature maps of the k-th layer corresponding to the map determines the value of the second loss function.

In a possible implementation manner, before the obtaining the transmitted light target image corresponding to the training image according to the difference between the training image and the reflective real image corresponding to the training image, the method further includes : According to one aspect of the present invention, there is provided a device for removing reflections in an image, including: a first acquisition module for acquiring an image to be processed; a second acquisition module for acquiring the image to be processed Multiple polarization images corresponding to the image, and polarization information corresponding to the image to be processed, wherein the multiple polarization images corresponding to the image to be processed are formed through polarizers with different angles; The first prediction module is used to determine the reflection prediction image corresponding to the image to be processed according to the plurality of polarization images corresponding to the image to be processed and the polarization information corresponding to the image to be processed; the second prediction model The group is used to determine the image after reflection removal corresponding to the image to be processed according to the multiple polarization images corresponding to the image to be processed and the light reflection prediction image corresponding to the image to be processed.

In a possible implementation manner, the second acquisition module is used to: separate pixels belonging to different polarizer angles in the image to be processed to obtain multiple polarization images corresponding to the image to be processed; Corresponding pixel points in the multiple polarization images corresponding to the image to be processed are processed to obtain polarization information corresponding to the image to be processed.

In a possible implementation, the polarization information corresponding to the image to be processed includes a first polarization information image corresponding to the image to be processed, a second polarization information image corresponding to the image to be processed, and At least one of the third polarization information image corresponding to the processed image and the fourth polarization information image corresponding to the image to be processed, wherein the first polarization information image corresponding to the image to be processed is used to represent the The polarization intensity of the multiple polarization images corresponding to the image to be processed, and the second polarization information image corresponding to the image to be processed is used to indicate the polarization degree of the multiple polarization images corresponding to the image to be processed. The third polarization information map corresponding to the processed image is used to indicate the polarization angles of the light of the multiple polarization maps corresponding to the image to be processed, and the fourth polarization information map corresponding to the image to be processed The information image is used to represent the information after the overexposure is removed from the multiple polarization images corresponding to the image to be processed.

In a possible implementation, the device further includes: a third acquisition module for acquiring multiple polarization maps corresponding to the training image, and polarization information corresponding to the training image, wherein the training image The multiple polarization images corresponding to the image are formed through polarizers with different angles; the third prediction module is used to input the multiple polarization images corresponding to the training image and the polarization information corresponding to the training image into the neural network The first sub-network of the road, through the first sub-network, outputs the reflection prediction map corresponding to the training image; the fourth prediction module is used to combine the multiple polarization maps corresponding to the training image and the The reflection prediction map corresponding to the training image is input to the second sub-network of the neural network, and the transmitted light prediction map corresponding to the training image is output through the second sub-network; the training module is used for at least Training the first sub-network and the second sub-network according to the transmitted light prediction map corresponding to the training image.

In a possible implementation, the third acquisition module is used to: separate pixels belonging to different polarizer angles in the training image to obtain multiple polarization maps corresponding to the training image; The corresponding pixel points in the multiple polarization images corresponding to the training image are processed to obtain the polarization information corresponding to the training image.

In a possible implementation, the polarization information corresponding to the training image includes a first polarization information image corresponding to the training image, a second polarization information image corresponding to the training image, and a second polarization information image corresponding to the training image. At least one of the third polarization information image corresponding to the training image and the fourth polarization information image corresponding to the training image, wherein the first polarization information image corresponding to the training image is used to indicate the multiple information corresponding to the training image. The polarization intensity of a polarization image, the second polarization information image corresponding to the training image is used to indicate the polarization degree of the multiple polarization images corresponding to the training image, and the third polarization information image corresponding to the training image Used to represent the polarization angles of the light of the multiple polarization images corresponding to the training image, and the fourth polarization information image corresponding to the training image is used to represent the multiple polarization images corresponding to the training image after the overexposure is removed Message.

In a possible implementation, the training module is configured to: determine the value of the first loss function according to the transmitted light prediction map corresponding to the training image and the reflection prediction map corresponding to the training image; at least according to The value of the first loss function trains the first subnet and the second subnet.

l

P，P表示所述第一預設網路的總層數；將所述歸一化的反光預測圖輸入所述第一預設網路，經由所述第一預設網路的第l層輸出所述歸一化的反光預測圖對應的第l層的特徵圖；根據所述歸一化的透射光預測圖對應的第l層的特徵圖與所述歸一化的反光預測圖對應的第l層的特徵圖之間的歸一化互相關值，確定第一損失函數的值。 In a possible implementation, the training module is used to: normalize the transmitted light prediction map corresponding to the training image and the reflection prediction map corresponding to the training image to obtain the training The normalized transmitted light prediction map and the normalized reflection prediction map corresponding to the image; input the normalized transmitted light prediction map into the first preset network, and the The first layer outputs the feature map of the first layer corresponding to the normalized transmitted light prediction map, where 1

l

P, P represents the total number of layers of the first preset network; input the normalized reflection prediction map into the first preset network, and pass through the first layer of the first preset network Output the feature map of the first layer corresponding to the normalized reflection prediction map; according to the feature map of the first layer corresponding to the normalized transmitted light prediction map corresponding to the normalized reflection prediction map The normalized cross-correlation value between the feature maps of the first layer determines the value of the first loss function.

In a possible implementation, the training module is configured to: obtain the transmitted light target image corresponding to the training image according to the difference between the training image and the reflective real image corresponding to the training image; The transmitted light prediction map and the transmitted light target map corresponding to the training image are used to determine the value of the second loss function; at least according to the value of the second loss function, the first subnet and the first subnet are trained Two subnets.

k

Q，Q表示所述第二預設網路的總層數；將所述去除過曝的透射光目標圖輸入所述第二預設網路，經由所述第二預設網路的第k層輸出所述去除過曝的透射光目標圖對應的第k層的特徵圖；根據所述去除過曝的透射光預測圖對應的第k層的特徵圖與所述去除過曝的透射光目標圖對應的第k層的特徵圖之間的差值，確定第二損失函數的值。 In a possible implementation, the training module is used to: according to the product of the transmitted light prediction image corresponding to the training image and the pixel value of the corresponding pixel in the fourth polarization information image corresponding to the training image , Obtain the transmitted light prediction map corresponding to the training image without overexposure, wherein, in the fourth polarization information map corresponding to the training image, the pixel value of the overexposed pixel is the first preset value, The pixel value of the non-overexposed pixel is a second preset value, wherein the first preset value is less than the second preset value; according to the transmitted light target image and the training image corresponding to the first The product of the pixel values of the corresponding pixels in the four-polarization information image to obtain the target image of the transmitted light for removing the overexposure corresponding to the training image; input the predicted image of the transmitted light for removing the overexposed light into the second preset network, Output the feature map of the k-th layer corresponding to the overexposed transmitted light prediction map through the k-th layer of the second preset network, where 1

k

Q, Q represents the total number of layers of the second preset network; input the overexposed transmitted light target map into the second preset network, and pass through the kth of the second preset network The layer outputs the feature map of the k-th layer corresponding to the overexposed transmitted light target image; according to the feature map of the k-th layer corresponding to the overexposed transmitted light prediction image and the removed overexposed transmitted light target The difference between the feature maps of the k-th layer corresponding to the map determines the value of the second loss function.

In a possible implementation manner, the device further includes: a collection module for collecting the training image and the reflective real image corresponding to the training image through the polarization sensor.

According to an aspect of the present invention, there is provided an electronic device, including: One or more processors; a memory used to store executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above-mentioned method.

According to one aspect of the present invention, there is provided a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

According to one aspect of the present invention, there is provided a computer program including computer-readable code, and when the computer-readable code is executed in an electronic device, a processor in the electronic device executes for realizing the above-mentioned method.

In the embodiment of the present invention, by acquiring the image to be processed, multiple polarization maps corresponding to the image to be processed are acquired, and polarization information corresponding to the image to be processed is obtained according to the number of polarization corresponding to the image to be processed. A polarization map and the polarization information corresponding to the image to be processed, determine the reflection prediction map corresponding to the image to be processed, according to the multiple polarization maps corresponding to the image to be processed and the polarization information corresponding to the image to be processed The reflection prediction map determines the image after the reflection is removed corresponding to the image to be processed, so that the reflection in the image to be processed can be accurately removed.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present invention.

According to the following detailed description of exemplary embodiments with reference to the drawings, other features and aspects of the present invention will become clear.

71: The first acquisition module

72: The second acquisition module

73: The first prediction module

74: The second prediction module

800: electronic equipment

802: Processing component

804: memory

806: Power Components

808: Multimedia components

810: Audio component

812: input/output interface

814: Sensor component

816: Communication component

820: processor

1900: electronic equipment

1922: processing components

1926: power supply components

1932: memory

1950: network interface

1958: input/output interface

S11~S14: steps

The drawings here are incorporated into the specification and constitute a part of this specification. These drawings show embodiments in accordance with the present invention, and together with the specification are used to illustrate the technical solution of the present invention: Figure 1 shows the implementation of the present invention. Example provides a flowchart of a method for removing reflections in an image; Figure 2 shows a schematic diagram of a neural network provided by an embodiment of the present invention; Figure 3 shows a transmission light prediction map corresponding to a training image in an embodiment of the present invention A schematic diagram of the monotonicity of the first loss function L _PNCC before and after normalization of the reflection prediction map corresponding to the training image; FIG. 4 shows a schematic diagram of the background image B, the transmitted light image T, the light reflection image R, and the mixed image M; 5 shows a schematic diagram of the acquisition method of the training image and the reflective real image corresponding to the training image; FIG. 6 shows the method for removing the reflection in the image provided by the embodiment of the present invention for the three different types of reflection A schematic diagram of an output image obtained after processing an input image; FIG. 7 shows a block diagram of a device for removing reflections in an image provided by an embodiment of the present invention; FIG. 8 shows an electronic device 800 provided by an embodiment of the present invention And Figure 9 shows a block diagram of an electronic device 1900 provided by an embodiment of the present invention.

Various exemplary embodiments, features, and aspects of the present invention will be described in detail below with reference to the drawings. The same drawing symbols in the drawings indicate elements with the same or similar functions. Although various aspects of the embodiments are shown in the drawings, unless otherwise noted, the drawings are not necessarily drawn to scale.

The dedicated word "exemplary" here means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" need not be construed as being superior or better than other embodiments.

The term "and/or" in this article is only an association relationship that describes the associated objects, which means that there can be three relationships, for example, A and/or B, which can mean: A alone exists, A and B exist at the same time, and B exists alone. three situations. In addition, the term "at least one" in this document means any one of multiple or any combination of at least two of the multiple, for example, including at least one of A, B, and C, and may mean including those made from A, B, and C Any one or more elements selected in the set.

In addition, in order to better illustrate the present invention, numerous specific details are given in the following specific embodiments. Those skilled in the art should understand that the present invention can also be implemented without certain specific details. In some instances, for those skilled in the art The well-known methods, means, components and circuits are not described in detail in order to highlight the gist of the present invention.

As mentioned above, due to the different lighting conditions on both sides of the glass, the glass surface may be reflective. This kind of reflection not only affects the beauty of the photo, but also may cause a lot of details of the real scene to be lost. For example, when a photo of an illegal vehicle is taken on a traffic road, the excessive reflection of the car window may make it impossible to see the driver's face.

In order to solve technical problems similar to those described above, embodiments of the present invention provide a method and device for removing light reflection in an image, electronic equipment, and storage media. By acquiring the image to be processed, the corresponding image to be processed is obtained. And the polarization information corresponding to the image to be processed, and the polarization information corresponding to the image to be processed and the polarization information corresponding to the image to be processed are used to determine the image to be processed Corresponding reflection prediction map, according to the multiple polarization maps corresponding to the image to be processed and the reflection prediction map corresponding to the image to be processed, determine the image after reflection removal corresponding to the image to be processed, thereby Can accurately remove the reflection in the image to be processed.

The embodiments of the present invention can be applied to various application scenarios. For example, in the case of taking pictures of still life outside through the window, taking pictures of the scenery outside the car windows in the car, taking pictures of exhibits in glass cabinets in museums, taking pictures of illegal vehicles on traffic roads, etc., the embodiments of the present invention can be used quickly. Remove the reflections in the photos taken, and provide users with photos without reflection interference. For another example, when shooting a portrait of a person wearing glasses, the embodiment of the present invention can be used to quickly remove the reflections in the captured photo, so that the eyes of the person are The eye area is more clear.

The embodiments of the present invention can be applied to the fields of computer vision, intelligent image processing, photographing, automatic driving, robot vision and the like.

Fig. 1 shows a flowchart of a method for removing light reflection in an image provided by an embodiment of the present invention. The execution subject of the method for removing light reflection in an image may be a device for removing light reflection in an image. For example, the method for removing light reflection in an image can be executed by a terminal device or a server or other processing device. Among them, the terminal device can be User Equipment (UE), mobile device, user terminal, terminal, mobile phone, wireless phone, personal digital assistant (Personal Digital Assistant, PDA), handheld device, computing device, vehicle-mounted device, or Wearable equipment, etc. In some possible implementation manners, the method for removing reflections in an image may be implemented by a processor invoking a computer-readable instruction stored in a memory. As shown in FIG. 1, the method for removing light reflection in an image includes step S11 to step S14.

In step S11, an image to be processed is acquired.

In the embodiment of the present invention, the image to be processed may be acquired by a polarization sensor, and the image to be processed may be a single-channel image. For example, the image to be processed may include image information obtained through polarizers at 4 angles of 0°, 45°, 90°, and 135°.

In step S12, obtain multiple polarizations corresponding to the image to be processed Figure, and the polarization information corresponding to the image to be processed, wherein the multiple polarization images corresponding to the image to be processed are formed through polarizers with different angles.

For example, the image to be processed includes image information with 4 polarizer angles of 0°, 45°, 90°, and 135°. Accordingly, the number of polarization images corresponding to the image to be processed may be 4. The four polarization images corresponding to the image to be processed respectively correspond to the four polarizer angles of 0°, 45°, 90°, and 135°.

In a possible implementation manner, the polarization image corresponding to the image to be processed may be a grayscale image.

In the embodiment of the present invention, the polarization information corresponding to the image to be processed may be determined according to multiple polarization images corresponding to the image to be processed.

In a possible implementation, the acquiring multiple polarization images corresponding to the image to be processed and the polarization information corresponding to the image to be processed includes: pixels belonging to different polarizer angles in the image to be processed Points are separated to obtain multiple polarization maps corresponding to the image to be processed; corresponding pixel points in the multiple polarization maps corresponding to the image to be processed are processed to obtain polarization information corresponding to the image to be processed . For example, if the image to be processed includes 4 polarizer angles of 0°, 45°, 90°, and 135°, the pixels of 0° in the image to be processed can be separated to obtain the image information. The first polarization image corresponding to the image to be processed is separated, the pixels belonging to 45° in the image to be processed are separated, the second polarization image corresponding to the image to be processed is obtained, and the image to be processed is separated The pixel point in the middle of 90° is obtained, and the third image corresponding to the image to be processed is obtained. Polarization map, separating the 135° pixels in the image to be processed to obtain the fourth polarization map corresponding to the image to be processed.

In a possible implementation, the polarization information corresponding to the image to be processed includes a first polarization information image corresponding to the image to be processed, a second polarization information image corresponding to the image to be processed, and At least one of the third polarization information image corresponding to the processed image and the fourth polarization information image corresponding to the image to be processed, wherein the first polarization information image corresponding to the image to be processed is used to represent the The polarization intensity of the multiple polarization images corresponding to the image to be processed, and the second polarization information image corresponding to the image to be processed is used to indicate the polarization degree of the multiple polarization images corresponding to the image to be processed. The third polarization information diagram corresponding to the processed image is used to represent the polarization angles of the light of the multiple polarization diagrams corresponding to the image to be processed, and the fourth polarization information diagram corresponding to the image to be processed is used to represent the Process the multiple polarization maps corresponding to the image to remove the overexposed information.

In step S13, a reflection prediction image corresponding to the image to be processed is determined according to the plurality of polarization images corresponding to the image to be processed and the polarization information corresponding to the image to be processed.

In a possible implementation, the multiple polarization maps corresponding to the image to be processed and the polarization information corresponding to the image to be processed can be input into the first subnet of the neural network, through the first subnet. The network outputs the reflection prediction map corresponding to the image to be processed. In this implementation, the reflection prediction map corresponding to the image to be processed may be To represent the reflection map corresponding to the image to be processed predicted by the neural network.

In step S14, according to the plurality of polarization maps corresponding to the image to be processed and the light reflection prediction map corresponding to the image to be processed, a light-reflection-removed image corresponding to the image to be processed is determined.

In a possible implementation, the multiple polarization maps corresponding to the image to be processed and the reflection prediction map corresponding to the image to be processed can be input into the second subnet of the neural network, and via the The second subnet outputs the image after the reflection is removed corresponding to the image to be processed.

In a possible implementation manner, before the obtaining the image to be processed, the method further includes: obtaining a plurality of polarization maps corresponding to the training image, and polarization information corresponding to the training image, wherein the The multiple polarization images corresponding to the training image are formed through polarizers with different angles; the multiple polarization images corresponding to the training image and the polarization information corresponding to the training image are input into the first subnet of the neural network Path, output the reflection prediction map corresponding to the training image via the first subnet; input the multiple polarization maps corresponding to the training image and the reflection prediction map corresponding to the training image into the neural network The second sub-network of the road, through the second sub-network, outputs the transmitted light prediction map corresponding to the training image; at least the first sub-network is trained based on the transmitted light prediction map corresponding to the training image And the second subnet.

H，寬可以為

W。其中，所述訓練圖像對應的偏振圖可以是灰階圖。在該實現方式中，所述訓練圖像對應的偏振訊息可以所述訓練圖像對應的多個偏振圖確定。所述訓練圖像對應的反光預測圖可以表示為

，所述訓練圖像對應的透射光預測圖可以表示為

。第一子網路輸出的反光預測圖

作為第二子網路的輸入，可以用於得到更高品質的透射光預測圖

。在該實現方式中，所述訓練圖像對應的反光預測圖可以表示所述神經網路預測的所述訓練圖像對應的反光圖。所述訓練圖像對應的透射光預測圖可以表示所述神經網路預測的所述訓練圖像去除反光後的圖像。 In this implementation, the training image may be a single-channel image. For example, the height of the training image is H and the width is W. As an example of this implementation, the training image may include image information obtained through polarizers with 4 angles of 0°, 45°, 90°, and 135°. Correspondingly, the number of polarization patterns corresponding to the training image may be 4, and the 4 polarization patterns corresponding to the training image respectively correspond to the 4 polarizer angles of 0°, 45°, 90°, and 135°. For example, the four polarization maps corresponding to the training image can be represented as I ₁ , I ₂ , I ₃ and I ₄ . Among them, the heights of _{I 1} , I ₂ , I ₃ and I _{4 can be}

H , the width can be

W. Wherein, the polarization image corresponding to the training image may be a grayscale image. In this implementation manner, the polarization information corresponding to the training image may be determined by multiple polarization images corresponding to the training image. The reflection prediction map corresponding to the training image can be expressed as

, The transmitted light prediction map corresponding to the training image can be expressed as

. Reflective prediction map output by the first subnet

As the input of the second subnet, it can be used to obtain higher quality transmitted light prediction map

. In this implementation, the reflection prediction image corresponding to the training image may represent the reflection image corresponding to the training image predicted by the neural network. The transmitted light prediction image corresponding to the training image may represent the image predicted by the neural network after the reflection of the training image is removed.

As an example of this implementation, the first subnet and the second subnet may adopt a U-Net structure. Of course, the embodiments of the present invention are not limited to this, and those skilled in the art can also flexibly select the type and structure of the first subnet and the second subnet according to actual application scenario requirements and/or personal preferences.

In a possible implementation manner, the acquiring multiple polarization maps corresponding to the training image and the polarization information corresponding to the training image includes: separating pixels belonging to different polarizer angles in the training image; Obtaining multiple polarization maps corresponding to the training image; processing corresponding pixels in the multiple polarization maps corresponding to the training image to obtain polarization information corresponding to the training image. For example, if the training image includes image information obtained through 4 polarizers with angles of 0°, 45°, 90°, and 135°, the pixels belonging to 0° in the training image can be separated to obtain The first polarization image I ₁ corresponding to the training image is separated, the pixels belonging to 45° in the training image are separated, the second polarization image I ₂ corresponding to the training image is obtained, and the training image is separated The pixels belonging to 90° in the image are obtained, and the third polarization image I ₃ corresponding to the training image is obtained. The pixels belonging to 135° in the training image are separated to obtain the fourth polarization corresponding to the training image. Figure I ₄ .

In a possible implementation, the polarization information corresponding to the training image includes a first polarization information image corresponding to the training image, a second polarization information image corresponding to the training image, and a second polarization information image corresponding to the training image. At least one of the third polarization information image corresponding to the training image and the fourth polarization information image corresponding to the training image, wherein the first polarization information image corresponding to the training image is used to indicate the multiple information corresponding to the training image. The polarization intensity of a polarization image, the second polarization information image corresponding to the training image is used to indicate the polarization degree of the multiple polarization images corresponding to the training image, and the third polarization information image corresponding to the training image It is used to represent the polarization angle of the light corresponding to the training image, and the fourth polarization information map corresponding to the training image is used to represent the information after the overexposure is removed from the multiple polarization maps corresponding to the training image.

For example, the first polarization information map corresponding to the training image can be expressed as I , the second polarization information map corresponding to the training image can be expressed as ρ, and the third polarization information map corresponding to the training image can be expressed as φ and training image. The corresponding fourth polarization information graph can be denoted as O.

i

H，1

j

W。 As an example of this implementation, formula 1 can be used to obtain the first polarization information map I corresponding to the training image: I ( x ) = ( I ₁ ( x ) + I ₂ ( x ) + I ₃ ( x ) + I ₄ ( x ))/2 Equation 1, where x is any pixel in the figure, and the coordinate of pixel x is (i, j), where 1

i

H , 1

j

W.

As an example of this implementation, Equation 2 can be used to obtain the second polarization information map ρ corresponding to the training image:

As an example of this implementation, Equation 3 can be used to obtain the third polarization information map φ corresponding to the training image:

例如，δ=0.98。其中，若max{I ₁(x),I ₂(x),I ₃(x),I ₄(x)}>δ，則可以表明像素點x過曝；若max{I ₁(x),I ₂(x),I ₃(x),I ₄(x)}

δ，則可以 表明像素點x非過曝。在訓練圖像對應的第四偏振訊息圖O中，若像素點x過曝，則像素點x的像素值為0；若像素點x非過曝，則像素點x的像素值為1。 As an example of this implementation, formula 4 can be used to obtain the fourth polarization information map O corresponding to the training image:

For example, δ=0.98. Among them, if max { I ₁ ( x ), I ₂ ( x ), I ₃ ( x ), I ₄ ( x )}>δ, it can indicate that the pixel point x is overexposed; if max { I ₁ ( x ), I ₂ ( x ), I ₃ ( x ), I ₄ ( x ))

δ, it can indicate that the pixel point x is not overexposed. In the fourth polarization information map O corresponding to the training image, if the pixel point x is overexposed, the pixel value of the pixel point x is 0; if the pixel point x is not overexposed, the pixel value of the pixel point x is 1.

In the above, the method for determining the first polarization information image, the second polarization information image, the third polarization information image, and the fourth polarization information image corresponding to the image to be processed is the first polarization information image corresponding to the training image The methods for determining the graph, the second polarization information graph, the third polarization information graph, and the fourth polarization information graph are similar, and will not be repeated in the embodiment of the present invention.

Since the reflection image and the transmission light image have a large difference in polarization information, the neural network is trained by using the polarization information corresponding to the training image, so that the neural network can learn to recognize the reflection image and the transmission light image and The ability to separate them.

In a possible implementation manner, the hypercolumn in VGG-19 can be added to the input of the neural network to enhance the effect of the neural network. For example, before the multiple polarization maps corresponding to the training image and the polarization information corresponding to the training image are input into the first subnet of the neural network, the conv1_2 pair I ₁ , I _{2 of VGG-19 can be used.} , I ₃ , I ₄ and I are processed, and the processing results are up-sampled by bilinear interpolation, so that _{the sizes of I 1} , I ₂ , I ₃ , I ₄ and I after up-sampling are the same as the training images . In order to be applicable to VGG-19, gamma correction can be performed on the input image (training image or image to be processed) of the neural network first.

H，寬為

W。對I₁、I₂、I₃和I₄進行處理，可以得到訓練圖像對應的第一偏振訊息圖I、訓練圖像對應的第二偏振訊息圖ρ、訓練圖像對應的第三偏振訊息圖φ和訓練圖像對應的第四偏振訊息圖O。將I₁、I₂、I₃、I₄、I、ρ、φ和O輸入第一子網路RNet，可以得到訓練圖像對應的反光預測圖

。將I₁、I₂、I₃、I₄和

輸入第二子網路TNet，可以得到訓練圖像對應的透射光預測圖

。 Fig. 2 shows a schematic diagram of a neural network provided by an embodiment of the present invention. In the example shown in Figure 2, the height of the training image is H and the width is W. After the training image is preprocessed, four polarization maps I ₁ , I ₂ , I ₃ and I ₄ corresponding to the training image can be obtained. Among them, the heights of _{I 1} , I ₂ , I ₃ and I _{4 are}

H , the width is

W. By _{processing I 1} , I ₂ , I ₃ and I ₄ , the first polarization information image I corresponding to the training image, the second polarization information image ρ corresponding to the training image, and the third polarization information image corresponding to the training image can be obtained The image φ and the fourth polarization information image O corresponding to the training image. Input I ₁ , I ₂ , I ₃ , I ₄ , I , ρ, φ, and O into the first subnet RNet, and the reflection prediction map corresponding to the training image can be obtained

. Combine I ₁ , I ₂ , I ₃ , I ₄ and

Enter the second subnet TNet to get the transmitted light prediction map corresponding to the training image

.

In a possible implementation, the training of the first sub-network and the second sub-network at least according to the transmitted light prediction map corresponding to the training image includes: corresponding to the training image The transmitted light prediction map and the reflection prediction map corresponding to the training image are determined to determine the value of the first loss function; at least according to the value of the first loss function, the first subnet and the second subnet are trained network.

l

P，P表示所述第一預設網路的總層數；將所述歸一化的反光預測圖輸入所述第一預設網路，經由所述第一預設網路的第l層輸出所述歸一化的反光預測圖對應的第l層的特徵圖；根據所述歸一化的透射光預測圖對應的第l層的特徵圖與所述歸一化的反光預測圖對應的第l層的特徵圖之間的歸一化互相關值，確定第一損失函數的值。 As an example of this implementation, the determining the value of the first loss function according to the transmitted light prediction map corresponding to the training image and the reflection prediction map corresponding to the training image includes: The corresponding transmitted light prediction image and the reflection prediction image corresponding to the training image are respectively normalized to obtain a normalized transmission light prediction image and a normalized reflection prediction image corresponding to the training image; The normalized transmitted light prediction map is input into a first preset network, and the characteristic map of the first layer corresponding to the normalized transmitted light prediction map is output through the first layer of the first preset network , Where 1

l

P, P represents the total number of layers of the first preset network; input the normalized reflection prediction map into the first preset network, and pass through the first layer of the first preset network Output the feature map of the first layer corresponding to the normalized reflection prediction map; according to the feature map of the first layer corresponding to the normalized transmitted light prediction map corresponding to the normalized reflection prediction map The normalized cross-correlation value between the feature maps of the first layer determines the value of the first loss function.

In this implementation manner, the first preset network may be VGG-19 or ResNet-18, etc., which is not limited in the embodiment of the present invention.

，訓練圖像對應的反光預測圖可以表示為

。第一損失函數L _PNCC (I _A ,I _B )可以根據式5得到：

其中，

，

，

表示I _A 的歸一化結果，

表示I _B 的歸一化結果，

表示

輸入第一預設網路之後得到的第l層的特徵圖，

表示

輸入第一預設網路之後得到的第l層的特徵圖，n表示用於確定第一損失函數的總層數。例如，可以使用第一預設網路的conv2_2、conv3_2和conv4_2這三層輸出的特徵圖確定第一損失函數，那麽，在式5中，n=3。 For example, the transmitted light prediction map corresponding to the training image can be expressed as

, The reflection prediction map corresponding to the training image can be expressed as

. The first loss function L _PNCC ( I _A , I _B ) can be obtained according to Equation 5:

in,

,

,

Represents the normalized result of I _A,

Represents the normalized result of I _B,

Express

The feature map of the first layer obtained after inputting the first preset network,

Express

The feature map of the first layer obtained after inputting the first preset network, n represents the total number of layers used to determine the first loss function. For example, the first loss function can be determined using the feature maps output by the three layers of conv2_2, conv3_2, and conv4_2 of the first preset network. Then, in Equation 5, n=3.

，

，0

α

1，

表示訓練圖像對應的透射光預測圖，

表示訓練圖像對應的反光預測圖。如圖3所示，通過對所述訓練圖像對應的透射光預測圖和所述訓練圖像對應的反光預測圖分別進行歸一化處理，能夠使第一損失函數L _PNCC 隨著α的增大單調遞減。 3 shows a schematic diagram of the monotonicity of the first loss function L _PNCC before and after normalization of the transmitted light prediction map corresponding to the training image and the reflection prediction map corresponding to the training image in the embodiment of the present invention. in,

,

, 0

α

1,

Represents the transmitted light prediction map corresponding to the training image,

Indicates the reflection prediction map corresponding to the training image. As shown in FIG. 3, by normalizing the transmitted light prediction map corresponding to the training image and the reflection prediction map corresponding to the training image, respectively, the first loss function L _PNCC can be increased with α . Big monotonous decline.

In a possible implementation, the training of the first sub-network and the second sub-network at least according to the transmitted light prediction map corresponding to the training image includes: according to the training image and The difference between the reflective real image corresponding to the training image is used to obtain the transmitted light target image corresponding to the training image; the second loss is determined according to the transmitted light prediction image corresponding to the training image and the transmitted light target image The value of the function; training the first sub-network and the second sub-network at least according to the value of the second loss function.

In this implementation, the pixel value of the pixel in the training image can be subtracted from the pixel value of the corresponding pixel in the reflective real image corresponding to the training image to obtain the transmitted light corresponding to the training image Target graph. Wherein, the transmitted light target image corresponding to the training image may represent the ground truth of the transmitted light image corresponding to the training image, that is, the transmitted light target image corresponding to the training image may represent the training image. The basic truth of the image after the image has been de-reflected.

Figure 4 shows the background image B, the transmitted light image T, the reflected light image R and the mixed image M Schematic diagram. Among them, the background image B represents the image obtained by directly photographing the subject (ie, the background) without passing through the glass. The hybrid map M represents a map obtained by photographing the subject behind the glass through the glass. In the related technology, the background image B is used as the supervision information of the network. Since the photo taken through the glass will be refracted, the position of the corresponding image information in the image is different in the background image B and the photo (mixed image M with reflection). The embodiment of the present invention obtains the transmitted light target image corresponding to the training image according to the difference between the training image and the reflective real image corresponding to the training image, and uses the transmitted light target image as the neural network Supervising the information, which can make the transmitted light target image and the corresponding image information in the training image have the same position in the image, and can solve the problem of the background image and the corresponding image information in the mixed image in the image. The alignment problem can provide a high-quality training data set, and the neural network obtained by this training can more accurately remove the reflections in the input image and obtain a higher-quality output image in actual application. Among them, the training image is a mixed image M with reflection, and the true reflection image is the reference truth of the reflection image R corresponding to the training image. According to the embodiment of the present invention, the transmitted light target image T corresponding to the training image can be obtained according to T=MR. . The method provided by this implementation mode can handle various forms of reflected light, so that the image reflection problem caused by complex light sources in the real world can be dealt with, and the generalization ability is strong.

As an example of this implementation, before the obtaining the transmitted light target image corresponding to the training image according to the difference between the training image and the reflective real image corresponding to the training image, the method further includes : Collect training through the polarization sensor The image and the reflective real image corresponding to the training image.

Fig. 5 shows a schematic diagram of a method for acquiring a training image and a reflective real image corresponding to the training image. For example, you can cover the back of the glass with a black cloth to block all the transmitted light, collect the real reflection image through the polarization sensor, then remove the black cloth, and collect the corresponding training images through the polarization sensor. In the embodiment of the present invention, different types of glasses may be used to obtain the training image and the reflective real image corresponding to the training image, thereby obtaining rich and diverse training data.

In the embodiment of the present invention, the training image and the reflective real image corresponding to the training image are collected in the above-mentioned manner, and the difference between the training image and the reflective real image corresponding to the training image is used as the transmission corresponding to the training image. Optical target map, so there is no need to require glass to have a special material, thickness, color, etc., that is, the glass applicable to the embodiment of the present invention can be flat, curved, thin, thick, colored, and uncolored. , So that it can be applied to a wider range of application scenarios.

As another example of this implementation, the training image and the reflective real image corresponding to the training image can also be obtained through a simulation system.

k

Q，Q表示所述第二預設網路的總層數；將所述去除過曝的透射光目標圖輸入所述第二預設網路，經由所述第二預設網路的第k層輸出所述去除過曝的透射光目標圖對應的第k層的特徵圖；根據所述去除過曝的透射光預測圖對應的第k層的特徵圖與所述去除過曝的透射光目標圖對應的第k層的特徵圖之間的差值，確定第二損失函數的值。 In a possible implementation manner, the determining the value of the second loss function according to the transmitted light prediction map corresponding to the training image and the transmitted light target map includes: according to the transmitted light corresponding to the training image The product of the prediction image and the pixel value of the corresponding pixel in the fourth polarization information image corresponding to the training image is obtained to obtain the transmission light prediction image corresponding to the training image without overexposure, wherein, in the training image In the corresponding fourth polarization information image, the pixel value of the overexposed pixel is the first preset value, and the pixel value of the non-overexposed pixel is the second preset value, wherein the first preset value is less than The second preset value; according to the product of the transmitted light target image and the pixel value of the corresponding pixel in the fourth polarization information image corresponding to the training image, obtain the overexposed removal corresponding to the training image Transmitted light target map; input the overexposed transmitted light predicted map into a second preset network, and output the corresponding to the removed overexposed transmitted light predicted map through the kth layer of the second preset network The feature map of the k-th layer, where 1

k

Q, Q represents the total number of layers of the second preset network; input the overexposed transmitted light target map into the second preset network, and pass through the kth of the second preset network The layer outputs the feature map of the k-th layer corresponding to the overexposed transmitted light target image; according to the feature map of the k-th layer corresponding to the overexposed transmitted light prediction image and the removed overexposed transmitted light target The difference between the feature maps of the k-th layer corresponding to the map determines the value of the second loss function.

In this implementation, the second preset network may be VGG-19 or ResNet-18, etc., which is not limited in the embodiment of the present invention.

As an example of this implementation, the first preset value may be 0, and the second preset value may be 1. Of course, the embodiment of the present invention is not limited to this. For example, the first preset value may be 0.01, and the second preset value may be 1.

其中，T表示透射光目標圖，

表示透射光預測圖，O表 示訓練圖像對應的第四偏振訊息圖，β _k 表示第k層的權重，β _k 可以基於各層參數的個數初始化，O×T表示去除過曝的透射光目標圖，O×

表示去除過曝的透射光預測圖，v _k (O×T)表示將O×T輸入第二預設網路之後得到的第k層的特徵圖，

表示將O×

輸入第二預設網路之後得到的第k層的特徵圖，m表示用於確定第二損失函數的總層數。例如，可以使用第二預設網路的conv1_1、conv1_2、conv2_2、conv3_2、conv4_2和conv5_2這6層輸出的特徵圖確定第二損失函數，那麽，在式6中，m=6。 For example, the second loss function can be expressed by Equation 6:

Among them, T represents the transmitted light target image,

Represents the transmitted light prediction image, O represents the fourth polarization information image corresponding to the training image, β _k represents the weight of the k-th layer, β _k can be initialized based on the number of parameters of each layer, O × T represents the removal of the overexposed transmitted light target Figure, O ×

Represents the transmitted light prediction map with overexposure removed, v _k ( O × T ) represents the feature map of the k-th layer obtained after O × T is input into the second preset network,

Means O ×

The feature map of the k-th layer obtained after inputting the second preset network, m represents the total number of layers used to determine the second loss function. For example, the second loss function can be determined using the feature maps output from the 6 layers of conv1_1, conv1_2, conv2_2, conv3_2, conv4_2, and conv5_2 of the second preset network. Then, in Equation 6, m=6.

In a possible implementation manner, the loss function of the neural network may be equal to the sum of the first loss function and the second loss function.

In a possible implementation, when training the neural network, you can first use Adam's gradient descent optimization method, set the learning rate to 0.0001, train for 200 epochs (periods), and then set the learning rate to 0.00001, continue Train for 200 epochs. Among them, the number of training images used in each epoch can be adjusted according to the total number of training images.

FIG. 6 shows a schematic diagram of an output image obtained after processing an input image with three different types of reflections using the method for removing reflections in an image provided by an embodiment of the present invention. As shown in FIG. 6, the neural network provided by the embodiment of the present invention can accurately remove the reflective layer in the input image, and obtain a higher-quality output image after the reflection is removed.

The method for removing light reflection in an image provided by the embodiment of the present invention does not limit the type of light reflection and the type of light source, and can deal with the problem of image reflection caused by complex light sources in the real world, and has a wide range of application scenarios. In addition, the neural network training method provided by the embodiment of the present invention can quickly complete the training of the network. The embodiment of the present invention uses the depth network to quickly and accurately predict the transmitted light prediction map (that is, the map after the reflection is removed).

It can be understood that the various method embodiments mentioned in the present invention can be combined with each other to form a combined embodiment without violating the principle and logic. The length is limited, and the present invention will not be repeated.

Those skilled in the art can understand that in the above methods of the specific implementation, the writing order of the steps does not mean a strict execution order but constitutes any limitation on the implementation process. The specific execution order of each step should be based on its function and possibility. The inner logic is determined.

In addition, the present invention also provides devices, electronic equipment, computer-readable storage media, and programs for removing reflections in images, all of which can be used to implement any of the methods provided by the present invention for removing reflections in images, and the corresponding technical solutions And descriptions and refer to the corresponding records in the method section, so I won’t repeat them.

Fig. 7 shows a block diagram of a device for removing reflections in an image provided by an embodiment of the present invention. As shown in Figure 7, the device for removing reflections in an image includes: a first acquisition module 71 for acquiring an image to be processed; a second acquisition module 72 for acquiring Taking multiple polarization images corresponding to the image to be processed and polarization information corresponding to the image to be processed, wherein the multiple polarization images corresponding to the image to be processed are formed by polarizing plates with different angles; The first prediction module 73 is configured to determine the reflection prediction image corresponding to the image to be processed according to the multiple polarization images corresponding to the image to be processed and the polarization information corresponding to the image to be processed; the second prediction The module 74 is configured to determine a de-reflective image corresponding to the image to be processed according to the multiple polarization images corresponding to the image to be processed and the light reflection prediction image corresponding to the image to be processed.

In a possible implementation, the second acquisition module 72 is used to: separate pixels belonging to different polarizer angles in the image to be processed to obtain multiple polarization images corresponding to the image to be processed; Corresponding pixel points in the multiple polarization images corresponding to the image to be processed are processed to obtain polarization information corresponding to the image to be processed.

In a possible implementation, the polarization information corresponding to the image to be processed includes a first polarization information image corresponding to the image to be processed, a second polarization information image corresponding to the image to be processed, and At least one of the third polarization information image corresponding to the processed image and the fourth polarization information image corresponding to the image to be processed, wherein the first polarization information image corresponding to the image to be processed is used to represent the The polarization intensity of the multiple polarization images corresponding to the image to be processed, and the second polarization information image corresponding to the image to be processed is used to indicate the polarization degree of the multiple polarization images corresponding to the image to be processed. The third polarization information map corresponding to the processed image is used to indicate that the image to be processed corresponds to The polarization angles of the light of the multiple polarization images, the fourth polarization information image corresponding to the image to be processed is used to represent the information after the overexposure is removed from the multiple polarization images corresponding to the image to be processed.

In a possible implementation, the device further includes: a third acquisition module for acquiring multiple polarization maps corresponding to the training image, and polarization information corresponding to the training image, wherein the training image The multiple polarization images corresponding to the image are formed through polarizers with different angles; the third prediction module is used to input the multiple polarization images corresponding to the training image and the polarization information corresponding to the training image into the neural network The first sub-network of the road, through the first sub-network, outputs the reflection prediction map corresponding to the training image; the fourth prediction module is used to combine the multiple polarization maps corresponding to the training image and the The reflection prediction map corresponding to the training image is input to the second sub-network of the neural network, and the transmitted light prediction map corresponding to the training image is output through the second sub-network; the training module is used for at least Training the first sub-network and the second sub-network according to the transmitted light prediction map corresponding to the training image.

In a possible implementation, the third acquisition module is used to: separate pixels belonging to different polarizer angles in the training image to obtain multiple polarization maps corresponding to the training image; The corresponding pixel points in the multiple polarization images corresponding to the training image are processed to obtain the polarization information corresponding to the training image.

In a possible implementation, the polarization information corresponding to the training image includes the first polarization information map corresponding to the training image, and the training image corresponding to the first polarization information map. At least one of the second polarization information image corresponding to the training image, the third polarization information image corresponding to the training image, and the fourth polarization information image corresponding to the training image, wherein the first polarization information image corresponding to the training image The message image is used to represent the polarization intensity of the multiple polarization images corresponding to the training image, and the second polarization information image corresponding to the training image is used to represent the polarization degree of the multiple polarization images corresponding to the training image , The third polarization information diagram corresponding to the training image is used to represent the polarization angles of the light of the multiple polarization diagrams corresponding to the training image, and the fourth polarization information diagram corresponding to the training image is used to represent the The multiple polarization maps corresponding to the training images remove the overexposed information.

In a possible implementation, the training module is configured to: determine the value of the first loss function according to the transmitted light prediction map corresponding to the training image and the reflection prediction map corresponding to the training image; at least according to The value of the first loss function trains the first subnet and the second subnet.

l

P，P表示所述第一預設網路的總層數；將所述歸一化的反光預測圖輸入所述第一預設網路，經由所述第一預設網路的第l層輸出所述歸一化的反光預 測圖對應的第l層的特徵圖；根據所述歸一化的透射光預測圖對應的第l層的特徵圖與所述歸一化的反光預測圖對應的第l層的特徵圖之間的歸一化互相關值，確定第一損失函數的值。 In a possible implementation, the training module is used to: normalize the transmitted light prediction map corresponding to the training image and the reflection prediction map corresponding to the training image to obtain the training The normalized transmitted light prediction map and the normalized reflection prediction map corresponding to the image; input the normalized transmitted light prediction map into the first preset network, and the The first layer outputs the feature map of the first layer corresponding to the normalized transmitted light prediction map, where 1

l

P, P represents the total number of layers of the first preset network; input the normalized reflection prediction map into the first preset network, and pass through the first layer of the first preset network Output the feature map of the first layer corresponding to the normalized reflection prediction map; according to the feature map of the first layer corresponding to the normalized transmitted light prediction map corresponding to the normalized reflection prediction map The normalized cross-correlation value between the feature maps of the first layer determines the value of the first loss function.

In a possible implementation, the training module is configured to: obtain the transmitted light target image corresponding to the training image according to the difference between the training image and the reflective real image corresponding to the training image; The transmitted light prediction map and the transmitted light target map corresponding to the training image are used to determine the value of the second loss function; at least according to the value of the second loss function, the first subnet and the first subnet are trained Two subnets.

k

Q，Q表示所述第二預設網路的總層數；將所述去除過曝的透射光目標圖輸入所述第二預設 網路，經由所述第二預設網路的第k層輸出所述去除過曝的透射光目標圖對應的第k層的特徵圖；根據所述去除過曝的透射光預測圖對應的第k層的特徵圖與所述去除過曝的透射光目標圖對應的第k層的特徵圖之間的差值，確定第二損失函數的值。 In a possible implementation, the training module is used to: according to the product of the transmitted light prediction image corresponding to the training image and the pixel value of the corresponding pixel in the fourth polarization information image corresponding to the training image , Obtain the transmitted light prediction map corresponding to the training image without overexposure, wherein, in the fourth polarization information map corresponding to the training image, the pixel value of the overexposed pixel is the first preset value, The pixel value of the non-overexposed pixel is a second preset value, wherein the first preset value is less than the second preset value; according to the transmitted light target image and the training image corresponding to the first The product of the pixel values of the corresponding pixels in the four-polarization information image to obtain the target image of the transmitted light for removing the overexposure corresponding to the training image; input the predicted image of the transmitted light for removing the overexposed light into the second preset network, Output the feature map of the k-th layer corresponding to the overexposed transmitted light prediction map through the k-th layer of the second preset network, where 1

k

Q, Q represents the total number of layers of the second preset network; input the overexposed transmitted light target map into the second preset network, and pass through the kth of the second preset network The layer outputs the feature map of the k-th layer corresponding to the overexposed transmitted light target image; according to the feature map of the k-th layer corresponding to the overexposed transmitted light prediction image and the removed overexposed transmitted light target The difference between the feature maps of the k-th layer corresponding to the map determines the value of the second loss function.

In a possible implementation manner, the device further includes: a collection module for collecting the training image and the reflective real image corresponding to the training image through the polarization sensor.

In some embodiments, the functions or modules included in the device provided in the embodiments of the present invention can be used to execute the methods described in the above method embodiments. For specific implementation, refer to the description of the above method embodiments. For brevity, I won't repeat it here.

An embodiment of the present invention also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above-mentioned method when executed by a processor. Wherein, the computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium.

The embodiment of the present invention also provides a computer program product, which includes computer-readable code. When the computer-readable code runs on an electronic device, the processor in the electronic device executes to implement the above-mentioned method.

The embodiment of the present invention also provides another computer program product for storing computer-readable instructions. When the instructions are executed, the computer executes the operations of the method for removing reflections in the image provided by any of the foregoing embodiments.

An embodiment of the present invention also provides an electronic device, including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the memory stored in the memory Execute instructions to perform the above methods.

Electronic devices can be provided as terminals, servers, or other types of devices.

FIG. 8 shows a block diagram of an electronic device 800 provided by an embodiment of the present invention. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and other terminals.

8, the electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensing The device component 814, and the communication component 816.

The processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the foregoing method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the multimedia component 808 and the processing group Interaction between pieces 802.

The memory 804 is configured to store various types of data to support operations in the electronic device 800. Examples of these data include instructions for any application or method operated on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be realized by any type of volatile or non-volatile storage devices or their combination, such as static random access memory (SRAM), electronically erasable programmable read-only memory (EEPROM), erasable In addition to programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, floppy disk or optical disk.

The power supply component 806 provides power for various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with the generation, management, and distribution of power for the electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, sliding, and gestures on the touch panel. The touch sensor can not only sense the boundary of a touch or sliding action, but also detect the duration and pressure related to the touch or sliding operation. In some embodiments, the multimedia component 808 includes a front camera and/or a rear camera. When electronic When the device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC). When the electronic device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module. The peripheral interface module may be a keyboard, a mouse, a button, and the like. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

The sensor component 814 includes one or more sensors for providing the electronic device 800 with various aspects of state evaluation. For example, the sensor component 814 can detect the on/off status of the electronic device 800 and the relative positioning of the components. For example, the component is the display and the keypad of the electronic device 800, and the sensor component 814 can also detect the electronic device 800 or The position of a component of the electronic device 800 changes, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration/deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor component 814 may include a proximity sensor and is configured Used to detect the presence of nearby objects when there is no physical contact. The sensor component 814 may also include a light sensor, such as a CMOS (Complementary Metal Oxide Semiconductor) or CCD (Charge Coupled Device) image sensor for use in imaging applications. In some embodiments, the sensor component 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access wireless networks based on communication standards, such as Wi-Fi (wireless network), 2G (second-generation mobile communication technology), 3G (third-generation mobile communication technology), 4G/LTE (fourth generation) Mobile communication technology/long-term evolution technology), 5G (fifth generation mobile communication technology) or their combination. In an exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related messages from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), Field programmable logic gate array (FPGA), controller, microcontroller, microprocessor or other electronics Component implementation, used to implement the above method.

In an exemplary embodiment, there is also provided a non-volatile computer-readable storage medium, such as a memory 804 including computer program instructions, which can be executed by the processor 820 of the electronic device 800 to complete the above method.

FIG. 9 shows a block diagram of an electronic device 1900 provided by an embodiment of the present invention. For example, the electronic device 1900 may be provided as a server. 9, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions that can be executed by the processing component 1922, such as application programs. The application program stored in the memory 1932 may include one or more modules each corresponding to a set of commands. In addition, the processing component 1922 is configured to execute instructions to perform the above-mentioned methods.

The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input and output (I/O) Interface 1958. The electronic device 1900 can operate based on the operating system stored in the memory 1932, such as Windows Server® (Microsoft operating system), Mac OS X® (Macintosh operating system), Unix® (UNIX operating system), Linux® (LINUX operating system) System), FreeBSD® (FreeBSD system) or similar.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions. The instructions can be executed by the processing component 1922 of the electronic device 1900 to complete the above-mentioned methods.

The present invention can be a system, a method and/or a computer program product. The computer program product may include a computer-readable storage medium loaded with computer-readable program instructions for enabling the processor to implement various aspects of the present invention.

The computer-readable storage medium may be a tangible device that can hold and store instructions used by the instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable and programmable Modified read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital multi-function audio-visual disc (DVD), memory Cards, magnetic sheets, mechanical encoding devices, such as punched cards on which instructions are stored or raised structures in grooves, and any suitable combination of the above. The computer-readable storage medium used here is not interpreted as the instantaneous signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (for example, light pulses through fiber optic cables), or through wires Transmission signal.

The computer-readable program instructions described here can be downloaded from a computer-readable storage medium to various computing/processing devices, or through a network, such as the Internet, Local area network, wide area network and/or wireless network download to external computer or external storage device. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. The network interface card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for computer-readable storage in each computing/processing device In the media.

The computer program instructions used to perform the operations of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, or in one or more programming languages. Source code or object code written in any combination, the programming language includes object-oriented programming languages-such as Smalltalk (object-oriented language), C++ (programming language), etc., and conventional procedural programming languages-such as " C" language or similar programming language. The computer-readable program instructions can be executed entirely on the user's computer, partly on the user's computer, executed as a stand-alone software package, partly on the user's computer and partly executed on the remote computer, or entirely on the remote computer or Execute on the server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network-including a local area network (LAN) or a wide area network (WAN)-or it can be connected to an external computer (for example, using the Internet). Internet service provider to connect via the Internet). In some embodiments, the electronic circuit is personalized by using the status information of the computer-readable program instructions, such as programmable logic circuit, field programmable Logic Gate Array (FPGA) or Programmable Logic Array (PLA), the electronic circuit can execute computer-readable program instructions to realize various aspects of the present invention.

Here, various aspects of the present invention are described with reference to flowcharts and/or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each block of the flowchart and/or block diagram and the combination of each block in the flowchart and/or block diagram can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to the processors of general-purpose computers, dedicated computers, or other programmable data processing devices, so as to produce a machine that allows these instructions to be executed by the processors of the computer or other programmable data processing devices. At this time, a device that implements the functions/actions specified in one or more blocks in the flowchart and/or block diagram is produced. These computer-readable program instructions can also be stored in a computer-readable storage medium. These instructions make the computer, the programmable data processing device, and/or other equipment work in a specific manner, so that the computer-readable medium storing the instructions is It includes an article of manufacture, which includes instructions for implementing various aspects of the functions/actions specified in one or more blocks in the flowchart and/or block diagram.

It is also possible to load computer-readable program instructions onto a computer, other programmable data processing device, or other equipment, so that a series of operation steps are executed on the computer, other programmable data processing device, or other equipment to generate a computer realization The process of making the instructions executed on a computer, other programmable data processing device, or other equipment realize the requirements specified in one or more blocks in the flowchart and/or block diagram Function/action.

The flowcharts and block diagrams in the drawings show the possible implementation architecture, functions, and operations of the system, method, and computer program product according to multiple embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, program segment, or part of an instruction, and the module, program segment, or part of an instruction includes one or more logic for implementing the specified Executable instructions for the function. In some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two consecutive blocks can actually be executed in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, as well as the combination of blocks in the block diagram and/or flowchart, can be implemented by a dedicated hardware-based system that performs prescribed functions or actions. It can be realized, or it can be realized by a combination of dedicated hardware and computer instructions.

The computer program product can be implemented by hardware, software, or a combination thereof. In an alternative embodiment, the computer program product is specifically embodied as a computer storage medium. In another alternative embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc. Wait.

The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Do not deviate Given the scope and spirit of the described embodiments, many modifications and changes are obvious to those of ordinary skill in the art. The choice of terms used herein is intended to best explain the principles, practical applications, or improvements to the technology in the market for each embodiment, or to enable those of ordinary skill in the art to understand the various embodiments disclosed herein.

S11~S14: steps

Claims (11)

A method for removing reflections in an image includes: acquiring an image to be processed; acquiring a plurality of polarization maps corresponding to the image to be processed, and polarization information corresponding to the image to be processed, wherein the to-be-processed image The multiple polarization images corresponding to the image are formed by polarizing plates with different angles; the multiple polarization images corresponding to the image to be processed and the polarization information corresponding to the image to be processed are used to determine the image to be processed Corresponding light reflection prediction map; determine the light reflection-removed image corresponding to the image to be processed according to the multiple polarization maps corresponding to the image to be processed and the light reflection prediction map corresponding to the image to be processed. The method according to claim 1, wherein the acquiring multiple polarization maps corresponding to the image to be processed and the polarization information corresponding to the image to be processed includes: different polarizers in the image to be processed The pixel points of the angle are separated to obtain multiple polarization images corresponding to the image to be processed; the corresponding pixel points in the multiple polarization images corresponding to the image to be processed are processed to obtain the image corresponding to the image to be processed The polarization message. The method according to claim 1 or 2, wherein the polarization information corresponding to the image to be processed includes a first polarization information map corresponding to the image to be processed, and a second polarization information corresponding to the image to be processed Figure. At least one of the third polarization information map corresponding to the image to be processed and the fourth polarization information map corresponding to the image to be processed, wherein the image to be processed corresponds to The first polarization information graph is used to represent the polarization intensity of the multiple polarization images corresponding to the image to be processed, and the second polarization information graph corresponding to the image to be processed is used to represent the multiple polarization images corresponding to the image to be processed. The polarization degree of a polarization image, the third polarization information image corresponding to the image to be processed is used to indicate the polarization angles of the light of the multiple polarization images corresponding to the image to be processed, and the first image corresponding to the image to be processed The four-polarization information image is used to represent the information after the overexposure is removed from the multiple polarization images corresponding to the image to be processed. The method according to claim 1, wherein, before the obtaining the image to be processed, the method further includes: obtaining a plurality of polarization maps corresponding to the training image, and polarization information corresponding to the training image, wherein The multiple polarization images corresponding to the training image are formed through polarizers with different angles; the multiple polarization images corresponding to the training image and the polarization information corresponding to the training image are input into the first neural network. A sub-network, which outputs the reflection prediction map corresponding to the training image via the first sub-network; inputs the multiple polarization maps corresponding to the training image and the reflection prediction map corresponding to the training image into the office The second sub-network of the neural network outputs the transmitted light prediction map corresponding to the training image via the second sub-network; at least the transmission light prediction map corresponding to the training image is used to train the first A subnet and the second subnet. The method according to claim 4, wherein the training the first sub-network and the second sub-network at least according to the transmitted light prediction map corresponding to the training image includes: according to the training The transmitted light prediction image corresponding to the image and the training image Determine the value of the first loss function like the corresponding reflection prediction map; at least according to the value of the first loss function, train the first subnet and the second subnet. The method according to claim 5, wherein the determining the value of the first loss function according to the transmitted light prediction map corresponding to the training image and the reflection prediction map corresponding to the training image includes: The transmitted light prediction map corresponding to the training image and the reflection prediction map corresponding to the training image are respectively normalized to obtain a normalized transmission light prediction map and a normalized reflection prediction map corresponding to the training image Figure; the normalized transmitted light prediction map is input into the first preset network, and the first layer corresponding to the normalized transmitted light prediction map is output through the first layer of the first preset network Feature map, where 1

l

P, P represents the total number of layers of the first preset network; input the normalized reflection prediction map into the first preset network, and pass through the first layer of the first preset network Output the feature map of the first layer corresponding to the normalized reflection prediction map; according to the feature map of the first layer corresponding to the normalized transmitted light prediction map corresponding to the normalized reflection prediction map The normalized cross-correlation value between the feature maps of the first layer determines the value of the first loss function. The method according to any one of claims 4 to 6, wherein the training of the first subnet and the second subnet is based on at least the transmitted light prediction map corresponding to the training image, It includes: obtaining the transmitted light target image corresponding to the training image according to the difference between the training image and the reflective real image corresponding to the training image; Determine the value of the second loss function according to the transmitted light prediction map and the transmitted light target map corresponding to the training image; at least according to the value of the second loss function, train the first subnet and the The second subnet. The method according to claim 7, wherein the determining the value of the second loss function according to the transmitted light prediction map corresponding to the training image and the transmitted light target map includes: corresponding to the training image The product of the predicted transmitted light map and the pixel values of the corresponding pixels in the fourth polarization information map corresponding to the training image to obtain the predicted transmitted light map corresponding to the training image without overexposure, wherein, in the In the fourth polarization information image corresponding to the training image, the pixel value of the overexposed pixel is the first preset value, and the pixel value of the non-overexposed pixel is the second preset value, wherein the first preset Set the value to be smaller than the second preset value; according to the product of the pixel value of the corresponding pixel in the fourth polarization information image corresponding to the transmitted light target image and the training image, the removal corresponding to the training image is obtained Overexposed transmitted light target map; input the overexposed transmitted light prediction map into a second preset network, and output the overexposed transmitted light prediction through the kth layer of the second preset network The feature map of the k-th layer corresponding to the image, where 1

k

Q, Q represents the total number of layers of the second preset network; input the overexposed transmitted light target map into the second preset network, and pass through the kth of the second preset network The layer outputs the feature map of the k-th layer corresponding to the overexposed transmitted light target image; according to the feature map of the k-th layer corresponding to the overexposed transmitted light prediction image and the removed overexposed transmitted light target The difference between the feature maps of the k-th layer corresponding to the map determines the value of the second loss function. The method according to claim 7, wherein, before obtaining the transmitted light target image corresponding to the training image according to the difference between the training image and the reflective real image corresponding to the training image, the The method further includes: collecting a training image and a reflective real image corresponding to the training image through a polarization sensor. An electronic device includes: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute such as The method described in any one of claims 1 to 9. A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the method described in any one of claim items 1 to 9.

Images