JP2021026451A - Image processor, image processing method, and program - Google Patents

Abstract

To realize processing using a model that appropriately learns an object in which contrast is decreased by a fog and a haze.SOLUTION: An image processor for generating a corrected image in which at least one portion of an influence of scattered light is removed for an image to be processed includes: determination means for determining whether a blank area is included for an image to be processed; first correction processing means for executing first correction processing to an image determined not to include a blank area by the determination means; and second correction processing means for executing second correction processing to an image determined to include a blank area by the determination means. The first correction processing executed by the first correction processing means is processing using a model that does not set an image only in a blank area having less influence of scattered light to be teacher data, but learns an image in an area other than a blank area with less influence of scattered light as teacher data.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing technique for reducing the influence of scattered light on a captured image.

In fields such as surveillance cameras, there is a problem that the contrast is lowered due to the influence of fine particles (for example, fog and haze) existing between the camera and the subject, and the image quality of the captured image is deteriorated. The cause of this is that when light passes through the atmosphere, the light is scattered by the fine particle components. An image processing technique for removing the influence of scattered light on an image whose visibility is deteriorated due to such scattered light is known.

Patent Document 1 describes a method of using machine learning as a mist removal process. Specifically, when a region of the fog image is input, the learning model generates a transmittance corresponding to the region, and a fog removal image is generated using the transmittance obtained by the learning model.

U.S. Patent Publication 2016/0005152

The change in visibility differs depending on the subject due to the influence of fine particles such as fog and haze. For example, in a textured building, the contrast decreases, while in an empty area with less texture, the hue changes. However, in the method according to Patent Document 1 described above, when learning to output a transmittance map from an image, it is not considered that the way of changing the visibility differs depending on the subject, and an appropriate learning model can always be obtained. Absent.

Therefore, an object of the present invention is to realize a process using a model appropriately learned for a subject whose contrast is lowered by the fog haze when performing a fog haze removal process on an image by using machine learning. ..

In order to solve the above problems, the present invention is an image processing apparatus that generates a corrected image in which at least a part of the influence of scattered light is removed from the image to be processed, and the image to be processed is subjected to the present invention. A determination means for determining whether or not an empty area is included, a first correction processing means for executing a first correction process on an image determined not to include an empty area by the determination means, and the determination. The first correction process executed by the first correction processing means, which has a second correction processing means for executing a second correction process on an image determined to include an empty area by the means. Is a process using a model in which an image of only an empty region, which is less affected by scattered light, is used as training data, and an image of a region other than the sky, which is less affected by scattered light, is used as training data.

INDUSTRIAL APPLICABILITY The present invention can realize a process using a model appropriately learned for a subject whose contrast is lowered by the fog haze when performing a fog haze removal process on an image by using machine learning.

The figure which shows the hardware configuration of an image processing apparatus The figure which shows the detailed functional structure of an image processing apparatus Diagram showing an example of a learning image Diagram explaining a neural network Flow chart of learning process Flowchart of fog haze removal process The figure which shows the detailed functional structure of an image processing apparatus Diagram showing GUI for inputting indoor / outdoor information Flowchart of fog haze removal process

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings, based on preferred embodiments. The configuration shown in the following embodiments is only an example, and the present invention is not necessarily limited to the illustrated configuration.

<First Embodiment>
In the first embodiment, an image processing apparatus that executes fog haze removal processing on an image will be described. The fog haze removal process means a process of reducing the influence of scattered light on an image whose contrast is lowered due to the influence of scattered light by fine particle components such as fog and haze. In the present embodiment, the empty area of the image is corrected for each pixel using a preset parameter, and the correlation between the image without fog haze and the image with fog haze is obtained for the area other than the empty area. Correction is performed using the parameters of the training model obtained by training.

FIG. 1 is a block diagram showing a hardware configuration of the image processing device 100 of the present embodiment. The image processing device 100 of this embodiment has a CPU 101, a RAM 102, a ROM 103, an HDD I / F 104, an HDD 105, an input I / F 106, an output I / F 107, and a system bus 108. The CPU 101 is a processor that comprehensively controls each configuration described below. The RAM 102 is a memory that functions as a main memory and a work area of the CPU 101, and a ROM 103 is a memory that stores a program that controls processing in the image processing device 100.

The HDD I / F 104 is an interface such as a serial ATA (SATA), and connects the HDD 105 as a secondary storage device to the system bus 108. The CPU 101 can read data from the HDD 105 and write data to the HDD 105 via the HDD I / F 104. Further, the CPU 101 can expand the data stored in the HDD 105 into the RAM 102, and similarly, can store the data expanded in the RAM 102 in the HDD 105. Then, the CPU 101 can consider the data expanded in the RAM 102 as a program and execute it. The secondary storage device may be a storage device such as an optical disk drive in addition to the HDD. The input I / F 106 is a serial bus interface such as USB or IEEE1394.

The image processing device 100 is connected to the external memory 109 and the image pickup unit 111 via the input I / F 106. The CPU 101 can acquire data from the external memory 109 and the imaging unit 111 via the input I / F 106. The output I / F 107 is a video output interface such as DVI or HDMI (registered trademark). The image processing device 100 is connected to the display unit 110 via the output I / F 107. The CPU 101 can output an image to the display unit 110 via the output I / F 107, and display the output image on the display unit 110.

The system bus 108 is a transfer path for various data, and each component in the image processing device 100 is connected to each other via the system bus 108.

The external memory 109 is a storage medium such as a hard disk, a memory card, a CF card, an SD card, or a USB memory, and can store image data or the like processed by the image processing device 100. The display unit 110 is a display device such as a display, and can display an image or the like processed by the image processing device 100.

The image pickup unit 111 is a camera that receives light information of a subject by a sensor and outputs the acquired image as digital data. In the captured image obtained by the imaging unit 111, the contrast may be lowered due to the influence of scattered light such as a scene in which fog haze or the like is generated. In the present embodiment, by executing the fog haze removal process on the image captured by the imaging unit 111, it is possible to generate an image in which the influence of scattered light is reduced.

FIG. 2 is a block diagram showing a detailed logical configuration of the image processing apparatus 100 according to the present embodiment. The image processing device 100 is composed of a fog haze removal processing unit 200 that executes a fog haze removal process and a learning data generation unit 201 for generating learning data.

First, the learning data generation unit 201 will be described. The learning data generation unit 201 generates learning data that is a pair of student data and teacher data for optimizing the parameters of the learning model that performs machine learning. The learning model is a network structure based on a neural network that outputs a fog haze reduction image corresponding to the student data from the input student data and its parameters. The learning model in the present embodiment is a model that estimates an image in which the influence of the scattered light by the fog haze is reduced from the image affected by the scattered light by the fog haze. Student data is generated from images whose contrast is reduced by fog haze. Further, the teacher data is generated from the fog haze reduction image when there is no contrast reduction due to fog haze in the image which is the student data.

The learning data generation 201 has a learning image acquisition unit 202, a block division unit 203, an area determination unit 204, and a learning unit 205. The learning image acquisition unit 202 acquires two images obtained by capturing the same scene from the same position under the same imaging conditions. FIG. 3A is a diagram showing a part of the image acquired by the learning image acquisition unit 202. The two images acquired at this time are a clear sky image obtained by capturing the same landscape from the same position in fine weather, and a fog haze image taken under the influence of scattered light due to fog haze and the like. Here, it is assumed that the clear sky image and the fog haze image are captured under the same imaging conditions (exposure, angle of view, etc.). The clear sky image and the fog haze image are both color images composed of R (red), G (green), and B (blue) planes. The learning image acquisition unit 202 acquires a plurality of pairs of such a clear sky image and a fog haze image.

The block division unit 203 divides the pair of the acquired clear sky image and the fog haze image, and outputs the block image. As will be described later, the block division unit 203 divides the image into blocks according to the number of nodes in the input layer of the neural network in the learning unit 204. The block division unit 203 outputs the divided block image pairs to the area determination unit 204 in order. The area determination unit 204 determines whether or not an empty area is included based on the blocks extracted from the clear sky image among the block pairs to be processed. The area determination unit 204 does not output the block determined to include the empty area to the learning unit 204, and outputs the block determined not to include the empty area to the learning unit 205.

The learning unit 205 learns the parameters of the neural network held inside by using the pair of block images that do not include the empty area output from the block dividing unit 203.

Here, the neural network will be described. FIG. 4 is a diagram illustrating a neural network. In FIG. 4, the intermediate layer is one layer for the sake of simplicity, but it is desirable that the intermediate layer is composed of two or more layers. In the neural network shown in FIG. 4, the input layer has Mi nodes (n11, n12, ..., N1Mi), the intermediate layer has Mh nodes (n21, n22, ..., N2Mh), and the output layer. The (final layer) has Mo nodes (n31, n32, ..., N3Mo). The nodes of each layer are connected to all the nodes of the adjacent layers, and form a three-layer hierarchical neural network that transmits information between the layers.

When an image is input to the input layer, the input layer is provided with nodes for the number of pixels to be input so that the input pixels and the nodes are one-to-one. Also, in the output layer, nodes for the number of pixels to be output are set. That is, in the present embodiment, since the block image of 16 pixels × 16 pixels is input and the pixel value of 16 pixels × 16 pixels is output, the number of nodes in the input layer and the output layer is 256. Data is passed from left to right in FIG. 4, that is, in the order of input layer, intermediate layer, and output layer. Each node in the input layer is connected to all the nodes in the middle layer, and the connections between the nodes have their own weights. The output value transmitted from one node through the join to the other node is enhanced or attenuated by the weight of the join. The set of weighting factors and bias values defined for such a connection is a parameter of the learning model. The activation function is not particularly limited, but a logistic sigmoid function, a Rectifier Unit (ReLU) function, or the like may be used. As a learning method, various proposed neural network learning methods can be applied. For example, the difference between the output obtained from the output layer when the student data is input to the input layer and the neural network is operated and the teacher data associated with the student data in advance is calculated, and the difference is calculated. Adjust the weighting coefficient and bias value to minimize it. The weighting coefficient and the bias value generated by such a learning process are parameters of the learning model. Since the learning unit 205 in the present embodiment uses the fog haze image as student data and the clear sky image as teacher data, a learning model that estimates an image in which the influence of scattering is reduced from the image affected by the scattered light by the fog haze. Generate parameters for.

Next, the flow of the learning process in the learning data generation unit 201 will be described in detail. FIG. 5 is a flowchart of the learning process. Each configuration (function) is realized by the CPU 101 reading and executing a program capable of realizing the flowchart shown in FIG. In the following, each process (step) in the flowchart will be described with "S".

In S501, the learning image acquisition unit 202 acquires a set of images prepared for learning from a memory (not shown). As described above, here, a clear sky image obtained by capturing the same wind condition from the same position in fine weather and a fog haze image taken under the influence of scattered light due to fog haze or the like are acquired. For example, two image data may be acquired by referring to the weather forecast from an external memory in which an image captured by a fixed surveillance camera installed at a predetermined position is stored. In this case, it is preferable to acquire an image pair from images captured by a plurality of different surveillance cameras.

In S502, the block division unit 203 divides a set of images into blocks. In the present embodiment, block images of 16 pixels × 16 pixels are extracted in order from the upper left of the image. FIG. 3B is a diagram illustrating a block to be divided by the block dividing unit 203. As shown in FIG. 3B, the image is divided into blocks in a tile shape, and a plurality of block images are extracted from one image. The block division unit 203 outputs the two blocks extracted from the same position in the clear sky image and the corresponding fog haze image to the area determination unit 204 as one block pair.

In S503, the area determination unit 204 determines whether or not the block image of the clear sky image includes an empty area among the acquired block image pairs. As a method for determining whether or not the region is empty, a known technique may be applied. In the present embodiment, the hue is calculated from the pixel values of R, G, and B of each pixel in the block, and it is determined whether or not the hue is in the preset range of hues that are likely to be empty. The area determination unit 204 calculates the number of pixels determined to be pixels in the empty area in the block, and if the number of pixels in the empty area is equal to or greater than a predetermined threshold, determines that the block image includes an empty area. S504 skips and proceeds to S505. If the number of pixels in the empty area is less than a predetermined threshold value, the area determination unit 204 determines that the block image does not include the empty area and proceeds to S504.

In S504, the area determination unit 204 outputs only the pair of block images determined not to include the empty area to the learning unit 205. In S505, the area determination unit 204 determines whether or not the area determination has been performed on all of the acquired block images. If the area is determined for all the block images, the process returns to S506, and if there is an unprocessed block image, the process returns to S503. Further, in S506, the learning image acquisition unit 202 determines whether or not all the learning images have been acquired. If all the learning images have been acquired, the process proceeds to S507, and if there are unacquired learning images, the process returns to S501.

In S507, the learning unit 205 generates parameters of a learning model that estimates an image in which the influence of scattering is reduced by using the input learning data. In S508, the learning unit 205 outputs the parameters of the generated learning model to the first correction unit 209 described later.

From here, the fog haze removal processing unit 200 that actually executes fog haze removal will be described. First, the image input unit 206 inputs an image to be processed for which the influence of scattering due to fog haze is to be removed. The block division unit 207 divides the image to be processed into blocks. In the present embodiment, the block division unit 207 divides into blocks having the same size as the block image extracted by the block division unit 203. The area determination unit 208 determines whether or not the block to be processed includes an empty area. The area determination unit 208 outputs a block that does not include an empty area to the first correction unit 209, and outputs a block that includes an empty area to the second correction unit 210.

The first correction unit 209 includes a neural network. Here, the first correction unit 209 has the same network structure as the learning unit 205. Further, each weighting coefficient and bias value in the neural network are set according to the parameters output from the learning unit 205. That is, the second correction unit 209 executes the correction processing on the block to be processed based on the model for removing the influence of the fog haze learned by using the image that does not include the empty area. The first correction unit 209 estimates an image having the original contrast obtained by removing the influence of fog haze from the image to be processed, and outputs the corrected block.

The second correction unit 210 sequentially executes arithmetic processing according to a predetermined algorithm for each pixel included in the block to be processed, and outputs the corrected block of the block including the empty area. It is known that when the image is originally affected by the fog haze, the contrast is lowered as compared with the image when the fog haze is not affected. However, the empty area often does not contain a complicated texture component, and if the contrast is increased as a fog haze removal process, the S / N ratio is lowered. Therefore, in the present embodiment, the second correction unit 210 executes the correction process suitable for the empty area.

The output image generation unit 211 accumulates the block image output from the first correction unit 209 and the block image output from the second correction unit 210 to generate and output one image.

The flow of the mist haze removal process executed by the mist haze removal process unit 200 will be described in detail. FIG. 6 is a flowchart of the fog haze removal process. Each configuration (function) is realized by the CPU 101 reading and executing a program capable of realizing the flowchart shown in FIG.

First, in S601, the image input unit 206 inputs an image to be processed. Here, as with the learning image data, a color image composed of R, G, and B planes is used as the image to be processed. In S602, the block division unit 203 divides the image to be processed. In the present embodiment, the image to be processed is divided into blocks of 16 pixels × 16 pixels.

In S603, the area determination unit 208 determines whether or not the block to be processed includes an empty area. Here, the area determination for the block is performed by the same method as the area determination unit 204 in S503 described above. When the area determination unit 208 determines that the block includes an empty area, the area determination unit 208 proceeds to S604 and outputs the block image to the first correction unit 209. If the area determination unit 208 determines that the block does not include an empty area, the area determination unit 208 proceeds to S605 and outputs the block image to the second correction 210.

In S605, the first correction unit 209 executes the first correction process on the input block image. Here, the first correction process is a process by a neural network in which parameters based on a learning model for estimating a fog haze removal image are set. By this process, the block image is converted into a corrected block in which the influence of the fog haze with increased contrast is reduced.

In S607, the second correction unit 210 executes the second correction process on the block image including the input empty area. In the second correction process in the present embodiment, the saturation correction process is executed. The second correction unit 210 sequentially sets the pixels included in the block to be processed as the pixels of interest, and first converts the pixel values of R, G, and B of the pixels of interest into the YCbCr color space. The color space conversion process is performed using a known calculation formula. The second correction unit 201 multiplies the Cb and Cr components by a predetermined gain.

In S608, the area determination unit 208 determines whether or not the determination process has been executed for all the blocks in the image to be processed, and if there are unprocessed blocks, returns to S603. If the determination processing for all is completed, the process proceeds to S609.

In S609, the output image generation unit 211 accumulates the data of the corrected block in the image to be processed, generates one corrected image, and outputs it. This completes the fog haze removal process.

As described above, in the present embodiment, a corrected image in which the influence of scattered light is reduced is generated by using a learning model obtained by supervised learning for a subject area that does not include an empty area in the image. On the other hand, for the empty region, a corrected image in which the influence of scattered light is reduced is generated by the saturation correction processing.

Normally, when affected by scattered light, the contrast decreases in regions other than the empty region. Therefore, it is desirable that the image from which the influence of the scattered light is removed is a correction process that enlarges the contrast as compared with the image affected by the scattered light. Therefore, the correction process using the learning model for estimating the image without reduction by using the pair of the image with reduced contrast and the image without reduction is suitable for the region other than the sky. However, in the empty region, since there is originally no texture component, the image quality may deteriorate due to the increase in the noise component when the contrast is increased. When a model trained with supervised learning using a clear sky image as teacher data is applied to an empty area, the image quality of the empty area deteriorates due to the expansion of contrast. That is, for the empty region, instead of the correction processing using the learned model, the saturation correction processing suitable for the empty region is executed to generate a corrected image with better reduced influence of scattered light. Can be done.

In addition, the effect of scattered light in the sky region is different from the effect of scattered light in regions other than the sky. For example, in the case of an empty region, the image is bluish in fine weather and grayish when affected by scattered light. Therefore, in the empty region, it is desirable that the achromatic color such as gray is corrected to be bluish by the fog haze removal treatment. On the other hand, in areas other than the sky, there are objects such as buildings whose subject itself is gray. Therefore, in a region other than the sky, even if it is gray in the fog haze image, it is not always desirable to correct it to blue as a result of the fog haze removal process. Therefore, when learning a model that estimates an image obtained by removing the influence of fog haze from a fog haze image using a clear sky image as teacher data, by learning using an image in a region other than the sky, it is possible to use an image in a region other than the sky. You can learn the learning model more appropriately.

<Second embodiment>
In the first embodiment, a method of determining whether or not an empty area is included in the image to be processed and switching between the first correction process and the second correction process has been described. In the present embodiment, a method of controlling the correction process by acquiring the position information and the posture information in which the image pickup device that has captured the image to be processed is installed will be described. The same components as those in the first embodiment are designated by the same reference numerals, and the description thereof will be omitted.

In addition to the first embodiment, the fog haze removing processing unit 700 in the present embodiment includes a position information acquisition unit 810, a posture information acquisition unit 711, and an image determination unit 712. The position information acquisition unit 710 causes the user to display the graphical user interface (GUI) shown in FIG. 8 and accepts input from the user. As a result, the position information acquisition unit 710 acquires indoor / outdoor information indicating whether or not the image pickup unit 111 is installed indoors or outdoors as position information.

The posture information acquisition unit 711 acquires information indicating the posture of the imaging unit 111 when the image to be processed is captured. Here, the posture information acquisition unit 711 acquires information regarding the elevation angle direction of the image pickup unit 111 as posture information from the image pickup unit 111. That is, the image pickup unit 111 has a built-in angle sensor (not shown) capable of detecting the angle in the elevation angle direction, and when the image to be processed is transferred to the fog haze removal processing unit 700, the angle of the angle sensor Information is also transferred.

The image determination unit 712 determines whether or not the image to be processed may include an empty area. The image to be processed is output to any of the block division unit 207, the first correction unit 209, and the second correction unit 210 according to the determination result.

The flow of the mist haze removal process executed by the mist haze removal process unit 700 will be described in detail. The parameters of the neural network in the first correction unit 209 are updated by the parameters adjusted by the same learning method as in the first embodiment. FIG. 9 is a flowchart of the fog haze removal process.

In S901, the fog haze removal processing unit 700 acquires an image to be processed. In S902, the position information acquisition unit 710 causes the display unit 110 to display the image shown in FIG. In S903, the position information acquisition unit 710 acquires indoor / outdoor information indicating the result of the user selecting either outdoor or indoor via the GUI.

In S905, the posture information acquisition unit 711 acquires information regarding the elevation angle direction of the image pickup unit 111 from the image pickup unit 111 as posture information. In the present embodiment, the attitude information acquisition unit 711 is -90 degrees when the image pickup unit 111 is facing the ground directly below, 0 degrees in the horizontal direction, and is facing the ceiling or the sky directly above. The angle in the elevation angle direction is acquired as 90 degrees.

In S905, the image determination unit 712 determines whether or not there is a possibility that the image to be processed includes an empty area based on the position information and the posture information. When indoors are selected by user input, and when outdoor is selected and the posture information indicates 45-90, it is determined that the image to be processed does not include an empty area. This is because even if the image is taken outdoors, it is unlikely that the image is oriented toward the ground or the floor and includes an empty area. When it is determined that the image to be processed does not include an empty area, the image determination unit 712 then proceeds to S911 and outputs the image to be processed to the second correction unit 210. If outdoor is selected and the posture information shows 45 to 90 degrees, it is determined that an empty area is included. If it is determined that the image includes an empty area, the image determination unit 712 proceeds to S906.

In S906, the image determination unit 712 further determines whether or not the image to be processed is only an empty area. In the present embodiment, the image determination unit 712 calculates the hue from the pixel value in the color image and stores it. Further, the image determination unit 712 counts the number of pixels in the range where the hue of each pixel is recognized as empty, and when a predetermined ratio, for example, 95% or more of the pixels are empty hues, only the empty area is used. Is determined to be. When the image determination unit 712 is an image having only an empty area, the process proceeds to S907, and the image to be processed is output to the first correction unit 209. On the other hand, when it is determined that the empty area is included but not only the empty area, the image determination unit 712 proceeds to S909.

S909 and S910 are the same processes as S902 and S903 in the first embodiment. However, in S903, the area determination unit 208 does not calculate the hue of each pixel, but the image determination unit 712 may refer to the calculated and stored hue of each pixel.

In S907, a block image including an empty area or an image of only an empty area is input to the second correction unit 210. In S908, the second correction unit 210 executes the second correction process on the input image.

In S911, a block image that does not include an empty area or an image that does not include an empty area is input to the first correction unit 210. In S912, the first correction unit 209 executes the first correction process on the input image. In S913, when the correction processing for all the pixels is completed, the output image generation unit 211 outputs the corrected image.

As described above, in the present embodiment, the fog haze removal process for the image is controlled according to the position information and the posture information. The fog haze removal process is classified into three methods, and the image is processed by one of the methods. The first is a method of executing a correction process for executing arithmetic processing for each pixel on an image, and the second is a method of executing a correction process using a neural network on an image. The second is a method of switching between the first method and the second method according to the block. For example, in the case of a surveillance camera, once it is installed, it is often fixed for a while, and especially when it is installed indoors, it is unlikely that the image captured by the surveillance camera contains an empty area. Therefore, in the present embodiment, it is first determined whether or not the image to be processed includes an empty area based on indoor / outdoor information and posture information. As a result, when only the empty area or not including the empty area, the block division and the area determination process are not executed, and the correction process suitable for each case can be executed. As a result, the processing load can be reduced. The process by the image determination unit 710 does not have to be executed every time the image to be processed is input. It may be executed only when the installation position is changed or the setting is reset, and in other cases, the result of the determination once may be diverted.

<Other Embodiments>
In the above-described embodiment, a case of learning a model for estimating an image in which fog haze is reduced from a fog haze image has been described using a clear sky image as teacher data. However, there are other methods for correction processing using opportunity learning. For example, a learning model for estimating the transmittance map from the fog haze image may be learned by learning the fog haze image as student data and the transmittance map corresponding to the fog haze image as teacher data. When executing the correction process in which the parameters of such a learning model are set, first, the transmittance map corresponding to the image to be processed is estimated via the neural network. After that, the process of adjusting the contrast for each pixel in the image to be processed may be executed with reference to the transmittance map. Since the fog haze removal treatment using the transmittance map is known, the description thereof will be omitted.

In the above-described embodiment, a mode in which it is determined whether or not the area is empty for each block and the first or second correction process is executed for each block has been described as an example. However, the processing does not necessarily have to be for each rectangular block. For example, it is divided into an empty region and a non-empty region by a known semantic region division, and a second correction process is performed on an image in which only an empty region is extracted, and a second correction process is performed on an image in which only a non-empty region is extracted. The correction process of 1 may be executed.

Further, in the description of the first embodiment, it has been described on the premise that the image to be processed contains a mixture of an empty area and a non-empty area. However, if the image does not include an empty area, the first correction process is executed for the entire area in the image.

In the above-described embodiment, an example in which a pair of a clear sky image and a fog haze image is used as a learning image has been described. However, there may be no two images captured from the same position under the same imaging conditions. For example, an image in which the fog haze is reduced by using a known fog haze removal process such as a dark channel method may be generated for the fog haze image and used as teacher data.

Further, as the region determination processing, a method of calculating the hue from the pixel value in the color image has been described, but for example, it may be determined whether or not the region is an empty region by using a distance image corresponding to the image.

In the above description, fog and fog have been described as examples of factors that generate scattered light, but for images whose contrast is reduced by scattered light due to other fine particle components such as yellow sand and PM2.5. Can be applied in the same way.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions. The input I / F 106 is a serial bus interface such as USB or IEEE1394.

202 Learning image acquisition unit 203 Block division unit 204 Area determination unit 205 Learning unit 206 Image input unit 207 Block division unit 208 Area determination unit 209 First correction unit 210 Second correction unit 211 Output image generation unit

Claims (12)

An image processing device that generates a corrected image in which at least a part of the influence of scattered light is removed from the image to be processed.
A determination means for determining whether or not an empty area is included in the image to be processed,
A first correction processing means that executes the first correction processing on an image determined not to include an empty area by the determination means, and
It has a second correction processing means for executing a second correction processing on an image determined to include an empty area by the determination means.
In the first correction process executed by the first correction processing means, the image of only the sky region less affected by the scattered light is not used as the teacher data, and the image of the region other than the sky less affected by the scattered light is used as the teacher data. An image processing device characterized in that it is a process using a model learned as. The image processing apparatus according to claim 1, wherein the first correction processing means has a neural network and parameters corresponding to the model are set. The image processing apparatus according to claim 1 or 2, wherein the second correction process is a pixel-by-pixel calculation process. The image processing apparatus according to any one of claims 1 to 3, wherein the determination means determines for each block of the image to be processed. The determination means determines the image to be processed and determines the image.
Claim 1 is characterized in that, when it is determined that the image to be processed does not include an empty area, only the first correction process by the first correction processing means is executed for the image to be processed. The image processing apparatus according to any one of 3 to 3. The image according to claim 5, wherein when the determination means determines that the image to be processed includes an empty area, it further determines whether or not the image to be processed contains only an empty area. Processing equipment. The sixth aspect of claim 6, wherein when it is determined that the determination means is only an empty area, only the second correction processing by the second correction processing means is executed for the image to be processed. Image processing equipment. When the determination means determines that the image to be processed includes an empty area but not only the empty area, the image to be processed is subjected to the first correction process or the second correction for each block. The image processing apparatus according to claim 6 or 7, wherein any of the processes is executed. The determination means is characterized in that it determines whether or not an empty region is included based on information indicating whether the image pickup device that has captured the image to be processed is installed indoors or outdoors. The image processing apparatus according to any one of claims 5 to 8. The image according to any one of claims 6 to 8, wherein the determination means determines whether or not the image is only in the empty region based on the attitude information indicating the attitude of the image pickup apparatus at the time of imaging. Processing equipment. A program for operating a computer as the image processing device according to any one of claims 1 to 10. An image processing method for generating a corrected image in which at least a part of the influence of scattered light is removed from the image to be processed.
It is determined whether or not the image to be processed includes an empty area, and the image is determined.
For the image determined not to include the empty region, the image of only the empty region less affected by the scattered light was not used as the teacher data, and the image of the region other than the sky less affected by the scattered light was learned as the teacher data. Perform the first correction process using the model and
An image processing method for executing a second correction process on an image determined to include an empty area.

Images