JP7406886B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to an image processing technique that reduces the influence of scattered light on captured images.

BACKGROUND ART In the field of surveillance cameras and the like, it has become a problem that the contrast is reduced due to the influence of particulates (for example, fog and haze) that exist between the camera and the subject, and the image quality of captured images is degraded. The cause of this is that light is scattered by particulate components when it passes through the atmosphere. Image processing techniques for removing the influence of scattered light from images whose visibility has been degraded due to such scattered light are known.

Patent Document 1 describes a method using machine learning as fog removal processing. Specifically, when a region of a foggy image is input, the learning model generates transmittance corresponding to the region, and uses the transmittance obtained by the learning model to generate a fog-removed image.

US Patent Publication 2016/0005152

Changes in visibility vary depending on the subject due to the influence of fine particles such as fog and haze. For example, the contrast of buildings with texture decreases, while the hue changes in areas of the sky with little texture. However, when learning to output a transmittance map from an image, the method disclosed in Patent Document 1 does not take into consideration the fact that visibility changes differently depending on the subject, and it is not always possible to obtain an appropriate learning model. do not have.

Therefore, it is an object of the present invention to realize processing using a model that has been appropriately learned for subjects whose contrast decreases due to fog when performing fog and haze removal processing on images using machine learning. .

In order to solve the above problems, the present invention provides an image processing apparatus that generates a corrected image in which at least a part of the influence of scattered light is removed from an image to be processed, the invention comprising: a determining means for determining whether or not a sky area is included; a first correction processing means for performing a first correction process on an image determined by the determining means to not include a sky area; A second correction process is executed by the means on the sky area of the image determined to include the sky area , and the first correction process is executed on the area other than the sky area of the image. The first correction processing executed by the first correction processing means does not use an image of only the sky region, which is less affected by scattered light, as training data; A feature of this process is that it uses a model learned using images of areas that do not include sky areas as training data.

According to the present invention, when performing fog/haze removal processing on an image using machine learning, it is possible to realize processing using a model appropriately learned for a subject whose contrast is reduced due to fog/haze.

Diagram showing the hardware configuration of the image processing device Diagram showing the detailed functional configuration of the image processing device Diagram showing an example of a learning image Diagram explaining neural network Learning process flowchart Flowchart of mist removal process Diagram showing the detailed functional configuration of the image processing device Diagram showing GUI for inputting indoor and outdoor information Flowchart of mist removal process

Hereinafter, the present invention will be described in detail based on preferred embodiments with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not necessarily limited to the illustrated configurations.

<First embodiment>
In the first embodiment, an image processing apparatus that performs fog removal processing on an image will be described. Note that fog/haze removal processing refers to processing for reducing the influence of scattered light on an image whose contrast has been reduced due to the influence of scattered light due to particulate components such as fog and haze. In this embodiment, the sky region of the image is corrected pixel by pixel using preset parameters, and the correlation between the image without fog and the image with fog is calculated for regions other than the sky region. Correction is made using the parameters of the learning model obtained through learning.

FIG. 1 is a block diagram showing the hardware configuration of an image processing apparatus 100 of this embodiment. The image processing apparatus 100 of this embodiment includes a CPU 101, a RAM 102, a ROM 103, an HDD I/F 104, an HDD 105, an input I/F 106, an output I/F 107, and a system bus 108. The CPU 101 is a processor that centrally controls each component described below. The RAM 102 is a memory that functions as the main memory and work area of the CPU 101, and the ROM 103 is a memory that stores programs that control processing within the image processing apparatus 100.

The HDD I/F 104 is, for example, an interface such as serial ATA (SATA), and connects the HDD 105 as a secondary storage device to the system bus 108. The CPU 101 is capable of reading data from and writing data to the HDD 105 via the HDD I/F 104. Further, the CPU 101 can expand the data stored in the HDD 105 into the RAM 102, and similarly can save the data expanded into the RAM 102 into the HDD 105. The CPU 101 can then regard the data developed in the RAM 102 as a program and execute it. Note that the secondary storage device may be a storage device such as an optical disk drive in addition to the HDD. The input I/F 106 is, for example, a serial bus interface such as USB or IEEE1394.

The image processing device 100 is connected to an external memory 109 and an imaging unit 111 via an input I/F 106. The CPU 101 can acquire data from the external memory 109 and the imaging unit 111 via the input I/F 106. The output I/F 107 is, for example, a video output interface such as DVI or HDMI (registered trademark). The image processing device 100 is connected to a display unit 110 via an output I/F 107. The CPU 101 can output an image to the display unit 110 via the output I/F 107 and display the output image on the display unit 110.

The system bus 108 is a transfer path for various data, and each component within the image processing apparatus 100 is interconnected via the system bus 108.

The external memory 109 is a storage medium such as a hard disk, a memory card, a CF card, an SD card, or a USB memory, and can store image data processed by the image processing apparatus 100. The display unit 110 is a display device such as a display, and can display images processed by the image processing device 100.

The imaging unit 111 is a camera that receives optical information of a subject with a sensor and outputs the acquired image as digital data. In a captured image obtained by imaging by the imaging unit 111, the contrast may be reduced due to the influence of scattered light, such as in a scene where fog or haze has occurred. In this embodiment, by performing fog removal processing on the image captured by the imaging unit 111, it is possible to generate an image in which the influence of scattered light is reduced.

FIG. 2 is a block diagram showing the detailed logical configuration of the image processing apparatus 100 in this embodiment. The image processing device 100 includes a fog/haze removal processing section 200 that performs fog/haze removal processing and a learning data generation section 201 that generates learning data.

First, the learning data generation unit 201 will be explained. The learning data generation unit 201 generates learning data that is a pair of student data and teacher data for optimizing the parameters of a learning model that performs machine learning. The learning model is a network structure based on a neural network that outputs a haze-reduced image corresponding to the student data from input student data, and its parameters. The learning model in this embodiment is a model that performs a process of estimating an image in which the influence of scattered light due to fog haze is reduced from an image influenced by scattered light due to fog haze. Student data is generated from images whose contrast is reduced due to fog. Further, the teacher data is generated from a fog-haze-reduced image when there is no reduction in contrast due to fog in the image that is student data.

The learning data generation 201 includes a learning image acquisition section 202 , a block division section 203 , an area determination section 204 , and a learning section 205 . The learning image acquisition unit 202 acquires two images of the same scene captured from the same position and under the same imaging conditions. FIG. 3A is a diagram showing a part of the image acquired by the learning image acquisition unit 202. The two images acquired at this time are a clear sky image taken from the same location and the same scenery on a clear day, and a foggy image taken under the influence of scattered light due to foggy haze or the like. Note that here, it is assumed that the clear sky image and the haze image are captured under the same imaging conditions (exposure, angle of view, etc.). Furthermore, both the clear sky image and the haze image are color images consisting of R (red), G (green), and B (blue) planes. The learning image acquisition unit 202 acquires a plurality of pairs of such clear sky images and haze images.

The block division unit 203 divides the acquired pair of the clear sky image and foggy image, and outputs block images. The block dividing unit 203 divides the image into blocks according to the number of nodes in the input layer of the neural network in the learning unit 204, as described later. The block dividing unit 203 sequentially outputs pairs of divided block images to the area determining unit 204. The region determining unit 204 determines whether or not a sky region is included, based on the block extracted from the clear sky image among the block pairs to be processed. The area determination unit 204 does not output blocks determined to include a sky area to the learning unit 204, and outputs blocks determined to not include a sky area to the learning unit 205.

The learning unit 205 uses the pair of block images that do not include a sky area and is output from the block dividing unit 203 to learn the parameters of the neural network held internally.

Here, neural networks will be explained. FIG. 4 is a diagram illustrating a neural network. In FIG. 4, the intermediate layer is shown as one layer to simplify the explanation, but it is desirable to configure the intermediate layer with two or more layers. In the neural network shown in FIG. 4, the input layer has Mi nodes (n11, n12, ..., n1Mi), the middle layer has Mh nodes (n21, n22, ..., n2Mh), and the output layer (Final layer) has Mo nodes (n31, n32, . . . , n3Mo). The nodes of each layer are connected to all the nodes of adjacent layers, forming a three-layer hierarchical neural network that transmits information between layers.

When inputting an image to the input layer, nodes for the number of input pixels are provided in the input layer so that there is a one-to-one relationship between input pixels and nodes. Further, nodes for the number of pixels to be output are set in the output layer as well. That is, in this embodiment, a block image of 16 pixels x 16 pixels is input, and pixel values of 16 pixels x 16 pixels are output, so there are 256 nodes in the input layer and the output layer. Data is passed from left to right in FIG. 4, that is, in the order of input layer, intermediate layer, and output layer. Each node in the input layer is connected to all nodes in the hidden layer, and each connection between nodes has a weight. The output value transmitted from one node to another node through a connection is enhanced or attenuated by the weight of the connection. A set of weighting coefficients and bias values determined for such connections are parameters of the learning model. Note that the activation function is not particularly limited, but a logistic sigmoid function, a rectified linear unit (ReLU) function, or the like may be used. As the learning method, various proposed neural network learning methods can be applied. For example, when student data is input to the input layer and a neural network is operated, the difference between the output obtained from the output layer and the teacher data that is previously associated with the student data is calculated, and the difference is calculated. The weighting coefficient and bias value are adjusted to minimize them. The weighting coefficients and bias values generated by such learning processing are parameters of the learning model. In this embodiment, the learning unit 205 uses a foggy image as student data and a clear sky image as teacher data, so the learning unit 205 uses a learning model that estimates an image in which the influence of scattering is reduced from an image affected by scattered light due to foggy haze. Generate parameters for .

Next, the flow of learning processing in the learning data generation unit 201 will be explained in detail. FIG. 5 is a flowchart of the learning process. Each configuration (function) is realized by the CPU 101 reading and executing a program that can realize the flowchart shown in FIG. In addition, below, each process (step) in a flowchart is attached|subjected with "S", and is demonstrated.

In S501, the learning image acquisition unit 202 acquires a set of images prepared for learning from a memory (not shown). As described above, here, a clear sky image captured from the same location under the same wind conditions on a clear day, and a foggy image captured under the influence of scattered light due to occurrence of foggy haze, etc. are acquired. For example, two pieces of image data may be acquired by referring to the weather forecast from an external memory in which images captured by a fixed surveillance camera installed at a predetermined position are stored. In this case, it is preferable to obtain an image pair from images captured by a plurality of different surveillance cameras.

In S502, the block dividing unit 203 divides one set of images into blocks. In this embodiment, block images of 16 pixels x 16 pixels are sequentially extracted from the upper left of the image. FIG. 3B is a diagram illustrating blocks divided by the block division unit 203. As shown in FIG. 3(b), the image is divided into blocks in the form of tiles, and a plurality of block images are extracted from one image. The block dividing unit 203 outputs two blocks extracted from the same position in the clear sky image and the corresponding haze image to the area determining unit 204 as one block pair.

In S503, the area determination unit 204 determines whether the block image of the clear sky image among the acquired pair of block images includes a sky area. A known technique may be used to determine whether the area is empty or not. In this embodiment, the hue is calculated from the R, G, and B pixel values of each pixel in the block, and it is determined whether the hue is within a preset range of hues that are likely to be sky. The area determination unit 204 calculates the number of pixels determined to be sky area pixels in the block, and determines that the block image includes a sky area if the number of sky area pixels is greater than or equal to a predetermined threshold; S504 is skipped and the process advances to S505. Further, if the number of pixels in the sky area is less than a predetermined threshold, the area determination unit 204 determines that the block image does not include a sky area, and the process proceeds to S504.

In S<b>504 , the area determination unit 204 outputs only pairs of block images determined not to include a sky area to the learning unit 205 . In S505, the area determination unit 204 determines whether area determination has been performed on all of the acquired block images. If area determination has been performed for all block images, the process returns to S506; if there are unprocessed block images, the process returns to S503. Further, in S506, the learning image acquisition unit 202 determines whether all learning images have been acquired. If all learning images have been acquired, the process advances to S507, and if there are learning images that have not yet been acquired, the process returns to S501.

In S507, the learning unit 205 uses the input learning data to generate parameters for a learning model that estimates an image with reduced effects of scattering. In S508, the learning unit 205 outputs the parameters of the generated learning model to the first correction unit 209, which will be described later.

From here, the fog/haze removal processing unit 200 that actually executes fog/haze removal will be explained. First, the image input unit 206 inputs an image to be processed in which the influence of scattering due to fog and haze is to be removed. The block dividing unit 207 divides the image to be processed into blocks. In this embodiment, the block dividing unit 207 divides the block image into blocks having the same size as the size of the block image extracted by the block dividing unit 203. The area determination unit 208 determines whether the block to be processed includes an empty area. The area determination unit 208 outputs blocks that do not include a sky area to the first correction unit 209 and outputs blocks that include a sky area to the second correction unit 210.

The first correction unit 209 includes a neural network. Here, the first correction section 209 has the same network structure as the learning section 205. Furthermore, each weighting coefficient and bias value in the neural network is set according to the parameters output from the learning section 205. In other words, the second correction unit 209 performs correction processing on the block to be processed based on a model that removes the influence of fog and haze, which is learned using an image that does not include a sky region. The first correction unit 209 estimates an image with the original contrast by removing the influence of fog from the image to be processed, and outputs a corrected block.

The second correction unit 210 sequentially performs arithmetic processing according to a predetermined algorithm on each pixel included in the block to be processed, and outputs a corrected block including a sky area. It is known that when an image is affected by fog and haze, the contrast decreases compared to an image that is not affected by fog and haze. However, the sky region often does not contain complex texture components, and if the contrast is expanded as part of the fog/haze removal process, the S/N ratio will decrease. Therefore, in this embodiment, the second correction unit 210 executes correction processing suitable for the sky area.

The output image generation unit 211 accumulates the block image output from the first correction unit 209 and the block image output from the second correction unit 210, generates one image, and outputs the same.

The flow of the fog and haze removal processing executed by the fog and haze removal processing unit 200 will be described in detail. FIG. 6 is a flowchart of fog removal processing. Each configuration (function) is realized by the CPU 101 reading and executing a program that can realize the flowchart shown in FIG.

First, in S601, the image input unit 206 inputs an image to be processed. Here, similarly to the learning image data, a color image consisting of R, G, and B planes is used as the image to be processed. In S602, the block dividing unit 203 divides the image to be processed. In this embodiment, the image to be processed is divided into blocks of 16 pixels x 16 pixels.

In S603, the area determination unit 208 determines whether the block to be processed includes an empty area. Here, area determination for the block is performed using a method similar to that of the area determination unit 204 in S503 described above. If the area determination unit 208 determines that the block includes a sky area, the process proceeds to S604 and outputs the block image to the first correction unit 209. If the area determination unit 208 determines that the block does not include a sky area, the process proceeds to step S<b>605 and outputs the block image to the second correction 210 .

In S605, the first correction unit 209 performs a first correction process on the input block image. Here, the first correction process is a process performed by a neural network in which parameters are set based on a learning model for estimating the fog-haze removed image. Through this processing, the block image is converted into a corrected block whose contrast has been increased and the influence of fog and haze has been reduced.

In S607, the second correction unit 210 performs the second correction process on the input block image including the sky area. In the present embodiment, the second correction process is a saturation correction process. The second correction unit 210 sequentially sets the pixels included in the block to be processed as the pixel of interest, and first converts the R, G, and B pixel values of the pixel of interest into the YCbCr color space. The color space conversion process is executed using a known calculation formula. The second correction unit 201 multiplies the Cb and Cr components by a predetermined gain.

In S608, the area determination unit 208 determines whether determination processing has been performed on all blocks in the image to be processed, and if there are unprocessed blocks, the process returns to S603. If the determination processing for all has been completed, the process advances to S609.

In S609, the output image generation unit 211 accumulates data of corrected blocks in the image to be processed, generates one corrected image, and outputs it. With this, the mist removal process is completed.

As described above, in this embodiment, a corrected image in which the influence of scattered light is reduced is generated for a subject region that does not include a sky region in an image using a learning model obtained by supervised learning. On the other hand, for the sky region, a corrected image in which the influence of scattered light is reduced is generated by saturation correction processing.

Normally, when affected by scattered light, contrast deteriorates in areas other than sky areas. Therefore, it is desirable that the image from which the influence of scattered light is removed is subjected to correction processing that increases the contrast of the image that is affected by scattered light. Therefore, correction processing using a learning model that estimates an image with contrast reduction using a pair of an image with reduced contrast and an image with no reduction is suitable for regions other than the sky. However, since there is no texture component in the sky region, increasing the contrast may degrade the image quality due to an increase in noise components. If a model that has undergone supervised learning using clear sky images as training data is applied to the sky region, the image quality of the sky region will deteriorate due to the increase in contrast. In other words, for the sky region, instead of correction processing using a learned model, a saturation correction process suitable for the sky region is executed to generate a corrected image that better reduces the effects of scattered light. Can be done.

Furthermore, the influence of scattered light in a sky region is different from the influence of scattered light in a region other than the sky. For example, in the case of a sky region, the image will appear bluish when the sky is clear, and grayish when affected by scattered light. Therefore, in the sky region, it is desirable that achromatic colors such as gray be corrected to appear bluish by fog removal processing. On the other hand, in areas other than the sky, there are objects such as buildings that are themselves gray. Therefore, in areas other than the sky, even if the area is gray in the fog image, it is not necessarily desirable to correct it to blue as a result of the fog removal process. Therefore, when learning a model that uses clear sky images as training data to estimate an image with the influence of fog and haze removed from a foggy image, it is possible to train using images of areas other than the sky. Learning models can be learned more appropriately.

<Second embodiment>
In the first embodiment, a method has been described in which it is determined whether or not an image to be processed includes a sky area, and the first correction process and the second correction process are switched. In the present embodiment, a method will be described in which correction processing is controlled by acquiring position information and orientation information where an imaging device that captured an image to be processed is installed. Note that the same components as in the first embodiment are given the same reference numerals, and the description thereof will be omitted.

The fog/haze removal processing unit 700 in this embodiment includes a position information acquisition unit 810, a posture information acquisition unit 711, and an image determination unit 712 in addition to those in the first embodiment. The position information acquisition unit 710 displays a graphical user interface (GUI) shown in FIG. 8 to the user and receives input from the user. Thereby, the position information acquisition unit 710 acquires indoor/outdoor information indicating whether the imaging unit 111 is installed indoors or outdoors as position information.

The posture information acquisition section 711 acquires information indicating the posture of the imaging section 111 when the processing target image is photographed. Here, the posture information acquisition section 711 acquires information regarding the elevation angle direction of the imaging section 111 from the imaging section 111 as posture information. That is, the imaging unit 111 has a built-in angle sensor (not shown) that can detect an angle in the elevation direction, and when transferring an image to be processed to the fog/haze removal processing unit 700, the angle of the angle sensor is Information will also be transferred.

The image determination unit 712 determines whether there is a possibility that the image to be processed includes a sky area. Depending on the determination result, the image to be processed is output to one of the block division section 207, the first correction section 209, and the second correction section 210.

The flow of the fog and haze removal processing executed by the fog and haze removal processing unit 700 will be described in detail. Note that the parameters of the neural network in the first correction unit 209 are updated by parameters adjusted by the same learning method as in the first embodiment. FIG. 9 is a flowchart of fog removal processing.

In S901, the fog/haze removal processing unit 700 acquires an image to be processed. In S902, the position information acquisition unit 710 displays the image shown in FIG. 8 on the display unit 110. In S903, the position information acquisition unit 710 acquires indoor/outdoor information indicating the result of the user selecting either outdoors or indoors via the GUI.

In step S<b>905 , the attitude information acquisition unit 711 acquires information regarding the elevation angle direction of the imaging unit 111 from the imaging unit 111 as attitude information. In this embodiment, the attitude information acquisition unit 711 determines -90 degrees when the imaging unit 111 is facing the ground directly below, 0 degrees when the horizontal direction is facing, and 0 degrees when the imaging unit 111 is facing the ceiling or sky directly above. The angle in the elevation direction is obtained as 90 degrees.

In S905, the image determination unit 712 determines whether or not there is a possibility that the image to be processed includes a sky area, based on the position information and orientation information. When indoors is selected by user input, or when outdoors is selected and the orientation information indicates -45 to -90, it is determined that the image to be processed does not include a sky area. This is because even if the image is captured outdoors, the imaging direction is facing the ground or the floor, and there is a low possibility that the image includes a sky area. If it is determined that the image to be processed does not include a sky area, the image determination unit 712 then proceeds to S911 and outputs the image to be processed to the second correction unit 210. Furthermore, if outdoors is selected and the orientation information indicates -45 to 90 degrees, it is determined that the sky area is included. When determining that the image includes a sky area, the image determining unit 712 advances to S906.

In S906, the image determination unit 712 further determines whether the image to be processed is only a sky area. In this embodiment, the image determination unit 712 calculates the hue from the pixel values in the color image and stores it. Furthermore, the image determination unit 712 counts the number of pixels in which the hue of each pixel is within a range recognized as sky, and if a predetermined ratio, for example, 95% or more of the pixels have a sky hue, only the sky area is counted. It is determined that If the image is a sky area-only image, the image determination unit 712 proceeds to S907 and outputs the image to be processed to the first correction unit 209. On the other hand, if it is determined that the image includes a sky area but is not only a sky area, the image determination unit 712 advances to S909.

S909 and S910 are the same processes as S902 and S903 in the first embodiment. However, in S903, the area determination unit 208 does not calculate the hue of each pixel, but may refer to the hue of each pixel calculated and stored by the image determination unit 712.

In S907, the second correction unit 210 receives a block image including a sky region or an image containing only a sky region. In S908, the second correction unit 210 performs second correction processing on the input image.

In S911, a block image that does not include a sky area or an image that does not include a sky area is input to the first correction unit 210. In S912, the first correction unit 209 performs first correction processing on the input image. In S913, when the correction processing for all pixels is completed, the output image generation unit 211 outputs a corrected image.

As described above, in this embodiment, fog removal processing for an image is controlled according to position information and orientation information. Fog removal processing is classified into three methods, and images are processed by one of these methods. The first is a method of performing correction processing that performs arithmetic processing on each pixel on the image, the second is a method of performing correction processing on the image using a neural network, and the third method is to perform correction processing on the image using a neural network. The second method is to switch between the first method and the second method depending on the block. For example, in the case of a surveillance camera, once it is installed, it is often fixed for a while, and especially when it is installed indoors, it is unlikely that the sky area will be included in the image captured by the surveillance camera. Therefore, in this embodiment, it is first determined whether the image to be processed includes a sky region based on indoor/outdoor information and posture information. As a result, when only a sky region or no sky region is included, it is possible to perform correction processing suitable for each case without performing block division or region determination processing. Thereby, the processing load can be reduced. Note that the processing by the image determination unit 710 does not have to be performed every time an image to be processed is input. It may be executed only when the installation position is changed or when settings are reset, and in other cases, the result determined once may be used.

<Other embodiments>
In the embodiment described above, a case has been described in which a model for estimating an image with reduced fog and haze from a fog and haze image is learned using a clear sky image as training data. However, there are other methods of correction processing using machine learning. For example, a learning model for estimating a transmittance map from a foggy haze image may be learned by learning a foggy haze image as student data and a transmittance map corresponding to the foggy haze image as teacher data. When performing a correction process in which parameters of such a learning model are set, first, a transmittance map corresponding to an image to be processed is estimated via a neural network. Thereafter, the contrast adjustment process may be performed for each pixel in the image to be processed by referring to the transmittance map. The mist removal process using the transmittance map is well known, so a description thereof will be omitted.

In the above-described embodiment, an example has been described in which it is determined whether each block is an empty area or not, and the first or second correction processing is executed for each block. However, the processing does not necessarily have to be performed for each rectangular block. For example, an image is divided into a sky region and a non-sky region using well-known semantic region segmentation, and an image in which only the sky region is extracted is subjected to the second correction process, and an image in which only the non-sky region is extracted is subjected to the second correction process. The first correction process may be executed.

Furthermore, the description of the first embodiment has been made on the premise that the image to be processed includes a sky region and a non-sky region. However, if the image does not include a sky area, the first correction process will be performed on the entire area in the image.

In the embodiment described above, an example was described in which a pair of a clear sky image and a haze image is used as a learning image. However, there are cases where there are no two images taken from the same position and under the same imaging conditions. For example, for a foggy image, a known foggy removal process such as a dark channel method may be used to generate an image in which foggy haze is reduced, and this image may be used as the teacher data.

Further, although a method of calculating hue from pixel values in a color image has been described as an area determination process, for example, a distance image corresponding to the image may be used to determine whether or not it is a sky area.

In the previous explanations, fog and mist have been used as examples of factors that cause scattered light. It can be applied in the same way.

The present invention provides a system or device with a program that implements one or more functions of the embodiments described above via a network or a storage medium, and one or more processors in a computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions. The input I/F 106 is, for example, a serial bus interface such as USB or IEEE1394.

202 Image acquisition unit for learning 203 Block division unit 204 Area determination unit 205 Learning unit 206 Image input unit 207 Block division unit 208 Area determination unit 209 First correction unit 210 Second correction unit 211 Output image generation unit

Claims (12)

An image processing device that generates a corrected image in which at least part of the influence of scattered light is removed from an image to be processed,
determining means for determining whether the image to be processed includes a sky area;
a first correction processing unit that performs a first correction process on an image that is determined by the determination unit to not include a sky area;
A second correction process is executed on the sky area of the image determined by the determination means to include a sky area , and the first correction process is executed on areas other than the sky area of the image. a second correction processing means;
The first correction processing executed by the first correction processing means does not use an image of only a sky region that is less affected by scattered light as training data, but uses an image of an area that does not include a sky region that is less affected by scattered light as training data. An image processing device characterized in that processing uses a learned model as teacher data. 2. The image processing apparatus according to claim 1, wherein the first correction processing means includes a neural network, and parameters are set according to the model. 3. The image processing apparatus according to claim 1, wherein the second correction process is an arithmetic process for each pixel. The image processing apparatus according to any one of claims 1 to 3, wherein the determination unit performs the determination for each block of the image to be processed. The determination means determines the image to be processed,
If it is determined that the image to be processed does not include a sky area, only the first correction process by the first correction processing means is performed on the image to be processed. 4. The image processing device according to any one of 3 to 3. 6. The image according to claim 5, wherein when the determining means determines that the image to be processed includes a sky area, the determining means further determines whether the image to be processed includes only a sky area. Processing equipment. 7. If the determining means determines that the image is only a sky area, only the second correction processing by the second correction processing means is performed on the image to be processed. image processing device. If the determination means determines that the image to be processed includes a sky area but not only a sky area, the image to be processed is subjected to the first correction process or the second correction process for each block. The image processing apparatus according to claim 6 or 7, wherein any one of the processes is executed. The determination means determines whether the image includes a sky area based on information indicating whether the imaging device that captured the image to be processed is installed indoors or outdoors. The image processing device according to any one of claims 5 to 8. 9. The determining means determines whether or not there is only the sky area based on posture information indicating the posture at the time of image capturing of the image capturing device that captured the image to be processed. The image processing device according to item (1). A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 10. An image processing method for generating a corrected image in which at least part of the influence of scattered light is removed from an image to be processed, the method comprising:
a determining means determines whether the image to be processed includes a sky area;
For images determined not to include sky regions , the first correction processing means does not use images of only sky regions that are less affected by scattered light as training data, but includes sky regions that are less affected by scattered light. Executing a first correction process using a model learned using an image of an area where there is no area as training data,
A second correction processing means executes a second correction process on a sky area of the image determined to include a sky area , and performs the first correction process on an area other than the sky area of the image. An image processing method that performs

Images