JP2021113753A - Fogging determination device and fogging determination method - Google Patents

Abstract

To provide a fogging determination device that can determine whether fogging occurs in an imaging optical system and the like of an imaging device with higher accuracy based on an image captured by the imaging device.SOLUTION: A fogging determination device comprises: a characteristic-amount calculation unit 231 that, as to each pixel included in an image generated by a camera 1 having an imaging optical system 11, calculates an amount of characteristic in which values in accordance with a luminance difference among respective pixels of a set of pairs of pixels of the pixel and a plurality of peripheral pixels located around the pixel are arrayed according to an alignment sequence of the set of pairs of pixels; and a fogging determination unit 233 that, when the image is acquired, determines whether fogging occurs in an imaging optical system and the like on the basis of the amount of characteristic.SELECTED DRAWING: Figure 1

Description

The present invention relates to a fogging determination device and a fogging determination method.

There is known a technique in which a travel control device controls the travel of a vehicle by using an image showing the surrounding condition of the vehicle acquired by a camera mounted on the vehicle. The travel control device uses information on objects such as objects and features recognized based on images captured by the camera to control travel operations such as vehicle power, braking, and steering.

When an object is attached to the image pickup optical system of the camera, the image acquired through such an image pickup optical system has a region in which the object becomes unclear. Since it is difficult to extract an object from such a region, it is difficult for the travel control device to safely control the travel of the vehicle using such an image.

Patent Document 1 describes a deposit detection device that determines the presence or absence of deposits based on a histogram of edge strength, brightness, and saturation in a determination target area in a captured image.

Japanese Unexamined Patent Publication No. 2018-071994

The deposit detection device described in Patent Document 1 has an object of improving the detection accuracy when detecting deposits such as water droplets, snowflakes, ice, dust, and mud that block a part of an imaging optical system based on a captured image. There is. On the other hand, the imaging optical system of the camera may become cloudy. In an image taken through a cloudy imaging optical system, the contrast of the entire image is lowered, and the recognition accuracy of an object or the like is lowered. The deposit detection device described in Patent Document 1 cannot always accurately determine whether or not the imaging optical system of the camera is cloudy.

An object of the present invention is to provide a cloudiness determination device that determines with higher accuracy whether or not cloudiness has occurred in the image pickup optical system of the image pickup device based on the image acquired by the image pickup device.

In the cloudiness determination device according to the present invention, for each pixel included in an image generated by an image pickup device having an imaging optical system, a plurality of pairs of pixels among the pixel and a plurality of peripheral pixels located around the pixel. When an image is generated based on the feature amount calculation unit and the feature amount calculation unit that calculates the feature amount in which the values corresponding to the brightness difference between the pixels in each of the sets are arranged according to the arrangement order of the pair of pixel sets. The image pickup optical system or a cloudiness determination unit for determining whether or not the transparent member located on the object side of the image pickup optical system is cloudy is provided.

Further, in the cloudiness determination device according to the present invention, the feature amount calculation unit is a value associated with each peripheral pixel depending on whether or not the absolute value of the brightness difference between pixels in the pair of pixels is equal to or greater than the brightness threshold value. It is preferable to calculate the feature amount by arranging the above according to the arrangement order of the pair of pixels.

Further, in the fogging determination device according to the present invention, it is preferable that the pair of pixels is two peripheral pixels located at point-symmetrical positions with respect to the pixel.

Further, in the cloudiness determination device according to the present invention, the feature amount is preferably an LBP (Local Binary Pattern) feature.

Further, the cloudiness determination device according to the present invention further includes a histogram generation unit that generates a histogram indicating the appearance frequency of the feature amount, and the cloudiness determination unit can determine whether or not cloudiness has occurred based on the histogram. preferable.

In the cloudiness determination device according to the present invention, it is preferable that the cloudiness determination unit determines whether or not cloudiness has occurred by inputting a feature amount into a discriminator learned in advance.

Further, in the cloudiness determination device according to the present invention, among a plurality of images generated at different times within a predetermined period by the imaging device, for each of the pair of image sets, between the corresponding pixels in the pair of image sets. A representative value map generation unit that obtains the similarity of the feature amounts of It is preferable to determine whether or not cloudiness has occurred based on the representative value map.

Further, in the fogging determination device according to the present invention, the fogging determination unit causes fogging when a region larger than a predetermined size in which pixels having a representative value larger than the representative value threshold value are continuously detected in the representative value map. It is preferable to determine that it was.

In the cloudiness determination method according to the present invention, for each pixel included in an image generated by an image pickup apparatus having an imaging optical system, the brightness between the pixel and a plurality of peripheral pixels located around the pixel is between a pair of pixels. The feature amount is calculated by arranging the values according to the difference according to the arrangement order of each of the plurality of peripheral pixels, and based on the feature amount, when the image is generated, it is arranged on the object side of the image pickup optical system or the image pickup optical system. It includes determining whether or not the transparent member is clouded.

According to the cloudiness determination device according to the present invention, it is possible to determine with higher accuracy whether or not cloudiness has occurred in the image pickup optical system or the like of the image pickup device based on the image generated by the image pickup device.

It is the operation outline figure of the ECU which concerns on 1st Embodiment. It is a schematic block diagram of the vehicle equipped with the ECU which concerns on 1st Embodiment. It is a hardware schematic diagram of the ECU which concerns on 1st Embodiment. It is a functional block diagram of the processor which the ECU which concerns on 1st Embodiment has. It is a schematic diagram explaining the calculation of a feature amount. It is a figure which shows the example of the pattern of the LBP (Local Binary Pattern) feature. It is a schematic diagram of a histogram. It is a schematic diagram explaining the classification by SVM (Support Vector Machine). It is a processing flowchart of the ECU which concerns on 1st Embodiment. It is a functional block diagram of the processor which the ECU which concerns on 2nd Embodiment has. It is a figure explaining the generation of the representative value map. (A) is an example of an image acquired when the imaging optical system is cloudy, and (b) is an example of a representative value map corresponding to the image of (a). It is a processing flowchart of the ECU which concerns on 2nd Embodiment. It is a flowchart of representative value map generation processing.

Hereinafter, the fogging determination device and the fogging determination method will be described in detail with reference to the drawings. However, it should be understood that the present invention is not limited to the drawings or embodiments described below.

The fogging determination device receives an input of an image generated by the image pickup device, and determines whether or not fogging has occurred in the image pickup optical system or the like of the image pickup device when the image is generated. Therefore, the cloudiness determination device responds to the brightness difference between the pixels in each of the pair of a plurality of pairs of pixels among the pixel and the plurality of peripheral pixels located around the pixel for each pixel included in the image. The feature amount is calculated by arranging the values according to the arrangement order of the pair of pixels. Then, the fogging determination device determines whether or not fogging has occurred in the imaging optical system or the like when the image is generated, based on the feature amount.

FIG. 1 is a schematic operation diagram of the ECU according to the first embodiment.

In the present embodiment, the ECU (Electronic Control Unit) 2 is an example of a fogging determination device. The ECU 2 accepts the input of the image generated by the camera 1 mounted on the vehicle V. The ECU 2 calculates the feature amount of each pixel included in the image and generates a histogram. The feature amount of each pixel is a value corresponding to the brightness difference between the pixels in each of the pair of pixels of the pixel and the plurality of peripheral pixels located around the pixel, and is an array of the pair of pixels. The values are arranged in order. The histogram shows the frequency of appearance of features in the image. Then, the ECU 2 determines, based on the histogram, whether or not the imaging optical system 11 or the like of the camera 1 is cloudy when the image is generated. In the example of FIG. 1, the ECU 2 determines that cloudiness has occurred when the image A is generated, and that cloudiness has not occurred when the image B is generated. As a result, the ECU 2 can determine with higher accuracy whether or not the imaging optical system 11 or the like of the camera 1 is clouded based on the image generated by the camera 1.

FIG. 2 is a schematic configuration diagram of a vehicle equipped with the ECU according to the first embodiment.

The vehicle V is equipped with a camera 1, an ECU 2, and a GNSS (Global Navigation Satellite System) sensor 3, which are examples of an imaging device.

It has a camera 1, an imaging optical system 11, and a two-dimensional optical sensor 12. The camera 1 transmits the image pickup optical system 11 and generates an image corresponding to the image formed on the two-dimensional optical sensor 12. The camera 1 is installed in the vehicle interior, for example, with the front of the vehicle V facing so as to generate an image showing information around the vehicle V. A windshield 4 which is an example of a transparent member is arranged on the object side of the image pickup optical system 11 of the camera 1 installed in the vehicle interior with the front facing forward. That is, the camera 1 generates an image corresponding to the light L that passes through the windshield 4 and the imaging optical system 11 and forms an image on the two-dimensional optical sensor 12.

The ECU 2 controls the traveling of the vehicle V based on the object detected from the image generated by the camera 1. For example, the ECU 2 identifies the position of the object from the image and sets the accelerator opening, the brake amount, and the steering angle of the vehicle V so that the vehicle V does not collide with the object. Then, the ECU 2 controls the fuel injection device (not shown) of the engine of the vehicle V, the brake (not shown), and the steering wheel of the vehicle V with control signals according to the set accelerator opening, brake amount, and steering angle. Output to the actuator (not shown).

When the imaging optical system 11 or the windshield 4 is cloudy, the image becomes unclear as a whole, so that the recognition accuracy of an object or the like based on the image is lowered. Therefore, it becomes difficult for the ECU 2 to safely control the traveling of the vehicle V based on such an image.

When it is determined that the cloudiness has not occurred, the ECU 2 continues the traveling control of the vehicle V based on the object detected from the image generated by the camera 1. On the other hand, when it is determined that cloudiness has occurred, the ECU 2 controls to ensure the safety of the vehicle V. For example, the ECU 2 causes the information display (not shown) of the vehicle V to display a message requesting the driver to shift from automatic driving to manual driving. Further, the ECU 2 is controlled by a fuel injection device (not shown) of the engine of the vehicle V, a brake (not shown), and an actuator (not shown) that controls the steering wheel of the vehicle V so as to stop the vehicle V at a safe place. A signal may be output. For the route to a safe place, based on the position information specified by the GNSS sensor 3, the feature information is acquired from the feature information master (not shown) that stores the feature information in association with the point, and the feature information is obtained. It can be determined by identifying a safe location near the current location based on.

FIG. 3 is a schematic hardware diagram of the ECU according to the first embodiment.

The ECU 2 receives the image generated by the camera 1 mounted on the vehicle V, and determines whether or not the image pickup optical system 11 or the like of the camera 1 is clouded when the image is generated. Therefore, the ECU 2 includes an input / output interface 21, a memory 22, and a processor 23.

The input / output interface 21 is an interface circuit for inputting / outputting data between the ECU 2 and an external device. The input / output interface 21 supplies the received image data to the processor 23. Further, the input / output interface 21 outputs the data supplied from the processor 23 to the outside. The input / output interface 21 is, for example, a circuit for performing communication by CAN (Controller Area Network). Further, the input / output interface 21 may be a circuit for connecting peripheral devices such as USB (Universal Serial Bus).

The memory 22 is, for example, a semiconductor memory or a magnetic disk device. The memory 22 stores application programs, data, and the like used for processing by the processor 23. For example, the memory 22 stores a feature amount calculation program, a cloudiness determination program, and the like as application programs. The various programs may be installed in the memory 22 from a computer-readable portable recording medium using a known setup program or the like. The computer-readable portable recording medium is, for example, a memory card, a portable hard disk drive, or the like.

The processor 23 is one or more processors and peripheral circuits thereof. The processor 23 controls the overall operation of the ECU 2 in an integrated manner, and is, for example, a CPU (Central Processing Unit). The processor 23 controls the operation of the input / output interface 21 and the like so that various processes of the ECU 2 are executed by appropriate means based on the program and the like stored in the memory 22. The processor 23 executes processing based on an application program or the like stored in the memory 22. Further, the processor 23 can execute a plurality of application programs and the like in parallel.

FIG. 4 is a functional block diagram of a processor included in the ECU according to the first embodiment.

The processor 23 of the ECU 2 has a feature amount calculation unit 231, a histogram generation unit 232, and a cloudiness determination unit 233 as functional blocks. Each of these parts of the processor 23 is a functional module implemented by a program executed on the processor 23. Alternatively, each of these parts of the processor 23 may be implemented in the ECU 2 as an independent integrated circuit, microprocessor, or firmware.

When the feature amount calculation unit 231 receives the input of the image generated by the camera 1 via the input / output interface 21, the feature amount calculation unit 231 calculates the feature amount for each pixel included in the image. The feature amount of each pixel is a value corresponding to the brightness difference between the pixels in each of the pair of pixels of the pixel and the plurality of peripheral pixels located around the pixel. The values are arranged according to the order. The feature amount calculation unit 231 may calculate the feature amount each time it receives an input of a predetermined number of images.

The feature amount calculation unit 231 acquires the brightness of the pixel and the surrounding pixels for each pixel. Then, the feature amount calculation unit 231 calculates the feature amount based on the brightness of the pixel and each peripheral pixel.

FIG. 5 is a schematic diagram illustrating the calculation of the feature amount.

Schematic diagram 500 shows the arrangement of the target pixels for which the feature amount is calculated and the peripheral pixels thereof. Around the target pixel c, eight peripheral pixels are arranged in the order of _{peripheral pixels c 0} and c _{1 clockwise from the right side.} The feature amount calculation unit 231 compares the brightness n _c of the target pixel c with the brightness n ₀ -n _{7 of} the peripheral pixels c ₀ -c ₇ . That is, the target pixel c and the peripheral pixel c ₀ , the target pixel c and the peripheral pixel c _{1, and the} like are examples of a pair of pixels. _{Then, the feature amount calculation unit 231 sets 0 to the peripheral pixels whose brightness is smaller than n c} , and the peripheral pixels whose brightness is larger than or equal to the brightness of the target pixel, according to the brightness difference between the pixels of the pair of pixels. Is associated with 1. The feature amount calculation unit 231 calculates the feature amount of each pixel by arranging the values associated with the peripheral pixels according to the arrangement order of the peripheral pixels (for example, _{clockwise from the peripheral pixel c 0 on the right side).} ..

The feature amount calculation unit 231 determines each peripheral pixel depending on whether or not the absolute value of the difference between the _{brightness n c} of the target pixel c and the brightness n ₀ -n ₇ of the peripheral pixels c ₀ -c _{7 is equal to or greater than the brightness threshold value.} The feature amount may be calculated by arranging the values associated with the above according to the positions of the surrounding pixels. For example, in a certain embodiment, the feature amount calculation unit 231 sets _{0 to the peripheral pixels whose absolute value of the difference between the target pixel c and the brightness n c} is smaller than the brightness threshold, and the absolute value of the difference between the target pixel c and the brightness n _c. 1 is associated with surrounding pixels whose values are greater than or equal to the luminance threshold. Then, the feature amount calculation unit 231 calculates the feature amount of each pixel by arranging the values associated with the peripheral pixels according to the arrangement order of the peripheral pixels (for example, clockwise from the right side). The ECU that calculates the feature amount in this way can control the sensitivity of cloudiness detection by changing the brightness threshold value.

The feature amount calculation unit 231 may calculate the LBP (Local Binary Pattern) feature as a feature amount according to the following equation (1). In equation (1), P is the number of surrounding pixels and T _LBP is the luminance threshold.

Further, the feature amount calculation unit 231 may calculate the feature amount according to CSST (Center-Symmetric Census Transform). The feature amount calculation unit 231 that calculates the feature amount according to the CSST combines two peripheral pixels (for example, peripheral pixel c ₀ and peripheral pixel c ₄ ) located at positions symmetrical with respect to the target pixel c as a pair of pixels. do. Then, the feature amount calculation unit 231 calculates the feature amount by arranging the values corresponding to the brightness difference between the pixels in the pair of pixels according to the arrangement order of the pair of pixels. The ECU that calculates the feature amount in this way can execute the process at high speed because the calculation amount required for calculating the feature amount is reduced.

The ECU that calculates the feature amount according to CSCT has a luminance difference between a pair of pixels located at points symmetrical with respect to the target pixel c (| n ₀ -n ₄ |, | n ₁ -n in the schematic diagram 500 of FIG. 5). ₅ |, | n ₂ -n ₆ |, | n ₃ -n ₇ |). Then, the ECU that calculates the feature amount according to CSCT sets the value according to the brightness difference according to the arrangement order of the pair of pixels (for example, one of the pair of pixels is continuous from the upper left to the lower right of the target pixel). ) The feature amount is calculated by arranging.

In the present embodiment, the peripheral pixels are 3 × 3 pixels surrounding the target pixel, but the peripheral pixels are not limited to this. The peripheral pixels may be, for example, 4 × 4 pixels surrounding the target area, 5 × 5 pixels, or the like.

FIG. 6 is a diagram showing an example of a pattern of LBP features.

In FIG. 6, black circles indicate peripheral pixels associated with 0, and white circles indicate peripheral pixels associated with 1. That is, for example, the pattern P1 corresponds to a pixel in which one of the peripheral pixels is 1 and all the other peripheral pixels are 0. Such a pattern P1 has a feature amount such as "00000010". LBP features are classified into 36 types of patterns shown in FIG. 6 by rotating and aggregating matching ones. Further, by integrating 27 types of patterns that do not correspond to the patterns P0-P8 shown in FIG. 6 as pattern P9, the LBP features are classified into 10 types of patterns.

LBP features may be categorized without aggregating rotating matches. Further, the pattern of the LBP feature, which is a collection of rotating and matching patterns, may be classified into other than the 10 types shown in FIG. When the peripheral pixels are not 3 × 3 pixels surrounding the target pixel, the pattern of the LBP feature is appropriately classified according to the number and arrangement of the peripheral pixels.

The feature amount calculation unit 231 classifies the feature amount of each pixel into patterns P0-P9, and calculates a value corresponding to the classified pattern as a feature amount.

The histogram generation unit 232 generates a histogram showing the appearance frequency of the feature amount in the input image. The histogram shows the appearance frequency of pixels having the feature amount for each feature amount.

The histogram generation unit 232 aggregates the appearance frequency of the pixels having the feature amount corresponding to the pattern for each pattern of the feature amount calculated in step S11 for each pixel.

FIG. 7 is a schematic diagram of a histogram.

In the histogram 600, the horizontal axis corresponds to the patterns P0-P9 of the LBP feature, and the vertical axis shows the appearance frequency of the pixels classified into each pattern. Histogram 600 shows that in image A, the frequencies of patterns P0-P9 are 0.4, 0.07, 0.03, 0.04, 0.01, 0.01, 0.06, 0.08, 0.06, 0.24, respectively. Histogram 600 also shows that in image B, the frequencies of patterns P0-P9 are 0.15, 0.06, 0.02, 0.03, 0.01, 0.02, 0.08, 0.2, 0.15, 0.28, respectively. Thus, in the histogram, the frequency may be expressed as an appearance rate.

The histogram 600 is described for convenience in order to explain the histogram. The histogram generation unit 232 outputs an aggregated value of the appearance frequency of pixels having a feature amount corresponding to the pattern.

The fogging determination unit 233 determines whether or not fogging has occurred in the imaging optical system or the like when the image is generated, based on the feature amount. In particular, the cloudiness determination unit 233 of the present embodiment determines whether or not cloudiness has occurred based on the histogram showing the appearance frequency of the feature amount.

In the present embodiment, the feature amount histogram is a 10-dimensional value because it has an appearance frequency classified into 10 types of patterns. By inputting the histogram to the classifier, the fogging determination unit 233 determines whether or not fogging has occurred in the imaging optical system or the like when the image is generated.

The classifier classifies the given data into one of the classes. In this embodiment, the classifier is an SVM (Support Vector Machine). The SVM classifies the given data into classes using a discriminant function set based on machine learning using the classified teacher data.

FIG. 8 is a schematic diagram illustrating classification by SVM.

Scatter plot 800 shows the scatter state of the data points given to the SVM. As described above, in the present embodiment, the histogram of the feature amount is a 10-dimensional value, and the scatter diagram 800 is a virtual arrangement of the 10-dimensional value on a two-dimensional plane for explanation.

The set of data points 811 indicated by black circles corresponds to the histogram of the cloudy image. The set of data points 812 indicated by white circles corresponds to a histogram of a clear image. The discriminant function 820 is set so as to be able to discriminate between the set 811 and the set 812.

The discriminator classifies the data points located on the origin side as a clear image and the data points located on the opposite side of the origin as a cloudy image with respect to the set discrimination function 820.

FIG. 9 is a processing flowchart of the ECU according to the first embodiment.

The ECU 2 starts the process of FIG. 9 in response to the input acceptance of the image generated by the camera 1 by the input / output interface 21. First, the feature amount calculation unit 231 calculates the feature amount for each pixel included in the received image (step S11).

Next, the histogram generation unit 232 generates a histogram showing the appearance frequency of the feature amount in the image (step S12).

Next, the cloudiness determination unit 233 determines whether or not cloudiness has occurred in the imaging optical system 11 or the like when the image is generated based on the histogram generated in step S12 (step S13), and performs processing. finish. The ECU 2 controls the traveling of the vehicle V according to the determination result.

By processing as described above, the ECU according to the present embodiment determines with higher accuracy whether or not the imaging optical system of the imaging device is cloudy based on the image generated by the imaging device. be able to.

Next, the cloudiness determination device according to the second embodiment will be described. In the present embodiment, the ECU 2', which is an example of the cloudiness determination device, has a set of images for each of a pair of images among a plurality of images generated by the camera 1 at different times within a predetermined period. Find the similarity of the features between the corresponding pixels. Then, the ECU 2'determines whether or not cloudiness has occurred based on a representative value map having a representative value of the similarity of the corresponding pixels in each of the pair of image sets as the value of the pixel. As a result, the ECU 2'can determine with higher accuracy whether or not the imaging optical system 11 or the like of the camera 1 is clouded based on the image generated by the camera 1.

The ECU 2'according to the present embodiment has the same hardware configuration as the ECU 2 according to the first embodiment. The reference numerals of the components of the ECU 2'common with the ECU 2 are given the same reference numerals as those of the ECU 2, and the description thereof will be omitted.

FIG. 10 is a functional block diagram of a processor included in the ECU according to the second embodiment.

The processor 23'held by the ECU 2'according to the second embodiment has a representative value map generation unit 234 instead of the histogram generation unit 232 and a cloudiness determination unit 233'instead of the cloudiness determination unit 233. It is different from the processor 23 according to one embodiment.

The representative value map generation unit 234 has a feature amount between the corresponding pixels in the pair of images for each of the pair of images among the plurality of images generated by the camera 1 at different times within a predetermined period. Find the similarity of. The representative value map generation unit 234 further generates a representative value map having a representative value of the similarity of the corresponding pixels in each of the pair of image sets as the value of the pixel.

The representative value map generation unit 234 obtains the similarity of the feature amounts between the target pixels in the pair of images by calculating the exclusive OR of the feature amounts of the corresponding pixels in the pair of images. For example, it is assumed that the feature amount of the pixel corresponding to the coordinates (10,10) is "01110011" in one image and "01001111" in the other image. The similarity of these pixels is determined to be "00111100".

FIG. 11 is a diagram illustrating the generation of a representative value map.

Images F ₁ -F ₄ are images generated by the camera 1 at different times t ₁ -t _{4 in a predetermined period T.} The feature amount calculation unit 231 calculates _{the feature amount S 1} -S _{4 of} the pixels corresponding to the coordinates (x, y) for the _{images F 1} -F ₄ .

The representative value map generation unit 234 obtains the similarity S _{12 between the feature amount S 1} and the feature amount S ₂ between the corresponding pixels for the _{image F 1} and the image F ₂ , and creates the similarity map R _{12. Similarly, the similarity map R 23} and the similarity map R ₃₄ are created based on the similarity of the features between the corresponding pixels of the image F ₂ and the image F ₃ and the corresponding pixels of the image F ₃ and the image F _4. .. Image F ₁ and image F ₂ , image F ₂ and image F ₃ , and image F ₃ and image F ₄ are examples of a pair of images.

The pair of images may be a set of adjacent images in a time series in the camera 1 that generates images at a predetermined period (for example, 1/30 second), and a predetermined number (for example, 10) in the time series. Image) It may be a set of adjacent images at intervals.

The representative value map generation unit 234 obtains the representative value of the similarity based on the similarity _{maps R 12} , R _23, and R _34. For example, the representative value map I ₁₂₃₄ is generated by obtaining the total value S _{1234 of the similarity S 12} , the similarity S _23, and the similarity S ₃₄ as the representative value of the similarity for each pixel. The representative value of the similarity may be the maximum value, the mode value, the average value, the median value, or the like of the similarity.

The fogging determination unit 233'determines whether or not fogging has occurred in the imaging optical system or the like during the period when a plurality of images are generated, based on the feature amount. In particular, the cloudiness determination unit 233'of the present embodiment determines whether or not cloudiness has occurred based on a representative value map generated based on the feature amount of each pixel in a plurality of images.

FIG. 12 (a) is an example of an image generated when the imaging optical system is cloudy, and FIG. 12 (b) is an example of a representative value map corresponding to the image of FIG. 12 (a).

The image F shown in FIG. 12A and the images (not shown) generated at different times before and after are cloudy images. The representative value map generation unit 234 generates the representative value map I shown in FIG. 12B based on these images.

In the representative value map I, a region D in which pixels having a representative value larger than the representative value threshold value are continuous is detected. The representative value threshold value is a value that is appropriately set and is stored in the memory 22. The area A is an area larger than a predetermined size set in advance. When the region D is detected in the representative value map I, the cloudiness determination unit 233'determines that the image pickup optical system or the like is cloudy during the period when the image used for generating the representative value map I is generated. ..

The cloudiness determination unit 233'may determine whether or not cloudiness has occurred based on the representative value map I'that has been smoothed on the representative value map I. By using the smoothed representative value map I', the ECU 2'can more accurately determine the presence or absence of fogging. The fogging determination unit 233'can create a representative value map I'that has been smoothed, for example, by applying a Gaussian Filter to the representative value map I.

The fogging determination unit 233'may determine that fogging has occurred when a region of pixels having a large similarity is detected in the region of the lower half of the representative value map I. The camera 1 mounted on the vehicle V so as to generate an image of the outside of the vehicle generally generates an image of the sky in the upper half. Since the empty region in the image has a small variation in time series, it may be detected as a region of pixels having a large similarity even when cloudiness does not occur. The cloudiness determination unit 233'can determine the presence or absence of cloudiness more accurately by performing the determination focusing on the lower half region of the accumulation map I.

FIG. 13 is a processing flowchart of the ECU of the second embodiment.

First, the feature amount calculation unit 231 calculates the feature amount of each pixel included in the image generated by the camera 1 (step S21). The process of step S21 is the same as the process of step S11.

Next, the representative value map generation unit 234 generates a representative value map based on the feature amounts of the corresponding pixels in the plurality of images generated by the camera 1 at different times within a predetermined period (step S22). In the representative value map, for each of the pair of images among the plurality of images, the similarity of the feature amounts between the corresponding pixels in the pair of images, and the similarity of the target pixels in each of the pair of images. It is a map generated by using the representative value of degree as the value of the pixel. The representative value map generation process will be described later.

Next, the cloudiness determination unit 233'performs the cloudiness determination based on the representative value map (step S23), and ends the process. The ECU 2'controls the traveling of the vehicle V according to the determination result.

FIG. 14 is a flowchart of the representative value map generation process.

When the representative value map generation process is started, the representative value map generation unit 234 describes the pair of images generated by the camera 1 at different times within a predetermined period for each of the pair of images. The similarity of the feature amounts in the corresponding pixels in the set of images is obtained (step S221).

Subsequently, the representative value map generation unit 234 generates a representative value map having the representative value of the similarity of the corresponding pixels in each of the pair of image sets as the value of the pixel (step S222), and the representative value map generation process. To finish.

The fogging determination device is not limited to the ECU mounted on the vehicle, and may be implemented as a general information processing device such as a personal computer.

It will be appreciated by those skilled in the art that various changes, substitutions and modifications can be made to this without departing from the spirit and scope of the invention.

2, 2'ECU
231 Feature calculation unit 232 Histogram generation unit 233, 233'Cloudiness judgment unit 234 Representative value map generation unit

Claims (9)

For each pixel included in an image generated by an image pickup apparatus having an imaging optical system, the brightness between the pixels in each of the pixel and a set of a plurality of pairs of pixels among a plurality of peripheral pixels located around the pixel. A feature amount calculation unit that calculates a feature amount in which values corresponding to the difference are arranged according to the arrangement order of the pair of pixels.
Based on the feature amount, a fogging determination unit for determining whether or not fogging has occurred in the imaging optical system or the transparent member located on the object side of the imaging optical system when the image is generated. A cloudiness judgment device provided. The feature amount calculation unit sets a value associated with each peripheral pixel depending on whether or not the absolute value of the brightness difference between pixels in the pair of pixels is equal to or greater than the brightness threshold value of the pair of pixels. The cloudiness determination device according to claim 1, wherein the feature amount is calculated by arranging the features according to the order of arrangement. The fogging determination device according to claim 1 or 2, wherein the pair of pixels is two peripheral pixels located at point-symmetrical positions with respect to the pixel. The cloudiness determination device according to any one of claims 1 to 3, wherein the feature amount is an LBP (Local Binary Pattern) feature. Further, a histogram generation unit for generating a histogram showing the appearance frequency of the feature amount is provided.
The cloudiness determination device according to any one of claims 1 to 4, wherein the cloudiness determination unit determines whether or not cloudiness has occurred based on the histogram. The cloudiness determination device according to claim 5, wherein the cloudiness determination unit determines whether or not cloudiness has occurred by inputting the feature amount into a pre-learned classifier. For each of the pair of image sets among the plurality of images generated by the image pickup device at different times within a predetermined period, the similarity of the feature amounts between the corresponding pixels in the pair of image sets is obtained. A representative value map generation unit for generating a representative value map having the representative value of the similarity of the corresponding pixel in each of the pair of images as the value of the pixel is further provided.
The cloudiness determination device according to any one of claims 1 to 4, wherein the cloudiness determination unit determines whether or not cloudiness has occurred based on the representative value map. 7. The cloudiness determination unit determines that cloudiness has occurred when a region larger than a predetermined size in which pixels having the representative value larger than the representative value threshold value are detected in the representative value map. The cloudiness determination device according to. For each pixel included in an image generated by an image pickup apparatus having an imaging optical system, the brightness between the pixels in each of the pixel and a set of a plurality of pairs of pixels among a plurality of peripheral pixels located around the pixel. The feature amount in which the values corresponding to the difference are arranged according to the arrangement order of the pair of pixels is calculated.
Cloudiness including determining whether or not cloudiness has occurred in the image pickup optical system or a transparent member arranged on the object side of the image pickup optical system when the image is generated based on the feature amount. Judgment method.

Images